Modelling and Transformation of Non-Player Characters' Behaviour

Modelling and Transformation of Non-Player Characters’ Behaviour in Modern Computer Games

Gino Wuytjens

Supervisor: Prof. Dr. Hans Vangheluwe

Faculty of Science University of Antwerp Antwerp, Belgium

May 1st, 2012

A thesis submitted to University of Antwerp in fulﬁlment of the requirements of the degree of Master of Science in Information Technology

During this thesis, a formal architecture for developing NPC behaviour through the use of the Stat- echarts formalism is developed. Creating NPC behaviour has so far been done using textual programming languages, often times using a state-based architecture. This lends itself to new research in using the Statecharts modelling formalism to improve several aspects of NPC behaviour and its development, including but not limited to re-usability of code, readability, automatic generation and variability. This thesis develops a testing platform in which such an architecture can be extensively studied and analysed, through the use of a Tank Wars style game. Furthermore, a Transformation platform is developed with the aim of generating NPC behaviour from a collection of components. This generation is done based on a provided heuristic, which determines what kind of behaviour is desired by the developer. The transformation platform further introduces variability into the generated behaviour by allowing parameter transformation, component structure transformation and Statecharts model transformation to be carried out following rules deﬁned by the user. The designs and technologies developed throughout this thesis are kept as general as possible, with the aim of making this work applicable in a wide array of contexts, such as time-based and event-based game engines. This thesis is a ﬁrst foray into this kind of behaviour development and lends itself well as the basis for even more advanced and expansive strategies for developing NPC behaviour based on the proposed architecture using the Statecharts formalism.

i Contents

1 Introduction 1 1.1 Problem Statement ...... 1 1.2 Objectives ...... 2 1.3 Considerations ...... 3 1.4 Personal Motivation ...... 4 1.5 Thesis Outline ...... 4

2 Statecharts Formalism 6 2.1 Introduction ...... 6 2.2 States ...... 8 2.3 Depth: Compound States ...... 9 2.4 Parallelism: Orthogonal States ...... 10 2.5 Formal Semantics ...... 12

3 Model Transformation 13 3.1 Introduction ...... 13 3.2 Features ...... 13 3.2.1 Transformation Rules ...... 13 3.2.2 Rule Application Scoping ...... 15 3.2.3 Source-Target Relationship ...... 15 3.2.4 Rule Application Strategy ...... 15 3.2.5 Rule Scheduling ...... 16 3.2.6 Rule Organisation ...... 17 3.2.7 Tracing ...... 17

ii 3.2.8 Directionality ...... 17 3.3 Categories ...... 17 3.3.1 Model-to-Code Transformation ...... 18 3.3.2 Model-to-Model Transformation ...... 18 3.4 In This Thesis ...... 19

4 Related Work 20 4.1 Modelling Artiﬁcial Intelligence ...... 20 4.1.1 Statecharts ...... 21 4.1.2 Decision Trees ...... 27 4.1.3 Visual Scripting ...... 27 4.1.4 Advanced Toolsets ...... 28 4.2 AI Model Transformation ...... 29 4.2.1 Procedural AI Generation ...... 30 4.2.2 Determining Good Transformations ...... 33

5 Statecharts as NPC AI 35 5.1 General Design for a Layered Architecture of Components ...... 35 5.1.1 Components ...... 36 5.1.2 Layers ...... 36 5.1.3 Communication ...... 39 5.1.4 Connection to a Player ...... 43 5.2 A Test Platform for Statechart AI ...... 43 5.2.1 Game Type: Tank Wars ...... 43 5.2.2 Creating the Test Platform ...... 44 5.2.3 Usage ...... 71 5.3 Creating NPC Behaviour Using Statecharts ...... 73 5.3.1 Determining the layers and their function ...... 73 5.3.2 Sensor components ...... 75 5.3.3 Analyser Components ...... 79 5.3.4 Memoriser Components ...... 81

iii 5.3.5 Strategic Decider Components ...... 83 5.3.6 Tactical Decider Components ...... 84 5.3.7 Executor Components ...... 90 5.3.8 Coordinator Components ...... 93 5.3.9 Actuator Components ...... 96

6 Transformation of Statechart AI 99 6.1 Parameter Transformation ...... 99 6.1.1 Implementation ...... 100 6.2 Component Structure Transformation ...... 101 6.2.1 Implementation ...... 102 6.3 Statecharts Model Transformation ...... 103 6.3.1 Implementation ...... 103 6.3.2 Example Transformation Rules ...... 104 6.4 Transformation Platform ...... 107 6.4.1 Goals ...... 107 6.4.2 Design ...... 107 6.4.3 Required Data ...... 109 6.4.4 Heuristics ...... 112 6.4.5 Usage ...... 113

7 Results 115 7.1 Platform Conﬁgurations ...... 115 7.1.1 Test Platform ...... 115 7.1.2 Transformation Platform ...... 116 7.2 Test Platform Results ...... 118 7.3 Transformation Platform Results ...... 124 7.4 Testing Generated NPC Behaviour Descriptions ...... 126 7.4.1 Interpreting These Results ...... 127

8 Conclusion 129

iv 8.1 Real World Applicability ...... 129 8.2 Behaviour Modelling Advantages ...... 129 8.3 Future Work ...... 130

Bibliography 132

v 1 Introduction

This introductory chapter serves to provide the reader with the required context to understand the problem addressed by this Master’s Thesis. Furthermore, a summary of various considerations that had to be made before work on the thesis could be started is provided, such as the choice of programming language, modelling formalism and modelling tools. The chapter ends with an outline of the thesis as a whole.

1.1 Problem Statement

In previous work, it has been shown that modelling the "Artificial Intelligence" (i.e. behaviour) of game characters is feasible. In [KDV07], a layered approach is formulated, which is expanded upon in [Bla08] to show that the Statecharts formalism is a natural fit for this purpose and that the (generic parts of) developed AI model can be easily re-used in various application domains, from a computer game to a real-life robot. Non-Player Characters (NPCs) in virtual environments or computer games exhibit autonomous and reactive behaviour that may seem intelligent and is hence often called Artificial Intelligence (AI). A popular and widespread technique is to have the NPCs be stateful. For example, an AI controlled tank in a Tank Wars type of game might have three possible states: idle, attacking and fleeing. The actual state of the tank will determine the actions the tank can take. This technique has been in use for a long time and has shown to be exceptionally useful to quickly develop interesting AI for games. However, the technique has not really evolved over the years. While computer graphics and gameplay have gone through big changes the last several decades, AI development for NPCs has not seen the same kind of change. This thesis proposes to leverage the findings of [KDV07] and [Bla08] to model the AI of NPCs completely in the Statecharts formalism, making use of all the features provided by the language to create a new paradigm for the modelling of timed, autonomous and reactive behaviour in NPCs. These models can then be used to analyse, simulate and ultimately synthesize code for NPC behaviour. Computer games that involve any kind of Non-Player Characters make use of AI to create a realistic and/or interesting game play experiences. Particularly in Massively Multiplayer Online Role Playing Games, players often encounter large groups of similar NPCs, usually powered by the exact same AI, but with different parameters. This causes players to lose their immersion in the interac- 1.2 Objectives 2 tive experience and start seeing it as a series of encounters that must be completed to continue the experience. It should be possible then, to create some AI for a particular type of NPC and have this AI transformed into several variants. Such variants would cause groups of NPCs to show different emergent behaviour, which differs on more than simply the amount of health, speed or other simple attributes. This heterogenity in NPC groups of the same type can be used to increase player immersion and simulation realism. It could potentially be used to tailor the game experience to the player, by transforming the original AI according to parameters retrieved from the player’s performance in previous phases of the game.

1.2 Objectives

The problem statement above highlights the current situation in AI development for NPCs and shows the areas which can be improved. To achieve the proposed functionality, several objectives were formulated that need to be fulﬁlled in order to achieve our goal.

Research Related Work First and foremost, related work in the ﬁelds of AI modelling, model transformation and NPC AI practices should be explored. Being aware of the current state of these ﬁelds is important to ensure the validity of this thesis, since it will build upon the previous work to tackle the previously mentioned problems.

Test Platform As the ﬁrst actual implementation task, a test platform should be developed that can be used to test the developed AI in a controlled environment. The platform should mirror the architecture of current commercial games closely, in order to face the same difﬁculties as we would in a commercial product, but also allowing us create AI that can later be used in such a commercial product.

AI Framework A framework should be developed to facilitate the modelling of AI and the compilation of the AI to the required programming language. The framework should be as seperate from the test platform in order to allow it to be used for various destination platforms.

AI Transformations Once it is possible to model AI, compile it and subsequently run the generated code on our test platform, transformation rules should be formulated to allow re-use of a single AI model, with properties as described in the problem statement above. The AI framework should be expanded to ease the execution of transformations on previously constructed models.

Analysis Platform Now that AI can be modelled and transformed, we should test how the transformations affect NPC behaviour. Since it would be less than desireable to look at the behaviour of a large group of AI variants manually, the test platform should be expanded to allow automated testing and 1.3 Considerations 3

analysis of the variants. Analysis should be performed through one or more interchangeable heuristics; at one point we might be looking for the single AI variant that is good at playing the game, but another time we might be interested in ﬁnding a collection of AI variants that provide a good game experience when pitted against a human player, which is not necessarily the same thing.

Commercial Product The AI framework developed over the course of this thesis should be expanded to allow us to test modelling of AI in an actual commercial product. This would show that this approach is not just academic in nature, but that it can actually be applied to existing products. The game chosen should be popular and advanced enough such that real-world application possibilities are presented to a wide audience. Note that this is outside the scope of this thesis.

1.3 Considerations

This thesis builds upon previous work in the ﬁels of AI modelling, transformation and current NPC AI practices. Because of this, the decisions that had to be made at the start of this thesis are heavily inﬂuenced by this body of previous work. Chapter 4 provides an overview of previous work, providing the reader with the required context.

Modelling Formalism Due to the previous work in [KDV07] and [Bla08], the choice of modelling formalism was obvious. This thesis will build upon the techniques discussed in these papers, modelling AI using the Statecharts formalism.

Modelling Environment Due to my association with the Modelling, Simulation and Design lab headed by my supervisor, Prof. Dr. Hans Vangheluwe, I have chosen to use the AToM3 tool to model all statecharts used throughout this thesis. The tool provides all functionality required to create models in the Statecharts formalism and perform transformations on these models. Furthermore, several other tools, such as the compiler used to compile statechart models into the required programming language (SCC), are integrated into AToM3, which eases the workﬂow proposed by the framework that will be developed throughout this thesis.

Programming Language Finally, I had to choose a programming language in which I would develop a test platform. There were various choices such as Java, C++ and Python. Since I already had quite some experience with Java and C++, I chose for the language I was least familiar with, namely Python. The language furthermore allows a developer to rapidly create a prototype application and test out designs and functionalities. Also, the third party PyGame library provides packages to facilitate the development of games, mirroring the architecture used in commercial games today. 1.4 Personal Motivation 4

1.4 Personal Motivation

Before starting my education at the University of Antwerp, I was very much intrigued by computer games and the ways in which they allow a story to be told interactively. This sparked my interest in the field of computer science and led me to start my current education. I have always felt computer games were not utilised to their full potential as a medium to convey experiences and stories. Because of their interactive nature, stories can become dynamic, changing in accordance with decisions made by a player. However, this kind of dynamic nature cannot be found in individual scenes or ’encounters’ in games. Encounters with friendly villagers or ferocious beasts quickly become stale and uninteresting when such NPCs always show exact same behaviour. For example, in the real world, a deer hunter might have to adjust his hunting strategy for each individual animal he is hunting, because some deer are more shy than others or have varying types of reactions to an unknown smell in their vicinity. This type of variation is thus far unavailable in computer games, causing players to quickly deduce the behaviour patterns for the various NPC types they encounter, rendering the experience boring and stale. I feel that through AI modelling and transformation, NPC AI can be varied to emulate the varying behaviour of individuals in the real world. This is but one application of the technologies discussed in the coming chapters, but it is one which I was thinking about long before starting my work on this thesis. Of course, apart from the opportunity to develop something new which the world has never before seen, I also have more selfish motivations. I feel this thesis is an excellent opportunity to work on a big project from start to finish, learning in detail how each stage of software development and research is performed. Previous courses at the university usually let us work on projects for a maximum of two months, much less time than real world software projects. I also look forward to exploring the topics of Artificial Intelligence, Modelling and Model Transformation in detail. I only got in touch with modelling in my second year at university, but it immediately caught my attention. I was thus far unaware of this new paradigm in computer programming and exploring it has really opened my eyes to new perspectives on software development, perspectives I hope to expand even further through my work on this thesis.

1.5 Thesis Outline

Chapter 2 will begin this thesis with a general introduction of the Statecharts modelling formalism used throughout this thesis. Afterwards in chapter 3, we provide an introduction to model transformation. After these concepts have been explored, the thesis closely follows the structure proposed by the objectives mentioned above. Chapter 4 explores the related work in the areas of AI modelling, transformation and current NPC AI practices. This related work serves as the foundation for the new developments introcuded throughout this thesis. Chapter 5 will chronicle the development of the behaviour architecture and testing framework and chapter 6 serves the same function for the transformation submodule of the framework. Finally, chapter 7 explores the results of the created testing and transformation platform. In the conclusion, we provide arguments for the real world applicability of this thesis and explore how further work in the ﬁeld of AI modelling and transformation may open 1.5 Thesis Outline 5 up even more ways to improve NPC behaviours. 2 Statecharts Formalism

Before we can model Artificial Intelligence, we first need to obtain an understanding of the modeling formalism we will use to achieve this goal. In this chapter, an introduction the syntax and semantics of the Statecharts formalism is provided. For detailed information regarding the concepts discussed within this chapter, refer to the cited sources. As was mentioned in the introductory chapter, this thesis builds upon previously published work. With regards to the modeling of AI using Statecharts, the most recent research upon this thesis is based was published in [Bla08], which itself was based on work published in [KDV07]. An overview of the concepts defined in work related to this thesis can be found in chapter 4.

2.1 Introduction

David Harel introduced the Statecharts formalism to the world in his publication Statecharts: A Visual Formalism For Complex Systems [Har87]. His motivation was the fact that, at the time, no suitable method had been found to describe large and complex reactive systems. Harel specifically created the Statecharts formalism to address this problem. Reactive systems, as opposed to transformational systems, specify an interaction between themselves and their environment. This inherently means they are event-based, continually reacting to external or internal stimuli. Some examples of reactive systems are telephones, cars, traffic control systems, and so on. Even the lights in our homes can be thought of as a reactive system, responding to button presses and the flow of electricity. Before the introduction of Statecharts, several methods existed to specify reactive systems, but they were generally accepted to only be applicable to relatively simple problems. Traditional state diagrams, such as the one seen in figure 2.1, are inadequate for describing complex reactive systems. These diagrams are simply directional graphs. Nodes in the graph denote states, whereas arrows – labeled with the triggering event and optional guarding condition – denote transitions between these states. Referring to figure 2.1, the states are A, B and C, while the transitions are the arrows between them. Furthermore, the labels α, β and γ denote events, while P denotes a certain condition. Such state diagrams contain no notion of depth (or state hierarchy) or parallelism (or orthogonality), which would cause a very large build up in the amount of states required to describe a system. 2.1 Introduction 7

A α

δ C γ(P) β

B α

Figure 2.1: A State Diagram

Furthermore, there is no way for one state to inﬂuence other states in the same system, because there is no construct for the broadcasting of events (in other words, the system cannot react to or even generate internal stimuli). Due to these shortcomings, Harel set out to create a visual formalism that would solve these issues, while at the same time allowing for visual inspection of the system. Harel summarizes this in [Har87] with the following ’equation’:

Statecharts =State Diagrams + Depth + Orthogonality + Broadcast communication

Harel only provided a brief description of how one could implement the semantics of Statecharts in his original paper. In one of his later works, The STATEMATE Semantics of Statecharts [HN96], he expands upon this to provide a complete formal semantics for the language. This was in fact the ﬁrst attempt at formalizing the semantics of the Statecharts formalism, and it has since been adopted by various modeling tools. Throughout this thesis, we will refer to the STATEMATE semantics to deﬁne the formal behavior of the models that are developed. The Statecharts formalism’s syntax is based upon traditional state diagrams and the concept of a higraph, which allows for the expression of state hierarchies. In what follows, we shall go over the various syntactic constructs provided by the Statecharts formalism. Keep in mind that the exact representation of states might differ between editors, some utilize rectangles where others use circles. For the most part however, these differences are limited to trivial details that have no bearing on the actual models that can be developed. We will use the syntax as proposed in [Har87] to illustrate the various concepts. As a special note, we will end this section by taking a look at how AToM3 implements the Statecharts syntax. 2.2 States 8

2.2 States

Single states are represented by labeled rectangles, as seen in ﬁgure 2.2. Here we see the model of a trivial system, consisting of only one state A. The unlabeled transition arriving at A signiﬁes that this is the initial state of the system. Since no other transitions are present, the system cannot change states.

Figure 2.2: Single Statecharts state

Figure 2.3 shows a system consisting of several states, with transitions between them labeled in the form Event(condition)/Action. The system consists of states A, B and C, where A is the initial state of the system. State A has one outgoing transition to state C, with the label α(Q)/δ, which signiﬁes this transition will be taken when an event α occurs, but only if condition Q evaluates as true. The action, δ signiﬁes an event that should be broadcast when this transition is taken. This event could in turn form the label of another transition in the system. Note that all parts of the label – the event, condition and action – are optional, and no transition besides the one just discussed includes all these components. It is thus also possible for a transition to exist without any label, signifying an instantaneous transition between states.

A α(Q)/δ

α C

γ(P) β/δ B

Figure 2.3: Multiple Statecharts states with labeled transitions 2.3 Depth: Compound States 9

2.3 Depth: Compound States

With the constructs discussed above, we can already start describing simple reactive systems, but no notion of state hierarchy is yet available. We should be able to cluster states together into compound states or reﬁne states into multiple substates. This clustering used for reﬁnement is made possible through XOR decomposition of states, meaning that when a compound state is active, exactly one of its substates must be active.

A B α

β γ(P) D

C δ

(a)

B α

β γ(P) D α β A β D C δ δ (b) (c)

Figure 2.4: State hierarchy

Figure 2.4a shows a system consisting of four states. State A is a compound state, consisting of the substates B and C. We see that state A is the initial state of the system and B is the initial state that will become active as soon as state A becomes active. Each level of hierarchy needs to specify an initial state, or the whole system would be underdeﬁned. We can furthermore see that transitions may depart from any level of the hierarchy and arrive at any other. Figure 2.4b is the same system, this time modeled without making use of compound states, causing the need for two transitions triggered 2.4 Parallelism: Orthogonal States 10 by event β instead of one departing from the compound state. In essence, we can arrive at (a) by clustering the states B and C in (b). Figure 2.4c on the other hand models a less detailed system. By reﬁning state A from (c) into the two substates B and C, we reached (a).

2.4 Parallelism: Orthogonal States

Now that we know how to structure the system’s model into state hierarchies, let’s take a look at how the Statecharts formalism provides us with the ability to model parallelism, in Statecharts terms referred to as orthogonality. Where XOR decomposition allowed one substate of a compound state to be active at any time, orthogonality is realized through AND decomposition into orthogonal states. This effectively signiﬁes that if an orthogonal state is active, all of its substates must be active too.

Y A D B E α

γ δ β/γ F (in F) α

C α G

Figure 2.5: System consisting of one orthogonal state

Figure 2.5 shows a system modeled using a single orthogonal state Y, which contains two substates A and D. Since Y is the initial state of the system (at the highest level in the state hierarchy), both A and D become active as soon as the system initiates. The A and D substates of Y are themselves compound states, consisting of 2, respectively 3 substates themselves.

Orthogonal states is the mechanism through which the Statecharts formalism avoids the infamous ’state blowup’ problem, where the models of complex systems would contain so many states and transitions that the complete model loses some of its most desirable properties: human readability and independent components. The system modeled in figure 2.5 could also be modeled without an orthogonal state, by flattening the model. The result of this can be seen in figure 2.6. Even for this 2.4 Parallelism: Orthogonal States 11 very simple model, the flattened version is a lot less readable. If we would name the states arbitrarily (as opposed to using the names of the original states), we would be hard pressed to find the different components in the system, while the model in figure 2.5 inherently shows us how the system is organized in the two components A and D.

α α α

BxE BxF BxG

β/γ P = (in BxF) OR (in CxF)

γ δ(P) γ δ(P) γ δ(P) β/γ

CxE CxF CxG

α α

Figure 2.6: Same system, modeled without orthogonal states by ﬂattening the original model

Apart from more obvious advantages such as readability and independent components, we can now also use special conditions in our models that check if the system is in some specified state. Refer to the model in figure 2.5, the transition from state C to B: in F, specifies that the transition can only be taken if state F is currently active. Note that such a condition was impossible to formulate before the introduction of orthogonal states! Furthermore, the transition from state F to state E contains an action: event γ is to be broadcast when this transition is taken. In the next step (see [HN96]) after this transition, event γ can be ’sensed’ by other states and/or transitions in this system. In fact, if state B is active and the transition from state F to E is taken, in the next step the transition from B to C will be triggered by the broadcasted γ event. 2.5 Formal Semantics 12

2.5 Formal Semantics

Since Statecharts models define the behavior of a system, we must also define some way to execute this behavior. In other words, we want to simulate these models. Reliable simulation can only be performed when the syntactic constructs discussed in the previous chapters are associated with formal semantics. Furthermore, since the formalism itself contains no notion of time, we should define the concept of time as it relates to the simulation of Statecharts models. There are two distinct methods to defining time, the first approach is discrete time, in which simulation occurs in steps, each step advancing the current time by one time unit. Formal semantics will then define what behavior of the model will be executed during each step, based on the state of the system at the start of each time step. Some of the questions that should be answered by a formal semantics are:

• Should transition actions be performed in the same step as the one in which the transition is taken, or in the next? Note that so-called micro-steps exist within a single time-step. • In a step in which a transition is taken between source state x and destination state y: Which of the special conditions in X and in Y should evaluate as true? Maybe they should both evaluate as true, or perhaps as false?

The second approach to deﬁning time takes the form of continuous time. Time advances normally during actions and simulation of the model, which corresponds to the natural notion of time as we know it. In this case, the formal semantics must deﬁne how the orthogonal components of the system can be simulated, such that the parallel and independent properties of Statecharts are preserved.

2.5.0.1 STATEMATE Semantics

As was previously mentioned, throughout this thesis we will refer to the formal semantics proposed in [HN96], in the following referred to as the Statemate semantics. Since the goal of this thesis is to define the behavior of entities in a virtual world in which time advances continuously, we will have to find a way to translate this continuous time into the time steps defined in the semantics. 3 Model Transformation

Model transformation consists of the creation and application of transformation rules which transform their model input to some output. This chapter will serve as an introduction to model transformation itself and how it will be applied in this thesis.

3.1 Introduction

Model transformation is a very important area of research in the field of model-driven engineering. It is a tool through which efficiency of software development can be greatly improved by automated transformation of input models. There are many forms such transformations may take and an effort to classify the various types of possible model transformations was made by K. Czarnecki and S. Helsen in Classification of Model Transformation Approaches (2003) [CH03]. This work clearly shows that model transformation consists of much more than simply defining transformation rules. This can be clearly seen in figure 3.1, representing the various properties of model transformations. In what follows we will go over these properties and find out how they are represented in this thesis.

3.2 Features

3.2.1 Transformation Rules

When dealing with model transformation, one creates transformation rules to describe the required transformations. These rules deﬁne how a model can be transformed into some output. This output is not necessarily another model, it could also be some textual representation of the original input. Transformation rules consist of a left-hand side and a right-hand side (further referred to as the LHS and RHS respectively). Both LHS and RHS may be formulated in terms of variables, patterns and/or logic.

Variables Variables representing meta-data about the (source and/or target) models may be used in the transformation rule to inﬂuence the actual transformation. Patterns 3.2 Features 14

Source-Target Relationship

Rule Application Rule Application Scoping Strategy

Model Transformation Transformation Rules Rule Scheduling

Directionality Rule Organisation Tracing

Figure 3.1: Model Transformation Taxonomy

Patterns are in fact model fragments, with zero or more variables. They can take various forms, such as strings, terms or graphs. In AToM3 and most model-to-model (see 3.3.2) methods, graph- based patterns are used as a visual method to construct transformation rules. Patterns may be represented using the abstract or concrete syntax of the model languages used and this syntax may be either textual or graphical in nature or some combination thereof.

Logic Logic can express computations or constraints on model elements used in the LHS and RHS of the transformation rule. This logic can be non-executable when it is used to specify relation- ships between models, but it can also be executable when it is used to retrieve elements from the source model or actually generate model elements when the transformation rule itself is executed.

Transformation rules may syntactically separate the LHS and RHS, but this is not necessarily the case, e.g., when the transformation is executed in-place. Furthermore, transformation rules may be parameterised to alter or ﬁne-tune the rules at the time of their execution. Figure 3.2 shows a very simple transformation rule. In this case, both LHS and RHS are explicitly separated and contain models in the Statecharts formalism. The models seen in the LHS and RHS are patterns in the terminology used above, represented in graph form. This very simple rule would connect two unconnected states in a Statecharts model using a transition which ﬁres on the event A. 3.2 Features 15

LHS RHS

A α A α B B α

C α C α

Figure 3.2: Statecharts Transformation rule

3.2.2 Rule Application Scoping

Rule application scoping allow us to specify which parts of the model will participate in the transformation. The application scope can be a whole model, but also some subset of the whole model upon which the transformation will be performed. In graph-based transformation rules, application scoping can take the form of a Negative Appli- cation Condition (further referred to as NAC). To continue with the example used in the previous section, figure 3.3 shows the transition rule from figure 3.2 augmented with a NAC. The NAC in this example specifies that the transformation can not be executed when the two states occurring in the LHS are contained in a compound state.

3.2.3 Source-Target Relationship

The source-target relationship in a model transformation can take several forms. In the simplest case, the target is a new, previously non-existing model. It’s also possible that the target is a pre-existing model and in this case we can differentiate between two scenarios: Firstly, the target might be the source model when the transformation is executed in-place. Secondly, the target might be a different, separate model, in which case the transformation serves to update this pre-existing model.

3.2.4 Rule Application Strategy

When transformation rules are applied to their source models, there may be more than one location in the model where the rule is applicable. In such cases, we need a rule application strategy to specify the where and/or in what order the rule should be applied on the source model. 3.2 Features 16

NAC LHS RHS

A α A α A α

β B B B α

C α C α C α

Figure 3.3: Statecharts Transformation rule with NAC

A rule application strategy can be deterministic, non-deterministic and interactive. Deterministic strategies might follow some well-deﬁned traversal strategy over the source model to iteratively apply the rule to all applicable locations. Non-deterministic approaches on the other hand might randomly select one or more location(s) to apply the rule. Finally, the interactive strategy allows the user to explicitly select where to apply transformation rules, based on manual inspection of the source model.

3.2.5 Rule Scheduling

When a set of transformation rules is available, we need a mechanism to determine the order in which these rules are selected for execution. Rule scheduling mechanisms are responsible for determining the order in which individual rules may be applied. [CH03] speciﬁes four areas in which such scheduling algorithms may differ:

Form Rule scheduling can be made implicit or explicit to the user. In the implicit case, the user has no direct control over the scheduling algorithm in use. The only way to inﬂuence implicit scheduling algorithms is by designing the transformation rules such that a strict order is imposed. On the other hand, explicit scheduling offers the user constructs to control the scheduling of rule execution.

Rule selection The selection of rules to execute can be performed by some explicit condition or a non-deterministic (random) choice. In both cases, priorities can be associated with rules to enact conﬂict resolution and/or provide a simple mechanism for ordering rule execution.

Rule iteration The way in which rule scheduling algorithms may iterate over the transformation rules may differ. Current approaches use either recursion, looping or ﬁxpoint iteration. The latter approach consists of executing rules until no more changes to the target model are detected. 3.3 Categories 17

Phasing Some scheduling algorithms allow the user to organise the transformation process into several phases. Each phase is then associated with a set of transformation rules that may be executed during that phase. For example, a user might create some transformation rules for a pre-processing phase of the transformation and other rules to perform the actually desired transformations.

3.2.6 Rule Organisation

Model transformation tools may offer the user the ability to organise the developed transformation rules in some useful way. Just as software systems are organised into logical packages, it is useful to organise transformation rules in a similar way. Mechanisms may be provided to allow a user to package rules into modules. This organisational system may follow the conventions placed forth by the source or target formalisms, but it can also be a completely independent, arbitrary system.

3.2.7 Tracing

Transformation rules can specify links between the source and target models. These links can serve various purposes, the most obvious of which is for identiﬁcation of model elements between source and target models, such that the transformation is actually applied on the desired model elements from the source model. Traceability links may also serve other purposes, they might for instance allow the user to analyse the impact of a transformation on the model at large, be used for debugging purposes, and so forth.

3.2.8 Directionality

Transformation rules can either be unidirectional or bidirectional. Unidirectional transformations are executable in one direction only, source models are transformed into target models. Bidirectional transformations however, can be executed in either direction. The importance of bidirectional transformation lies mostly in synchronisation of models.

3.3 Categories

We have now discussed the various features of model transformation and how these features are the ways in which these currently expressed in model transformation tools. We have cleverly avoided talking about the categories or approaches to transformations that are possible, although we hinted at this in the examples using graph-based transformation rules. [CH03] deﬁnes two major categories for model transformations: model-to-code and model-to- model transformations. As mentioned in this work, model-to-code approaches are in fact model-to- model approaches where the target meta-model is a programming language and the models are valid programs with respect to that programming language. However, these approaches have some funda- 3.3 Categories 18 mental differences in how the transformation rules may be constructed, so it is useful to look at these approaches as two categories, rather than bunching them together without regard for these differences.

3.3.1 Model-to-Code Transformation

Model-to-code methods, often simply referred to as code generation, are used in various scenarios, e.g., to serialise the model to a file on a storage medium, to generate a simulator for the system specified by the model, and so forth. There are two approaches to code generation: visitor-based and template-based. We will use model-to-code transformations to generate the code defined by the Statecharts models created throughout the thesis.

Visitor-Based Approach

The simplest method for model-to-code generation is the visitor mechanism. Here, the internal representation of the source model is traversed and code is written to a text stream, according to the transformation rule deﬁned for the various model elements. This approach is quite similar to the design of compilers in traditional programming languages.

Template-Based Approach

Template-based approaches usually consist of a document in the target language, which is interspersed with metacode referencing the source model in order to generate the required target code. This approach will use executable logic in the LHS of the transformation rules to access the source model and string patterns in the RHS to select code to execute in that location and to perform model expansion. For a more in depth look at template-based code generation methods, refer to [CC01]. Templates greatly resemble the target code to be generated, since the templates themselves are actually ’incomplete’ versions of the required target code (it still contains metacode that must be sub- stituted during rule execution). On the other hand, the visitor-based approach is much more declarative and shows little resemblance to the target code. This is not necessarily a positive or negative aspect of either approach, but could be a desired property in some circumstances.

3.3.2 Model-to-Model Transformation

Model-to-model transformation methods concern techniques used to translate a source model to a target model. These models may or may not belong to the same meta-model. There are various model- to-model transformation methods, we will discuss some of the most prominent approaches here, such as: direct-manipulation, relational, graph-transformation, structure-driven and hybrid approaches.

Direct-Manipulation Approach

Direct-manipulation approaches provide the user with some API to manipulate the internal representation of models. The API usually takes the form of an object-oriented framework, which may or may not provide the user with some mechanism to structure transformation rules into logical sets. 3.4 In This Thesis 19

Relational Approach

Relational approaches to model-to-model transformation can take various forms, but they are always declarative, with the main idea being that mathematical relations are used to specify the transformation from source to target model.

Graph-Transformation Approach

Graph-transformation approaches operate on typed, attributed and labeled graphs. This is also the method with which model transformations are used in the AToM3 tool, where the collection of transformation rules are referred to as graph grammars. The general notion is that transformations consist of graph patterns on the LHS and RHS of the transformation rule, which may or may not be rendered in the concrete syntax of the meta-model(s). The LHS pattern is matched in the source model and transformed into or replaced by the RHS pattern in-place. In order to provide more control over the matching algorithm, the LHS can be augmented with NAC(s) as was mentioned in the above section on rule application scoping.

Structure-Driven Approach

These approaches typically work in two phases. First, the overall structure of the target model is generated. The second phase then consists of setting all attributes and other properties of the model elements generated in the ﬁrst phase.

Hybrid Approach

Finally, model transformation mechanisms exist that do not conform to only one approach category. Such transformations use a combination of the above approaches. Examples of such approaches are the Transformation Rule Language proposed in [Omg08] and the Atlas Transformation Language ﬁrst introduced in [BDJ+03]. Hypbrid approaches are not unusually the consequence of a framework that supports various approaches, in order to allow the user the freedom to choose the approach most applicable to a speciﬁc problem.

3.4 In This Thesis

As was previously mentioned, model-to-code transformation shall be leveraged when we compile Statecharts AI models to executable code for use in our test platform. In that case, we will use the pre- existing SCC compiler to perform the actual code generation. Model-to-model transformation shall return in the chapter 6, when we construct transformation rules to alter previously created AI models. We will attempt to alter the AI models in an effort to improve their performance with regards to some heuristic(s) and construct a library of model-to-model transformation rules to that effect. Finally, an automated system to apply these transformations will be created, with the goal of allowing model transformation to occur semi-automatically, based on preferences stipulated by the user. 4 Related Work

The work presented in this thesis is for a large part based on previous research and related work. Exploring and understanding this related work has been a large part of the effort to create this thesis; this chapter provides an overview of the work that has served as the basis for this thesis’ efforts. Since presenting all details of the related work would deter from the actual goals of this thesis, references to the mentioned works will be placed throughout this paper to aid the reader. The related work used in this thesis can be divided into two research areas: research towards methods for modelling AI and towards methods for transforming models, more speciﬁcally AI models. As such, this chapter is divided into two sections corresponding to these related but different areas of research.

4.1 Modelling Artiﬁcial Intelligence

Artificial Intelligence for non-player characters is traditionally developed using a textual programming (or scripting) language. Historically, creating AI for games was rather simple. Sets of rules would be formulated which dictated the actions NPCs were to take if some conditions were met. As the virtual world wherein these NPCs lived became much more complex and dynamic, AI programming needed to evolve. As an illustration of this problem we can take a look at the subproblem of writing pathfinding AI. Writing AI for pathfinding in a static 2D grid-based world is relatively straightforward. Writing the same pathfinding AI for a 3D polygon-based world in which obstacles move about constantly is a much bigger challenge. It is becoming much more difficult to create believ- able AI than before, because the platforms for which this AI must be written are drastically increasing in complexity. In large part due to the increasing complexity of virtual worlds, there has been a surge of interest towards modelling AI1. Non-player characters are prime examples of reactive systems: They exist solely to react and interact with a player. Current methods for programming NPC AI contain notions of state to represent the current condition of characters, e.g., Idle, Fleeing, Busy, ... It seems then that the use of Statecharts to describe the character’s behaviour is an interesting area of research. It should come as no surprise that research into this has already been performed. There are alternatives to be

1The reasons for this increasing interest is more complex however, as code readability, maintenance, reuse and efficiency all play a big role in this evolution towards modelling rather than programming. 4.1 Modelling Artificial Intelligence 21 considered however, which we will also discuss in the following sections. Let’s first take a look at the efforts made towards leveraging Statecharts as a formalism for modelling NPC AI.

4.1.1 Statecharts

Using Statecharts as a formalism to model NPC AI is a relatively new evolution in the game AI world. As such, this practice has so far been largely confined to the world of academic research and no commercial products currently exist that use Statecharts to describe game character behaviour. Most research in this area has been performed at the McGill University in Montreal, Canada under the supervision of this thesis’ promotor, Professor Hans Vangheluwe. The very first attempt at using finite state machines for the description of game character behaviour can be found in an online publication on Gamasutra, named "Visual Finite State Machine AI Systems" [Gil04]. The solution proposes a work flow using the general purpose graphing tool Microsoft Visio to draw finite state machines, which are subsequently exported to XML and parsed by a C++ application to create the actual AI. Note that the state machines are not compiled to a programming language directly and no optimisation of the resulting code is performed. This raises certain questions concerning scalability and applicability to a wider range of game engine architectures. Further more, the state machines used have no formally defined semantics. The solution works in one application area but there is no guarantee on the reliability of the observed behaviour. The first time that the actual Statecharts formalism was leveraged to describe NPC behaviour can be found in the 2006 publication "Model-based Design of Game AI" [DKV06]. In this paper, a method is proposed to allow Statecharts modelling of game character behaviour using the DCharts formalism developed for the AToM3 tool created by Thomas Feng [Fen04]. This paper uses the problem of describing the behaviour of a tank in a 2D world to construct a layered approach to describing NPC behaviour. The basic principle whereupon this paper is based can be seen in figure 4.1. On the left hand side we see models responsible for sensing the environment, followed by models that analyze this information. On the right hand side we find models responsible for performing actions, preceded by models that plan how these actions should be performed. Finally, the middle layer is stated as the actual AI models, these models describe on a high level the behaviour of a tank in a simple tank wars style game. The work discussed above served as the precursor to [KDV07], in which a much more thorough account is given of the same approach. We will come back to this improved architecture in section 4.1.1.1. Let’s first take a look at the reasoning behind choosing the Statecharts architecture for the description of a character’s behaviour. One year after the publication of the above works, in [Bla08], the previously proposed methods are expanded upon even further. This master’s thesis was written with the intent to define a way to implement the behaviour of an actual robot using a modelling formalism. It quickly concludes that statecharts is a perfect fit, since many of the formalism’s constructs lend itself well to implementing more complex behaviours. It stipulates a couple of requirements for a formalism that is to be used in the context of describing the behaviour of a character in a virtual world. 4.1 Modelling Artificial Intelligence 22

Figure 4.1: First proposed Statecharts AI architecture

• State/Event based: Our formalism must have a notion of the state of a component and allow for events to be received and broadcast. • Autonomous/reactive behaviour: Components must react to both internal and external events. Events trigger transitions in state, and in turn transitions might broadcast new events to other components. • Modularity: Ease of assembly and reusability of components is highly desireable. The chosen formalism must allow us to reuse components in a different context and provide mechanisms to model our components in a modular fashion. • Notion of time: The current time must retain its meaning in the modelling formalism. The time might be used in the decision to transition from one state of our model to another. • Well known: It is highly desirable for the modelling formalism to have a wide availability of educational material and supporting tools.

So far, both Petri Nets and Statecharts fulﬁll these requirements and are appropriate formalisms to consider when modelling a reactive system. However, the Statecharts formalism provides us with some extra functionality that aids in modelling behaviour speciﬁcally.

• Composition: XOR (exclusive-or) decomposition of states, which captures the property that, being in a state, the system must be in only one of it’s composite components. • Orthogonality: AND decomposition of states, which captures the property that, being in a state, the system must be in all of its orthogonal components. 4.1 Modelling Artiﬁcial Intelligence 23

• History: In the Statecharts formalism syntax, entering a history state is tantamount to entering the most recently visited state.

These constructs greatly aid us in developing behaviour in a modular fashion, enabling component reuse without the need to redesign our implementation. Composition allows us to easily define different behaviour for different conditions, while orthogonality gives us easy access to concurrency, as it might be used in e.g., tracking the state of a lot of different sensors. While [Bla08] admits there was no real option to choose a modelling formalism (due to the author’s recent work in a course at McGill University), we posit that there was no better option than Statecharts. NPCs in games are reactive systems: they respond to the player, other NPCs and their environment, which makes Statecharts – as a formalism designed to be used to describe reactive systems – a natural fit. Furthermore AI for non-player characters is currently often implemented as having a certain state and events are almost always used as the main source of information input. Since these are both syntactic constructs in the Statecharts formalism, we can make use of research already performed on programming AI for non-player characters, without translating the findings to the syntactic constructs of another formalism. While the use of states and events in a textual programming language might not completely overlap with its implementation in Statecharts, the comparison can be made and the use of Statecharts can be seen as an evolution of what is currently accepted as best practice in the field of game AI.

4.1.1.1 Model Architecture

In [KDV07], an architecture for modelling reactive behaviour is ﬁrst proposed, this architecture is used and somewhat expanded upon in [Bla08]. As previously mentioned, the architecture is formulated in terms of a Tank Wars style game, much like the game used as a test platform in this thesis. Later we will expand upon this work and introduce model transformation as a way to achieve different behaviour from the same source model(s). The proposed architecture is a layered model, where events propagate along the layers, input arrives at one end of the layered stack and output is generated at the other end. The complete architecture is described in Figure 4.2. Note that components towards the ends of the layered architecture – sensors and actuators have the lowest level of abstraction, they correspond closely to the actual components of the character for which the AI is written. Let’s take a look at the various layers and their responsibilities.

Sensors

Sensors are components of the character that deal with keeping track of the current state of the character itself and the character’s world. This means we can distinguish between two types of sensors: internal and external sensors. Internal sensors keep the internal state of the character up to date, they concern themselves with e.g., fuel levels, health, energy, ... External sensors on the other hand sample the world to try and keep track of what is happening around the character. Examples of external sensors would be: eyes (or any other system enabling vision such as radar or cameras), thermometers, altimeters, ... 4.1 Modelling Artiﬁcial Intelligence 24

Input Events

Sensors Low High

Analysers Abstraction Levelof

Memorisers

Strategic deciders

Tactical deciders

Executors

Event Event Propagation Coordinators

Actuators Output Events

Figure 4.2: Layered AI architecture

Each sensor component gets information through (internal or external) events and polling of the game world. This allows sensors to provide an abstract view of the actual state of the character and/or world through the generation of events. In our running tank wars example, the tank behaviour should probably not be inﬂuenced by the exact fuel level, but we would like to trigger some kind of behaviour when the fuel level is getting low and/or critical.

Analysers

Not all information needed to describe a character’s behaviour can be derived from individual sensors. Sometimes a correlation between various inputs is needed to comprehend the current situation. For example, to determine if some enemy is in range of a character, we need to correlate the information from at least two different inputs. First, we need to ﬁnd the range of the enemy and secondly we need to correlate this with the range of our character (this usually refers to the range of a weapon carried by the character). Analysers will correlate these inputs and generate appropriate events for use in the subsequent layers.

Memorisers

Natural behaviour is not only based on the current state of a system, but is also for a large part based on previous events or states. This implies a need to model the memory of a character. Memorisers fulfil this role, allowing us to store important information to reuse at a later time. This information can be stored programmatically for easy access in conditions, but the memory can also be explicitly modelled using Statecharts. In the tank wars example, we could model a system that remembers the 4.1 Modelling Artificial Intelligence 25 last known position of an enemy when this enemy is no longer visible. This information could then be used to navigate towards this location in an effort to find the lost enemy.

Strategic Deciders

Sensors, analysers and memorisers take care of event generation based on the environment, character state and memory. Using this information, the character will plan to achieve certain goals. This is done by the strategic deciders. The goals decided upon by strategic deciders should be high level goals. They will therefore correspond to the ’state’ values we often see in current commercial games, e.g., Idle, Attacking, Fleeing, ...

Tactical Deciders

High level goals decided upon by strategic deciders must be translated into lower level commands to the various actuators. This translation is not trivial however, since it might require some complex tactical plans to be made. To facilitate this, the tactical deciders layer was introduced. A good example of a tactical decider component is pathfinding. In a certain situation the strategic deciders might decide that fleeing the scene is the best course of action. Without any planning step dealing with how to achieve the ’fleeing’ goal, a character might be able to move away from danger, but this move might bring the character face-first into a wall or into the arms of another enemy. It is clear that more planning is needed. In this case, a tactical decider ’pathfinding’ might exist that can decide waypoints to follow when fleeing, avoiding dangers and obstacles.

Executors

Executors take on the role of mapping the decisions made by tactical deciders to events that can be processed by the actuators. The actual mapping will depend on the facilities offered by the game or simulation platform in question, but typically one executor is used for every available actuator. Returning to the tank wars example, when a tank is ﬂeeing and the tactical deciders have decided upon a route to follow, executors will translate the route to events directed at the engine and steering actuators in order to move towards the current waypoint.

Coordinators

In some cases, decisions made in previous layers will result in incorrect or opposed instructions sent to the actuators. For example, assume a tank is moving forwards with its turret pointed forward as well. If the tank now spots an enemy approaching from its right hand side, it is clear that in order to attack this enemy, the turret must be turned to the right. However, if the tank reaches a waypoint at the same time as the other tank is spotted, and the next waypoint is towards the south-east relative to the tank, the optimal turn for the turret to take in order to take aim at the enemy is no longer a turn to the right, but rather a turn to the left. These cases can be detected and solved by coordinators. 4.1 Modelling Artiﬁcial Intelligence 26

Actuators

Actuators perform the actual commands that are the result of decision made by previous layers. Events taken as input by actuators should be relatively simple commands, such as speedup, turnleft, ﬁreturret, etc.

4.1.1.2 Mapping To A Platform

The model architecture explained above results in a complete description of the behaviour of a non- player character. Since we are using the Statecharts platform, the architecture is completely event- based, meaning that time is regarded as being continuous and events can occur at any time. Modelling the behaviour in continuous time has several advantages according to [KDV07]:

Model freedom The modeller is not constrained by implementation issues and can focus on the required behaviour logic.

Symbolic analysis The model can be analysed using timed logic to prove properties.

Simulation Simulation can occur with inﬁnite accuracy in simulation environments.

Reuse Continuous time is the most general formalism possible and can thus be used in any simulation environment.

When the Statecharts AI models we create are used in some environment, actual code must be generated for execution in that specific environment. When using AI models on event-based platforms, there is no issue. We can use the models as they are, feeding input events to the sensor layer and processing events generated by the actuator layer. The situation becomes less obvious when the destination platform is time-sliced, as is the case in most game engines. When the platform is ’time- sliced’, we mean that time does not advance continuously but in small, discrete steps of e.g., 50 ms. For such platforms, we will need to bridge the event-based – time-sliced gap. To enable event-based reasoning in a time-sliced environment, we will need to perform some logic to bridge these two paradigms. [KDV07] proposes the use of a function void AI(tankAIInput, tankAIOutput), in which the variable tankAIInput is some data structure containing the state of the tank at time of the function call and tankAIOutput contains the state of the tank after the model logic has been executed. This function then proceeds by copying the appropriate values to the tank sensors and subsequently evaluating the guards of transitions, firing their corresponding events if applicable. Both [KDV07] and [Bla08] propose mapping to events at the sensors layer. From the sensor layer onwards, all events are propagated and actions are triggered within the Statecharts formalism. The 4.1 Modelling Artificial Intelligence 27 last step is then copying the appropriate values from the actuator layer back into the tankAIOutput variable. It is important to note that both [KDV07] and [Bla08] approach this mapping from the viewpoint of the EA Tank Wars [Art05] platform. This platform was used in the context of a competition where students would create AI for tanks and let their creations compete against each other. This implies that the platform was heavily constrained, in the sense that the AI had no direct control over the various components of the tank, but the state had to be communicated through a software layer that checked against cheating in some way. In this thesis we explore how we can use Statecharts to model AI in any game platform, which implies the AI has the ability to directly control various components of the characters. We will explore how to exercise this control in chapter 5.

4.1.2 Decision Trees

Decision trees and algorithms for their induction as described in [Qui86] have been used for creating AI in games for a while and the approach is still gaining in popularity. They provide a visual system to specify how decisions should be taken by non-player characters. A simple example of such a decision tree can be seen in figure 4.3. In this example, a decision tree is used to formalise the decision whether or not to flee from enemies. If the health value is above 20%, the decision is not to flee and keep engaging any enemies. If the health value is below 20%, we first check to see if there is a first aid kit nearby. If there is no first aid kit in the vicinity, we decide to flee. However, if there is such an item nearby, we flee enemies while moving towards this first aid kit. Decisions taken as a consequence of following nodes on the decision tree are always leaf nodes. While decision trees do indeed provide a visual formalism to model some key aspects of non- player character behaviour – much like the Statecharts formalism – the approach is effectively no different than a nested if-construct in a textual programming language. It offers no real benefits other than the ability to visually edit behaviour and as a consequence, the approach is equivalent to using a textual programming language to specify which behaviour should occur under which circumstances. Our approach on the other hand does not concern itself with which behaviour needs to be observed, but rather with the consequences of external and internal stimuli on the internal state of the character, which in turn results in some actions being taken. Our approach thus lends itself well to emerging behaviour rather than scripted behaviour as is the case with decision trees.

4.1.3 Visual Scripting

Another approach to modelling artificial intelligence is through the scripting of actions. Advanced systems exist to provide a visual environment in which to compose such scripts. However, the act of scripting in and of itself implies behaviour that is set in stone to achieve some desired effect. Scripting and visual scripting is in most cases employed to create an immersive experience through the complex organisation of actions that are performed when some conditions are met. The Unreal 3 Engine [Inc12c] for example offers users the Unreal Kismet visual scripting formalism, which is shown in figure 4.4. Kismet can be used by ’artists and level designers’ to control ’how 4.1 Modelling Artificial Intelligence 28

Flee from enemies?

health health < 20% > 20%

No, keep Is a ﬁrst aid kit engaging nearby? enemies

No Yes

Flee Flee towards ﬁrst aid kit

Figure 4.3: Decision tree formalising the decision to ﬂee from enemies or not a level will play without touching a single line of code’, according to the ofﬁcial website. It allows developers to formalise how certain scenes will play out, essentially providing a visual formalism for scripted actions. While this allows non-programmers to create scenes, it does not in fact concern itself with modelling a reactive system.

4.1.4 Advanced Toolsets

Another movement in the game AI world looks to provide developers with an advanced set of tools, which require the developer to follow a predefined workflow to develop AI. This workflow is often optimised to achieve quick and impressive AI, but the developer remains tied to the specific implementation of AI in the toolset used. An notable example is Simbionic [FHJ03], a toolset for the description of the behaviour of intelligent agents using finite state machines. The basic workflow proposed by Simbionic can be seen in 4.5. 4.2 AI Model Transformation 29

Figure 4.4: Unreal Kismet script for loading a speciﬁc level

This premise shows remarkable parallels with our approach, but some important differences must be noted. For one, the state of agents is represented as actions. Transitions from one action to another is done using conditions and guards. Furthermore, the framework operates exclusively in a time-sliced fashion, not allowing for the abstraction of time to continuous time when modelling an agent’s behaviour. Lastly Stottler Henke Associates, Inc, the company behind the Simbionic project, has not released any new research since 2003, indicating that the drawbacks of the Simbionic framework are not expected to be addressed anytime soon.

4.2 AI Model Transformation

Despite the fact that model transformation is a very broad area of research, there has not been a great deal of research invested in transforming AI models. This should not come as a surprise to the astute reader, since the practice of visually modelling AI itself has not gained widespread usage at this time. Transformation of AI models is most useful for the generation of variants from a certain starting model. The most advanced research in this line of work comes from the Modelling, Simulation and Design Lab at the McGill University. The last few years, work has been performed in the area of procedural generation of Statechart AI. In Generating Extras: Procedural AI with Statecharts [DKV11], the authors lay the groundwork for procedural AI generation by transforming AI models made using the Statecharts formalism. Generating variants is not the only use for model transformations in the context of game AI, 4.2 AI Model Transformation 30

Figure 4.5: Simbionic workflow [FHJ03] however. Altering AI models such that some sort of desired behaviour results from a (sequence) of transformation is another useful application. The desired behaviour can take many forms. Model transformation can be employed to generate enemy characters who are ’just right’ for some player, posing the player with a challenge which is at the same time difficult, but easy enough to still be entertaining. It can also be used to find the ’best’ AI for a specific task. As a matter of fact, creating a ’pleasing’ game experience can be viewed as a task in this context, and finding the ’best’ AI for a task can be determined by some heuristic that evaluates the AI’s performance. Unfortunately, the research that has been performed in this area is exceptionally limited, a short overview of this work will be given in section 4.2.2.

4.2.1 Procedural AI Generation

The authors of [DKV11] limit themselves to the introduction of variation and personality to pre- existing AI. This pre-existing AI is modelled using the layered Statecharts architecture outlined in the previous section. Three methods for introducing this variety are proposed: varying parameter values, component conﬁgurations and component models.

4.2.1.1 Varying Parameter Values

Non-player characters have many properties defined for each instance of the NPC. Examples of NPC properties are health, fuel level, etc. Varying these properties for each new instance is an obvious way to introduce variety in a group. The proposed technique associates valid ranges with each numerical property, allowing an algorithm to choose a value in this range conforming to some specified probability distribution. However, since some properties might depend upon others, the need for dynamic ranges arises. Let’s take a look at a simple example character to better understand this technique. Consider an 4.2 AI Model Transformation 31 enemy soldier NPC with two properties: weight and fitness. Weight might influence how the physics reacts to the NPC’s presence and fitness might be a property used to determine how long the character can sprint or take other actions. It is clear that in the real world, a certain correlation exists between a person’s weight and their level of fitness. Assume we associate the range [60kg,90kg] with the weight property and [0.5,1.5] with the fitness property (if a value of 1 indicates a ’normal’ fitness level). It would then be possible that characters are generated with a high weight and high fitness property, which might not be desirable. It would be better to relate these two properties in some way, such that the fitness range will be higher or lower depending on the character’s weight. We could define the fitness range as follows:

weightRange = [60,90] fitnessRange = [0.5 + weightDeviation(),1.5 + weightDeviation()] where

minWeight − weight + 0.5(maxWeight − minWeight) weightDeviation() = maxWeight − minWeight

This new, dynamic range for the fitness property ensures that heavy characters get a possible range for the fitness property that is much lower than the possible range for lighter characters. In this simple example, a character with a weight of 60kg would have a possible fitness range of [1,2] and a weight value of 90kg would allow a fitness range of [0,1]. The existence of dynamic ranges also implies a need for organising the evaluation of the various properties, all dependencies must be analysed such that no unresolved values for properties occur when evaluating the ranges. Additionally, we’ll need to check for circular dependencies and erroneous values. The complete proposed method for varying parameter values is given in listing 4.1.

1. For each parameter: 2. Create a range for each parameter. 3. Decide on the probability distribution best suited for that parameter. 4. Examine parameters for dependencies. For each dependency: 5. Determine if the dependency is critical. 6. If critical , assign dynamic ranges to resolve problem . 7 . I f non−critical , resolve if desired , perhaps using dynamic ranges. Listing 4.1: Proposed method for varying parameter values 4.2 AI Model Transformation 32

4.2.1.2 Varying Component Conﬁgurations

The previous method proposed varying the parameter values with the intent of achieving different behaviours for individual components. This section presents a method for changing the overall behaviours of an AI by assembling existing components in different conﬁgurations.

Removing Components The simplest method for generating new component configurations is to remove components. Because our model architecture implies a loose coupling between components, this is relatively easy to do. Events are generated without regard for the components that will react to these events and reaction to events is done without regard for the component(s) that might generate them. Theoretically, for an AI consisting of n components, 2n different component configurations are possible through removing components. However, in practice this is not the case. Often times, there is only one strategic decider component. This layer functions as the connection between the input and output of the AI and severing this connection would lead to an AI that is unable to react to any input. Furthermore, removal of sensor and actuator components is dangerous, because they form the link between the AI and the outside world. This could potentially be leveraged when handicaps are desired (no vision, no movement, no hearing, etc.). Replacing Components Another method is to replace components with other equivalent components. Components in our model architecture are loosely coupled with the classes for which they define behaviour, allowing us to do one of two things. We could associate an existing class with a new Statecharts model or the opposite, create a new class and associate it to an existing Statecharts model. There is one caveat to this: components might have semantic equivalence, but not necessarily syntactical equivalence. This means that a replacement component might not generate the events that are expected by components in the following layers. So in order to automate generation of new component configurations using replacement of components, we need to create a library of semantically equivalent components and a mapping to specify the correspondence between events. Adding Components A final method consists of adding new components to an existing configuration. Adding components to the initial layers – sensors up to strategic deciders – is quite easy. Adding new sensors allows the AI to react to new data about the world and new analysers could improve the AI by detecting new higher level events from a correlation of lower level events. Adding components to the later layers – strategic deciders up to actuators – is somewhat less trivial. These layers react to events generated by earlier layers, so an introduction of a new component at these layers typically means more steps are required to actually utilise this new component, such as event renaming.

[DKV11] further states that the true power of varying component conﬁgurations is to be found in the combination of the above methods. Combining event renaming and component addition allows for completely new types of behaviours that work together well with the existing conﬁguration. 4.2 AI Model Transformation 33

4.2.1.3 Varying Component Models

Arbitrary structural modifications of the AI behaviour models is the most drastic approach to altering the described behaviour. [DKV11] suggests modelling such modifications explicitly using transformation rules, as discussed in chapter 3. Specifically, rule-based model transformation using graph transformations is employed. Some typical modifications applied using this method are resetting components (enclosing a state in a superstate with a transition to itself on some event or interval) and introducing orthogonality (moving a global state into an orthogonal component). These kinds of modifications perform structural changes that are not possible through other methods.

4.2.2 Determining Good Transformations

The introduction of this section hinted at the possibility of finding an AI that is the ’best’ at a certain task through transformations. This goes a step further than the previous section about procedural generation, since we now want to generate variants with some sort of desired behaviour in mind. There has not been extensive research in this field, marking this thesis as the first to attempt to automate this process. There has, however, been some research towards automatic customisation based on the actions of a player in the game. Let’s first have a look at how this is currently done in commercial products. Here we can separate games into two large categories: level-based and non-level-based games.

Level-based This category contains all games in which the player has a certain level, which usually starts at 1 and may or may not have an upper limit. Arguably the most popular example work in this category is World of Warcraft [Inc12b], which brought the concept of character levels to the attention of a broader audience and heralded its introduction in games outside of the RPG2 genre. The level of a player determines his or her strength, as relating to any part of the game. This often means that as players play the game, their level rises and enemies become increasingly easy to defeat. Some games assign a level to enemy characters to, allowing the enemies to level up together with the player. In effect, this creates a sort of binary playing experience. Either enemies are too hard for the player’s level or they are too easy. Only when following a prede- termined path through the game – usually corresponding to the main story line – will enemies be perfectly tailored to the player. A lot of effort has gone into this system in the last few years, some games apply levels better than others, but overall this system is recognised as being too superﬁcial to create a realistic notion of difﬁculty.

Non-level-based Any game in which the player does not have a level belongs to this category. There are various ways in which games try to offer an interesting difﬁculty curve to players without introducing levels: learning new abilities, increasing enemy character’s damage done, etc. This creates an experience that evolves in blocks, each time something new is introduced to alter the difﬁculty

2Role Playing Game 4.2 AI Model Transformation 34

of the game the player’s experience changes. If this is executed with care, it creates very interesting dynamics between the player and the enemies. However, there is no real alteration in behaviour for the enemies. Some enemies are more powerful than others, but on the whole, no signiﬁcant change in behaviour is introduced. A popular example for this category is Bioshock [Sof08], which introduces the player to new abilities, enemies and environments in order to create a difﬁculty curve that feels naturally tailored to the player, but is actually set in stone.

Both categories of games do not actually look at how the player is playing the game in order to determine how the enemies should act. A skilful player could therefore rush through the first levels, where a less experienced player might have a hard time advancing. We must note that some games do try to solve this problem. A popular example is the Call of Duty [Inc12a] series of games. In the last iterations, these games offered a training level in which the player’s performance is assessed. Depending on the assessment, one of the available difficulty levels (easy, normal or hard) is suggested. It’s obvious that such broad difficulty levels can impossibly tailor the experience to every player.

Automatic Transformation

Recent work has tried to offer ways to tailer the whole experience to the individual player. In [Li08b] and [Li08a], the possibility of automatically customising NPC’s based on a player’s temperament is discussed. While this thesis does not concern itself with determining a player’s status in order to transform NPC’s behaviour, this related work nonetheless offers some good insights in how variations can aid in creating an optimal game experience. The author proposed to present a questionnaire to the player when a game is started. The analysis of this questionnaire will then prompt certain variations to arise in the NPC behaviour. This questionnaire can be presented at regular intervals, to allow the experience to keep evolving as the player’s temperament might evolve. The actual variations are performed using transformation rules on a Statecharts AI model. While some questions may be posed concerning the intrusiveness of presenting the player with a questionnaire before gameplay can start, this work shows that determining a player’s abilities and/or state of mind is a very helpful tool to determine the kind of variations that are desirable. The speciﬁc model to use when describing the various aspects of an individual player remains an open question. 5 Statecharts as NPC AI

During this chapter we will develop a detailed architecture for modelling a Non-Player Character’s AI with the Statecharts formalism. The architecture is based on previous work as discussed in section 4.1. Previous work has always operated under the assumption of modelling the behaviour of extras such as in [DKV11], or of robots in a tightly defined context such as in [Bla08]. This thesis concerns itself with describing the behaviour of NPCs that take part in a game, without imposing harsh limits on the type of game or environment in which the NPC will find itself. This means the architecture must be as general as possible, without losing the advantages the Statecharts behaviour modelling approach offers us. To achieve this generality, we will first define a general design for the architecture. Afterwards, we will further develop this architecture to achieve a specific architecture for use in the next chapters of this thesis.

5.1 General Design for a Layered Architecture of Components

The general design for the proposed architecture is an abstraction of the proposed architecture in the related work discussed in section 4.1. Such a layered structure allows a developer to iteratively develop AI that functions in discrete components. Thus far, no actual definition of the principles behind this architecture has been given. This section aims to provide such definitions, while creating a framework for more specific derived architectures that are applicable in specific contexts. We will go over the various parts that make up the architecture and discuss how to design them such that a robust, general architecture for Statecharts behaviour modelling for NPCs can be achieved. We will start by taking a look at how components are linked to the behaviour models in the Statecharts formalism. We’ll then take a closer look at how the layers are formed, the amount of layers that make up the architecture, followed by the level of abstraction from the context that offered by the layers. Then we can take a look at the way in which communication can take place between the various pieces making up the complete architecture: inter-layer communication and intra-layer communication (between components of the same layer). We will end this section with a visual overview of the resulting concept. 5.1 General Design for a Layered Architecture of Components 36

5.1.1 Components

We want to divide the complete description of an NPCs behaviour into smaller parts, such that we can re-use and transform these small parts individually. To allow for this, we introduce the concept of a component. Components are discrete parts of the behaviour description of an NPC. Each such component consists of a class, which contains data and methods on this data relevant to the behaviour described by this component. Each component is also associated with a behaviour model. This model is created in the Statecharts formalism and describes the actual behaviour performed by a single component. Figure 5.1 shows this association of class and model. Class Ac provides access to relevant data and methods, while model Am describes the behaviour for class A.

Component A

Statecharts Model Am data & method access ...

Class Ac +data ... +methods behaviour description ...

Figure 5.1: Single component A consisting of class Ac and behaviour describing model Am

5.1.2 Layers

The various components that we will develop need to be structured in some way. We will retain the notion of layers as was previously introduced. However, we will now generalise this notion to allow for a wider area of application. Layers may consist of only a single component or a very large number of components. The actual content of the layer is completely dependent of the complexity of the context in which the NPC will be required to function. We will think of layers as sets of components. There is no further ordering enforced on the components within a layer. Figure 5.2 shows a single layer L consisting of several components A, B and C. 5.1 General Design for a Layered Architecture of Components 37

Layer L

Component A Component B Component C

Figure 5.2: Single layer L consisting of components A, B and C

Amount of Layers

Different contexts will require a different number of component layers to provide an effective and modular description of the NPC behaviour. NPC AI is usually embedded in a very large game engine with functionality that often overlaps with functionality embedded in the eight layers proposed in the previous work. For example, in the case of spotting enemies, some game engines such as the Unreal 3 [Inc12c] engine generate events upon which the developed AI must respond. This functionality overlaps with the functionality of a sensors layer, making most (if not all) of the components in this layer superfluous. The same issue is encountered with the actuators layer, since most game engines provide an API for the AI to move towards certain locations, rotate towards a direction, etc. This signifies to us that the layered architecture must be flexible and allow any amount of layers to be applicable to any new context in which it is to be applied. This conclusion reduces the layered architecture’s demand for eight layers to simply the concept of a layered architecture in which the layers consist of discrete components. The roles of the layers will then depend upon the functionality provided by the context and the desired behaviour of the NPCs. Figure 5.3 shows the proposed architecture so far: an arbitrary number of layers stacked on top of each other, each layer consisting of an arbitrary number of components.

Abstraction Levels

Since we have now removed the explicit demand for eight layers, we must take a new look at the abstraction levels provided by these layers. The convention is that the top layer receives impulses from the context, meaning the low-level input is performed at the top layers. The bottom layers on the other hand perform interactions with the context, signifying low-level output. As a consequence, there must be a shift between handling input and creating output somewhere in the layered architecture. We call layers that deal with both input and output the middle layers, since these will always be located at or near the middle layer, when they are visually represented as a stack such as in Figure 5.3. It is now clear that the top and bottom layers signify layers with the lowest level of abstraction and that layers closer to the middle will have a higher level of abstraction. This is made clear in ﬁgure 5.4 5.1 General Design for a Layered Architecture of Components 38

...

Layer La

Component A Component B

Layer Lb

Component C Component D Component E

Layer Lc

Component F

...

Figure 5.3: Layered archtecture consisting of arbitrary number of layers (three are visible in this example) for a behaviour description consisting of five component layers. This varying level of abstraction is a consequence and innate property of the layered approach, and derivations of this principle signify the developer that there is a need for more components in some other layer. For example, if the developer requires one of the middle layers to know about obstacles near the player in order to choose a direction to move towards, the middle layer should never perform the detection of these obstacles. Rather, the topmost layer should detect the presence of an obstacle. This presence can then be communicated to the following layers – possibly performing additional logic to include more information about the obstacle – before this information is received by the middle layer(s) and processed by the higher level decision-making components. In any layered AI architecture, the middle layers will be those with the highest level of abstraction. This can be explained very easily, since the middle layers signify a shift between interpreting impulses of the virtual world and acting upon these impulses. The flow of information also remains as it was defined previously: events arrive at the top layers (directly or through API calls), trickle down the layered architecture while new events are generated in each layer and finally the bottom layer is able to generate events or call APIs to interact with the world. 5.1 General Design for a Layered Architecture of Components 39

Layer L1

Component Component A B Low High

Layer L2

Component Component Component C D E Level of Abstraction Levelof

Layer L3

Component Component F G

Layer L4

Component H

Layer L5

Component Component Component I J K

Figure 5.4: Layered archtecture consisting of 5 layers, showing the relative abstraction levels of each layer

5.1.3 Communication

Statechart models react to events that are introduced to them and are able to generate events themselves. We will use these events as the main method for communication between the components and layers of the architecture. It is important to have a good overview of how and when the events are propagated, to understand how the overall behaviour description will be executed. The methods for communication are divisible in two large segments: intra-layer communication – describing the way in which components of one layer communicate with each other – and inter-layer communication – describing the way components of one layer can communicate with components of other layers. 5.1 General Design for a Layered Architecture of Components 40

Intra-Layer Communication

When a layer consists of multiple components, it is desirable that when events are generated by a component, the other components in the same layer can receive these events and respond to them. Since each component has their own Statecharts model, events generated in one of these models are not directly accessible by the component models in the same layer. However, this is a highly desirable property. Since decisions in one component of a layer might influence decisions taken by another component in the same layer, we would like these components to be able to receive each other’s events. If this would not be possible, all these conflicts would have to be solved by a coordination layer. The previously proposed architecture already contained one coordination layer, which served to coordinate decisions taken by the components of the executor layer. This coordination layer however solved the problem of allowing different actuators make the most efficient decisions, given the decisions taken by the executor layer. This layer thus solves the problem of coordinating two different layers, not components within one layer itself. To coordinate the decisions made by components of the same layer, we will catch events generated by the components of each layer and push them to the other components in the same layer, before they are pushed to the subsequent layers. This effectually places the component models of one layer in one large Statecharts model as orthogonal components, while still allowing modular development through different components (and component models). The best of both worlds.

EventQueue Layer L event generation

event propagation Component B Component A

Figure 5.5: Layer with two components and an Event Queue

We will achieve this functionality by introducing an event queue in each layer. The component models can then provide their events to this event queue, which will take care of further propagating the events. We associate this event queue with the entire component, which means that not only is the component model capable of generating events, but the class contained within a component is likewise capable of generating events if desired. Figure 5.5 shows a single layer, now with event queue attached.

Inter-Layer Communication

The previous version of this architecture used narrow-casting in order to allow inter-layer com- 5.1 General Design for a Layered Architecture of Components 41 munication between components. This has several advantages, such as avoiding namespace saturation and increased traceability of the cause for certain behaviours. However, when we want to automatically transform the NPC behaviour as in this thesis, this narrow-casting approach becomes cum- bersome. We need an architecture in which we can easily substitute components with others, leave out some components or add others. With narrow-casting this would quickly cause conflicts due to missing components. To avoid such issues, will use an approach called ’event-bubbling’. Event-bubbling means that when an event is generated in a layer, it will be propagated to all subsequent layers, layer by layer, from the topmost (input) layer to the bottom (output) layer. Figure 5.6 shows how this principle works when an event is generated in the topmost layer. Events generated by components from layer La are first pushed to the components of layer La itself, as mentioned in the previous chapter. The events are then pushed to layer Lb, Lc and Ld respectively, whose event queues will accept the events and propagate them to the components of their layer. Events generated by components from layer Lb are pushed to components of layer Lb first, then to layer Lc and Ld. Events from components in layer Lc are pushed to components from layer Lc and then to layer Ld. Finally, events from components of layer D are only pushed to components in layer Ld, because no subsequent layers exist. Events thus propagate in one direction only, from the input side of the architecture to the output side. The main goal of this method for propagating events was twofold: retaining the logical path from input to output on the other hand, but also forcing the developer to take care when designing the NPC behaviour, meaning the resulting components will be more modular and abstracted than they would have been without this event bubbling functionality. 5.1 General Design for a Layered Architecture of Components 42

EVENT A

Layer La EVENT A EventQueue Layer La EventQueue Layer La EventQueue

Component A Component B Component A Component B Component A Component B EVENT A

Layer Lb EventQueue Layer Lb EventQueue Layer Lb EventQueue

Component C Component D Component E Component C Component D Component E Component C Component D Component E

Layer Lc EventQueue Layer Lc EventQueue Layer Lc EventQueue

Component F Component F Component F

Layer Ld EventQueue Layer Ld EventQueue Layer Ld EventQueue

Component G Component H Component G Component H Component G Component H

(a) (b) (c)

EventQueue Layer La EventQueue Layer La Layer La EventQueue

Component A Component B Component A Component B Component A Component B

EventQueue Layer Lb EVENT A EventQueue Layer Lb Layer Lb EventQueue

Component C Component D Component E Component C Component D Component E EVENT A Component C Component D Component E

Layer Lc EventQueue Layer Lc EventQueue Layer Lc EventQueue

Component F Component F Component F EVENT A

Layer Ld EventQueue Layer Ld EventQueue Layer Ld EventQueue

Component G Component H Component G Component H Component G Component H

(d) (e) (f)

EventQueue Layer La Layer La EventQueue

Component A Component B Component A Component B

EventQueue Layer Lb Layer Lb EventQueue

Component C Component D Component E Component C Component D Component E

Layer Lc EventQueue Layer Lc EventQueue

Component F EVENT A Component F

Layer Ld EventQueue Layer Ld EVENT A EventQueue

Component G Component H Component G Component H

(g) (h)

Figure 5.6: Event Propagation through Event Bubbling 5.2 A Test Platform for Statechart AI 43

5.1.4 Connection to a Player

So far, we’ve discussed how to organise various components such that we can describe the behaviour of a player. There is one very important piece of functionality that we have not discussed yet: how to connect the component layers to a player such that they may receive input and provide output to the player. To this end, we will adopt the controller mechanism for controlling an object through its controller. The various layers and the components within are stored inside of a controller object, which can take control of player objects. This approach is widely used throughout the industry to abstract the logic for performing actions from the actual representation of a player in the game. The controller class should provide access to the player representation to allow the input layer(s) to collect information from the game and output layer(s) to perform actions. Finally, the controller class should also take care of the translation from continuous time to discrete time, if the game engine context is designed in discrete time. Statecharts is an event-based formalism, meaning that by deﬁnition, it is designed with continuous time in mind. When Statecharts models are used in a discrete time environment, one must take care to correctly slice the time and guarantee correct and in-order propagation of the events.

5.2 A Test Platform for Statechart AI

In order to create a concrete implementation of the proposed architecture from section 5.1, we need to create a test platform that will serve as the context for which we develop NPCs and their corresponding behaviour. This test platform should conform to the modern practices used in developing game engines, such that the test platform can serve as a good indicator of how the tested NPC behaviour will perform in an actual game engine. Previous work has traditionally adopted the concept of a Tank Wars type of game to showcase the proposed techniques. The Tank Wars concept provides a simple, yet powerful testing environment for analysing NPC behaviour. In this thesis, we will one again use this type of game as a means to further develop the proposed architecture. More information concerning the Tank Wars type of gameplay is provided in subsection 5.2.1. The test platform needs to contain all aspects of a modern game engine, such that it can serve as a useful subject of comparison with commercial products. This means its functionality should be quite broad, but we are able to simplify much of this functionality that is not directly related to the development of NPC behaviour. Finally, the test platform should also be able to run in a headless mode, so that it may serve as an analysis platform for the performance of behaviours (more on this in chapter 6.

5.2.1 Game Type: Tank Wars

Tank Wars games typically provide a top down view of a map in which several tanks are spawned. Such a map can be a simple empty space, but might also be a more complex area with obstacles that 5.2 A Test Platform for Statechart AI 44 must be avoided or useful pickups that can aid tanks in defeating enemies. Tanks may or may not be organised in teams. The tanks themselves consist of two parts: the tank body and the tank turret. These can turn independently from each other and each part may have their own ﬁeld of vision, usually implemented in the form of radars. Figure 5.7 shows a tank from a Tank Wars style game.

Tank Body

fov β Tank Turret Tank Radar range y

fov α

Turretrange Radar x

Figure 5.7: A Tank in a Tank Wars Style Game

A popular example of this style of game is Robocode [NL12]. This game provides an environment in which tanks are pitted against each other in tournaments. The AI for each tank is written by the players, and the goal is to write the best AI such that your own tank consistently beats the other players’ tanks. In what follows, some concepts used in this project will make an appearance, since this project shows very interesting similarities to this thesis’ goal of analysing NPC behaviour.

5.2.2 Creating the Test Platform

The creation of the test platform was started near the end of the first semester of the academic year. Ideally, you first design the software before you make decisions as to the programming language you would like to use. However, there were some constraints that required me to decide upon the programming language before I could actually begin my work on designing the actual application. I was going to reuse a lot of software developed years prior to my own thesis, which put a harsh constraint on the available choices. The Statecharts compiler I was going to use (SCC [Fen03]) could only generate code in a handful of programming languages, with even fewer programming languages available for which it could compile completely (some subsets of the Statecharts formalism were not available depending on the target language). Furthermore, I wanted to use a programming language 5.2 A Test Platform for Statechart AI 45 that I had not used extensively before, but it also had to provide adequate support for fast prototyping and a decent 2D graphics package needed to be available. The above requirements were all met by the Python [vRD01] programming language. It is a very versatile language with a very broad standard library. This means one can easily develop more complex applications without requiring the use of third party libraries. Furthermore, the documentation on the official Python website proved most useful in learning the ins and outs of the language and got me up to speed very quickly.

2D Graphics Package

Now that I had settled on a programming language to use, I needed to decide upon the speciﬁc 2D graphics package I would use. In order to create a test platform, I needed a decent graphics package to allow me to create the dynamic game environment needed to serve as a test platform for NPC behaviour. After a bit of research and some talks with Professor Vangheluwe, I settled on the Pygame [Shi12] package. Pygame provides a high-level 2D graphics API, while still allowing ﬁne grained control over more low-level functionality if so desired. It is in essence a Python wrapper for the SDL library, which is a popular cross-platform multimedia library. The available documentation for the Pygame project is not very detailed, but it is adequate and I had no trouble in learning to use this third party software.

Required Functionality

With the practical decisions out of the way, I was certain that I would be able to create the functionality I needed. Now I only needed to determine what kind of functionality would be required for this test platform. Not all functionality ended up being used in the later parts of this thesis, but it may still serve as building blocks for future work in this area. The following is a short list of all the functionality I deemed required when designing the test platform from scratch.

Players Obviously I needed to implement the actual players. They should be able to control some pawn or pawns in the game and should themselves be controlled by either a human – through mouse or keyboard input – or Statechart-powered AI – through the above mentioned architecture for Statechart modelled behaviour. Furthermore, since we have already decided upon building a Tank Wars style game, a pawn in the form of a tank should be implemented. They should function as one expects a tank from a Tank Wars type of game to function. The turret should be able to ﬁre projectiles and be able to turn independently from the body. However, when the body turns, the turret should turn with it, since its point of attachment is the tank body itself.

Maps Using only an open space as the map would not allow many of the more interesting scenarios to be tested. NPCs that are able to take cover or use the environment in any way give a more 5.2 A Test Platform for Statechart AI 46

realistic impression to human players, so it seemed obvious that I needed some map system that would enable me to create more complex maps. Furthermore, the test platform should be able to go through a list of maps in one execution of the platform. This meant I also needed to be able to specify the list of maps for the current simulation run. Rounds Because spawning some tanks on a map and seeing what happens is not very test-able nor is it very useful, I required the game to support playing rounds. A round would start by spawning all the players present in the game and end when only one player remains or the round time limit is reached. Rounds also enable the tracking of scores over the course of time. Without rounds, the game would have to be restarted every time you would want to test the NPC behaviour again and there would be no way to save or compare scores of the players. Sessions Eventually, the test platform would have to be used in an automated fashion. This implied that all the configuration data for a single execution needed to be stored in some easily accessible format, that I could also generate on the fly. This lead to the concept of sessions, which encom- pass the entire configuration for a single execution: which players take part in the game, what maps will be played, how many rounds per map, and so forth. Simple starting and stopping of simulation The final kind of functionality I wanted to achieve wasn’t particularly difficult to implement. However, it signifies the functional nature of the test platform. It is exactly as its name implies, a test platform. This means I wanted to start the simulation of NPC behaviour with a single command, observe the results and be able to stop the simulation at any time. As little run-time configuration as possible should be required. Time-sliced design I could choose between creating an event-based game engine or a time-sliced game engine. I decided upon designing the game in a time-sliced fashion, because a the code generated by the Statecharts compiler is completely event-based and it would be quite trivial to also design the entire test platform in an event-based way. By opting for the time-sliced design, I could explore both the event-based nature of Statecharts while dealing with the complexity of the time-sliced architecture used in many commercial products today, such as the Valve Source Engine [Cor12]. Headless mode The test platform should be able to run without a ’head’, in the sense that it should be able to run without rendering anything to the screen. This will allow for the simulation to occur as fast as the machine running the program is able to execute the code, allowing for automatic testing of large groups of NPC behaviours.

5.2.2.1 Design

Now that I had a broad description of the required functionality, I got to work creating class diagrams for the design of the test platform. The actual implementation was a very iterative process. To be 5.2 A Test Platform for Statechart AI 47 completely honest, I did not always follow the design I had put forth exactly and often times the test platform grew organically while I was solving some particular problem. I found this to be a very good introduction to real-world application development, where the design often clashes with technological difficulties or new requirements set forth by the client. In this case I was my own client, but like any client in a software project, I often changed my mind about how I wanted some particular feature to look like, which resulted in changes to the design. During the final iteration, the design seen in figure 5.8 was created. It is the way the current version of the test platform is structured and provides a very good overview of the various components that make up the platform. In what follows I will provide an overview of the various entities in the class diagram and explain their function.

General Remarks

The class diagram shown does not provide any indication concerning the use of configuration files or third party libraries. Where configuration files are used, they will be fully described in the following paragraphs. A lot of classes use third party libraries in some fashion, which will also be described in the following paragraphs. Classes provided by the Pygame library are always referred to in the form of pygame.Class, as this is also the way in which pygame provides access to these classes when importing them in your own project. Due to the nature of the Python programming language, class types do not correspond completely between the class diagram seen in figure 5.8 and the actual source code. I have opted to provide the harshest constraints on the types in the class diagram, the source code may not place a type constraint on some variables, rather requiring a certain method to be available. This in essence corresponds to requiring the variable to conform to an interface, which takes the form of a very simple base class in the source code. Finally, when the class diagram shows some variable to be of type {}, this means the variable is a dictionary type. Dictionaries are used often throughout the source code to save data for configuration or serialisation purposes. The Python notation for lists is also employed in the class diagram, meaning a notation of the form [Class] is a list with elements of type Class. 5.2 A Test Platform for Statechart AI 48 Map Game gamemath -config: {} -config: Game -game: pygame.Surface -blueprint: [pygame.Rect] -walls: int -ambient: -surface: pygame.Surface Spritesheet +spritesheet: pygame.Mask +collisionMask: (int, +area: int) mapFile) +Map(game, -createGraph() -determineWalls() findTargets=True):+collision(sprite, colliders) (bool, (float, float) +getNodePos(node): (int, int) +getPosNode(pos): +getWalls(): [pygame.Rect] +getWallMask(): pygame.Mask bool +haveLOS(): +loadPickups() [Player] radius=1): +playersAt(pos, pygame.Surface +render(): (float, float) +spawn(player): int) +tick(time_passed: +unload() -settings: -settings: dict dict -sessionconfig: bool -headless: -clock: pygame.time.Clock pygame.Surface -screen: dict -schedule: int -timepassed: +players: pygame.sprite.OrderedUpdates +pickups: pygame.sprite.RenderUpdates pygame.sprite.RenderUpdates +hud: Luminosity +luminosity: +Game(config) +after(time, function) +runAfters(time_passed) +time(): int +start() maps +distance(x, y): +distance(x, float y0), (x1, y1)): +distance((x0, float sprite1): +distanceSprite(sprite0, float y0), (x1, y1)): float +angle((x0, float sprite1): +angleSprite(sprite0, angle): +rotate(image, pygame.Surface y0, x1, +tracePoint(x0, y1, l): float y0), +intersects((x0, (x1, y1)): bool b, c): +area(a, float +isOnLeft(a, b, c): bool b, c): +isOnRight(a, bool b, c): bool +isCollinear(a, * lib flow of flow uses controls game with game 1 Session 1 * 1 Light -game: Game -game: {} -config: -startTime: int -maps: [String] Map +current_map: int -roundsPerMap: int -currentRound: [Player] -players: Game, configFile: +Session(g: String) Player) +addPlayer(p: Player) +removePlayer(p: +switchMap() +start() +stop() +findLiveItems(sprites): [Player] [Player] +getLivePlayers(): int) +tick(time_passed: takes takes part in Luminosity 1 groups -game: Game -game: int -ambient: [Light] -lights: Game, +Luminosity(g: int) ambient: +setAmbient(intensity: int) +clear(s: bg: pygame.Surface, pygame.Surface) +update() +draw(s: pygame.Surface) Light) +addLight(l: Light) +removeLight(l: -game: Game -game: int -radius: Player -source: int -intensity: pygame.Surface -image: -rect: pygame.Rect Player) +Light(owner: +get_bounding_rect(): pygame.Rect int) +update(time_passed: pygame.Surface) +draw(s: +vanish() * Round consists of consists Bullet Event Projectile -game: Game -game: Session -session: Map -current_map: int -roundNum: -startTime: int -stopTime: int int -timeLimit: ScoreBoard -scoreBoard: Game, roundNum: +Round(g: int) +start() +stop() bool +isFinished(): int) +tick(time_passed: StatBar 1 BoundingBox 1 -_type: -_type: string -_params -_tickid +getType(): string any +getParams(): -game: Game -game: -strength: int -image: pygame.Surface -rect: pygame.Rect +Projectile(owner: Player, imgID: String, float, strength: speed: int) +update(time_passed: int) +vanish() +explode() Player) +Bullet(owner: * -owner: Player -owner: int -totalHeight: pygame.Surface -image: -rect: pygame.Rect Player) +StatBar(owner: int) +update(time_passed: +vanish() Player -owner: int -totalHeight: pygame.Surface -image: -rect: pygame.Rect +BoundingBox(owner: Player) +vanish() 1 causes internal generates representation 1 1 ScoreBoard 1 -scores: -scores: [Score] int -round: +ScoreBoard(round: int) Player) +addPlayer(p: Player) +getScore(p: +log() +output(): {} Text RepairShop * contains Explosion -msg: String pygame.Color -colour: -font: pygame.font.SysFont bool -antialias: pygame.Surface -image: -rect: pygame.Rect +Text(g: text:game, String, size: int, colour: pygame.Color, font: String, bool) antialias: +setFont(f: String, size: int) +setText(text: String) float, y: +place(x: float) +center(x: float, y: float) int) +update(time_passed: +vanish() +RepairShop(spsh: +RepairShop(spsh: g: Game, Spritesheet, pos: (float, float)) +affectPlayer(p: Player) HUD Elements HUD -game: Game -game: Spritesheet -spritesheet: pygame.Surface -image: -rect: pygame.Rect int -magnitude: int -time_passed: int -phase: int -increment: Game, owner: +Explosion(g: Player, pos: (float, float), int) magnitude: +explode() int) +update(time_passed: +vanish() EventQueue Score TankBrigade rules -player_name: String -player_name: int -player_team: int -timealive: int -nr_kills: int -damage_taken: int -damage_done: int -nr_healthpickups: int -nr_fuelpickups: bool -stranded: bool -died: Player,+Score(player: {}) dictionary: +log() +output(): {} deque -deque: bool -_tickid: +EventQueue() string, params=None) +push(type: +get(type: string) +getAll() +clean() played with played +TankBrigade() +load() Pickup FuelBarrel 1 * +FuelBarrel(spsh: Spritesheet, Spritesheet, +FuelBarrel(spsh: g: Game, pos: (float, float)) +affectPlayer(p: Player) -game: Game -game: -reappearingTime: int pygame.Surface -image: -rect: pygame.Rect g: Spritesheet, +Pickup(spsh: Game, imgID: String, pos: (float, float)) +getPos(): (float, float) bool) +setActive(active: +isActive(): bool int) +update(time_passed: bool) +vanish(reappear: +affectPlayer(p: Player) int) +tick(time_passed: Player communicates using communicates Spritesheet 1 -session: Session -session: Game -game: int -createdat: int -deathdat: Controller -controller: int -health: int -maxHealth: -fuel: int int -maxFuel: String +name: +team: int Game, +Player(game: String, team: name: int) +setFuel(f: int) int +getFuel(): int) +setHealth(h: int +getHealth(): int) +setMaxHealth(mh: int +getMaxHealth(): +setMaxFuel(mf: int) int +getMaxFuel(): Controller) +setController(c: +unsetController() +isControlled() +despawn() +spawn() bool +alive(): int) +damage(amount: +kill() bool Player): +isAlly(p: int) +tick(time_passed: -file: -file: String int -tilesize: String -name: -key: int -sheet: pygame.Surface -sprites: {} String) +Spritesheet(filename: String, +defineSprite(name: rect:colorkey: pygame.Rect, int) +getSpriteImage(name: String): pygame.Surface String): +knownSprite(name: bool +makeTiledSurface(tile: String, size: (int, int)): pygame.Surface Tank 1 -game: Game -game: pygame.Surface -body_img: pygame.Surface -image: -rect: pygame.Rect bool -imagechange: +Tank(game: Game, player: Player, int) colour: float) +setAngle(angle: +setTurretAngle(angle: float) +getTurretAngle(): float +getTurret(): Turret +turnTurretTo(target: MoveableSprite) +refreshImage() int) +tick(time_passed: int) +update(time_passed: +vanish() +fire() +isEnemy(other: OwnedObject) +isFriendly(other: OwnedObject) representation 1 1 Figure 5.8: Class diagram for the complete test platform has a has 1 NonPlayer HumanPlayer Component +HumanPlayer(game: Game, +HumanPlayer(game: String, team: name: int) int) +tick(time_passed: +spawn() Game, +NonPlayer(game: String, team: name: int, config: String) int) +tick(time_passed: +spawn() Turret CollideableSprite -game: Game -game: Player -player: {} -config: ComponentLayer -layer: Player,Game, player: +Component(game: String, localVars)componentconfig: +start() +stop() +readConfigValue(key: String): String ComponentLayer) +setLayer(layer: OwnedObject * -turret_img: -turret_img: pygame.Surface pygame.Surface -image: -rect: pygame.Rect float -angle: float -turnSpeed: -turnToAngle: float bool -imagechange: +Turret(game: Game, Player,owner: colour: int) +getPos(): (float, float) float) +setAngle(angle: float +getAngle(): +setTurnSpeed(ts: float) +getTurnSpeed(): float +turnTo(angle: float, ts: float) +refreshImage() +update(time_passed: int) +fire() +CollideableSprite(bbox: +CollideableSprite(bbox: Map, world: pygame.Rect, [pygame.Group]) containers: float, -clip(x: y: float) -collide() +setPos((x: float, y: float) -owner +OwnedObject(owner) owner's +getOwner(): Type +setOwner(owner) bool +isMine(object): contains InputController controlled by controlled ComponentController -x: int -y: int int) +tick(time_passed: 1 -layers: [ComponentLayer] -layers: +ComponentController(player: Player, Game) game: +start() +stop() int) +tick(time_passed: 1 * 1 ComponentLayer controls Tombstone Controller MoveableSprite 1 sprites -image: pygame.Surface -image: -rect: pygame.Rect +TombStone(pos: (float, float), map: Map, containers: [pygame.Group] +player +active Player) +Controller(player: +start() +stop() +isActive(): bool +tick(time_passed) players -components: [Component] -components: +events: EventQueue +ComponentLayer([(componentName: Component)] String, component: String, params) +event(event: +start() +stop() +get(): [Component] -bbox: pygame.Rect -bbox: -x: float -y: float float -speed: float -turnSpeed: float -angle: -ax: float -ay: float Map -world: -moveToPos: (float, float) -turnToAngle: float +MoveableSprite(bbox: Map, world: pygame.Rect, [pygame.Group]) containers: +setPos((x: float, y: float) +getPos(): (float, float) float) +setAngle(alpha: float +getAngle(): (float, float) +getAngleProjection(): float) +setSpeed(s: float +getSpeed(): +setTurnSpeed(s: float) +getTurnSpeed(): float pygame.Rect +getBoundingBox(): int) +updatePosition(time_position: +turnTo(angle: float, turnSpeed: float) +moveTo(object: MoveableSprite) float, y: +isNear((x: float)) +isMovingToDestination(): bool +getClosest(objects: [MoveableSprite]) 5.2 A Test Platform for Statechart AI 49

Package: Players

players Controller Player +player 1 controlled by 1 -session: Session +active -game: Game +Controller(player: Player) -createdat: int InputController HumanPlayer +start() -deathdat: int -x: int +HumanPlayer(game: Game, +stop() -controller: Controller -y: int name: String, team: int) +isActive(): bool -health: int +tick(time_passed: int) +tick(time_passed: int) +tick(time_passed) -maxHealth: int +spawn() -fuel: int -maxFuel: int +name: String ComponentController +team: int -layers: [ComponentLayer] NonPlayer +Player(game: Game, +ComponentController(player: +NonPlayer(game: Game, name: String, team: int) 1 Player, game: Game) name: String, team: int, config: +setFuel(f: int) +start() String) +getFuel(): int +stop() +tick(time_passed: int) +setHealth(h: int) +tick(time_passed: int) +spawn() controls +getHealth(): int * +setMaxHealth(mh: int) ComponentLayer +getMaxHealth(): int -components: [Component] Component +setMaxFuel(mf: int) +events: EventQueue -game: Game +getMaxFuel(): int +ComponentLayer([(componentName: -player: Player +setController(c: Controller) String, component: Component)] -config: {} +unsetController() +event(event: String, params) -layer: ComponentLayer +isControlled() +start() +Component(game: Game, player: Player, +despawn() +stop() componentconfig: String, localVars) +spawn() +get(): [Component] +start() +alive(): bool * +damage(amount: int) 1 +stop() +readConfigValue(key: String): String +kill() contains +setLayer(layer: ComponentLayer) +isAlly(p: Player): bool +tick(time_passed: int)

Figure 5.9: Class diagram for Players Package

The Players package consists of all classes required or the implementation of players that take part in the game. It contains logic implementing the architecture described in the section on a general design for layered architecture of components. We will now go over the various classes and their purposes.

Component The class that represents a component as it was described in section 5.1.1. It is constructed for a specific instance of the Game class and a single Player instance. The model and its parameters are specified in a component configuration file, whose location on the local system is specified using the parameter componentconfig. The format of the component configuration file is fully described in section 5.2.2.2 Finally, the parameter localVars allows a developer to introduce new variables to the scope of the component object, which can then be used in the component’s model. 5.2 A Test Platform for Statechart AI 50

Components can be started and stopped, which allows us to create components once and restart them each time the player will take part in the game again. Each component can be added to a layer using the setLayer method. Finally, a convenience method is provided to more easily read parameters from the component’s conﬁguration. To create actual components, the user should inherit from this base class for components. The start and stop methods can then be overridden to perform additional tasks, but it is always required to call the base class’ respective methods, since they handle the component model’s initialisation and other administrative tasks.

ComponentLayer This class groups components into layers. It consists of a list of components in arbitrary order and a single EventQueue. Instances of this class can be created by providing a list of pairs of a component name and the corresponding instance of the Component class. Events may be sent to a layer using the event method. As was the case with the Component class, the ComponentLayer class can be started and stopped when required. Finally, all components from the layer can be retrieved using the get method.

Controller The base class for all controllers. A controller’s task is to, in effect, control all actions undertaken by a player. It is constructed for a single instance of the Player class and can be started and stopped when desired. This class also contains a tick method, which implements the time-sliced notion of time as it was put forth in the required functionality paragraph. This method receives a parameter time_passed which is the number of milliseconds passed in the current time slice. The content of this method must then perform all logic required for the current time slice.

InputController This simple class inherits from the base Controller class implements a keyboard and mouse controller to allow simple control of a player by the user. It only overrides the tick method, to catch the required input events and subsequently call the appropriate methods on the Player object it controls.

ComponentController This class inherits from Controller and implements the control of a player by a layered architecture of components described in 5.1. It is created for a single Player instance taking part in a game. It overrides the start and stop methods to perform initialisation of the component layers and the overridden tick method implements the actual time-sliced logic to allow the event-based Statecharts models to function. An instance of this class is always created from a configuration file. This configuration file can always be found in controller.cfg in the configuration folder of the player the controller controls. A full description can be found in section 5.2.2.2.

Player This is the base class for any player that will take part in the game. Players are always created 5.2 A Test Platform for Statechart AI 51

for a game, using a player name and team. Teams are implemented as simple integers, to allow for an arbitrary number of teams, if so desired. It contains many convenience functions to retrieve data about the player, such as its health and fuel levels, methods for checking if the player is alive or if another player is an ally or not. The more important methods to note here are as follows. setController allows any controller to be associated with the player at any time. This means that, if desired, the same player can be controlled by different controllers in different rounds of the game, without needing to recreate an instance of the Player class. Finally, the tick method contains logic that updates any attributes of the player and calls the same method on the player’s controller. HumanPlayer Humans can also take part in the game, through instances of this class. Its constructor takes the exact same parameters as the Player base class does, no extra configuration is required. The spawn method is overridden in order to associate an InputController instance with the player every time the player is spawned. NonPlayer Any player controlled by a Statecharts behaviour description is an instance of this class. Its constructor takes one extra parameter: config, which is the name of this player’s configuration directory. This directory contains the configuration of the player controller, all of its components and the component models. More on player configuration directories in section 5.2.2.2.

Package: Rules

rules

Score ScoreBoard Round Session -player_name: String -scores: [Score] -game: Game -game: Game -player_team: int -round: int -session: Session -config: {} -timealive: int +ScoreBoard(round: -current_map: Map -startTime: int -nr_kills: int int) -roundNum: int -maps: [String] -damage_taken: int +addPlayer(p: Player) -startTime: int +current_map: Map -damage_done: int +getScore(p: Player) -stopTime: int -roundsPerMap: int -nr_healthpickups: int +log() -timeLimit: int -currentRound: int -nr_fuelpickups: int +output(): {} -scoreBoard: ScoreBoard -players: [Player] -stranded: bool +Round(g: Game, roundNum: +Session(g: Game, configFile: 1 1 generates -died: bool int) String) +Score(player: Player, 1 +start() +addPlayer(p: Player) dictionary: {}) * +stop() +removePlayer(p: Player) +log() contains +isFinished(): bool +switchMap() +output(): {} +tick(time_passed: int) +start() +stop() consists of * +findLiveItems(sprites): 1 [Player] +getLivePlayers(): [Player] +tick(time_passed: int)

Figure 5.10: Class diagram for Rules Package 5.2 A Test Platform for Statechart AI 52

The rules package contains any and all classes that deal with the flow of the game. There are not a very large amount of rules that need to be enforced, compared to a fully featured commercial product. However, the ’rules’ contained within this package provide the function of enabling repeatability of experiments and providing structure to the flow of the game, which in turn allows us to save scores for well-defined short gameplay sessions. The classes contained within this package are as follows.

Session Each game consists of one session. To create a session, we need to supply the Game instance and the path to the session configuration file, which is described in detail in section 5.2.2.2. Sessions consist of a list of maps that will be played, the number of rounds played for each map and finally a list of players that will take part in the session. Sessions can be started and stopped with the appropriate methods and some convenience methods are provided to retrieve living players present in the current session. Finally, the tick method performs all logic required for the current time-slice. Most notably, it calls the same method on the current instance of the Round class, after checking if the current round is still active. If the current round is no longer active, it will be restarted if the maximum number of rounds for the current map has not yet been reached. If the maximum number of rounds for the map have already been played, the next map on the map list is loaded and the first round for that map is started.

Round As was already mentioned in the paragraph concerning Session, this class represents a short, well-deﬁned round of gameplay on a single map. It is created for a Game instance, a round number must also be provided on creation of instances of this class, in order to identify and distinguish different rounds when performing analysis tasks later on. Rounds can again be started and stopped, but for convenience an isFinished method is provided. This method will automatically stop the round if it detects the current round is over. There are two possible reasons for a round to be ﬁnished: either the round time limit has been reached or there is only one surviving player left. Rounds also keep track of a ScoreBoard, meaning each new round starts off with a score of zero for every player.

Score In order to keep track of the performance of each player – for the purposes of analysis – we need to create instances of the Score class for each player participating in the game. The class is designed to be easily expandable, as what is saved by this class limits the type of analysis that can be performed later on. Instances of this class can be created in one of two ways: By providing an instance of the Player class, in which case a score of zero for all attributes is created. It can also be created by specifying a dictionary, containing the serialised form of an instance of this class. This dictionary will then be parsed and all attributes will be assigned to the values found in the dictionary. 5.2 A Test Platform for Statechart AI 53

The following table shows the attributes that are saved and available for analysis of the player’s performance.

Attribute Type Meaning player_name String Name of the player associated with this score instance player_team Integer Identiﬁer for the team of the player associated with this score instance timealive Integer Seconds this player was alive nr_kills Integer Number of kills made by the player damage_taken Integer Amount of damage taken by enemies damage_done Integer Amount of damage done to enemies nr_healthpickups Integer Number of pickups restoring health consumed nr_fuelpickups Integer Number of pickups restoring fuel consumed stranded Boolean Did the player lose all fuel? This causes the player to become immobile. died Boolean Did the player die?

ScoreBoard This class keeps track of the Score instances for each player present in the current round. It must be created with the current round number, such that when it the scoreboard is outputted to ﬁle, the various rounds can still be individually identiﬁed and distinguished from each other.

Package: Sprites

sprites MoveableSprite Spritesheet HUD Elements -bbox: pygame.Rect -file: String Luminosity Text StatBar -game: Game -x: float -tilesize: int -msg: String -owner: Player -y: float CollideableSprite -name: String -ambient: int +CollideableSprite(bbox: -colour: pygame.Color -totalHeight: int -lights: [Light] -speed: float -key: int -font: pygame.font.SysFont -image: pygame.Surface -turnSpeed: float pygame.Rect, world: Map, -sheet: pygame.Surface +Luminosity(g: Game, containers: [pygame.Group]) -antialias: bool -rect: pygame.Rect ambient: int) -angle: float -sprites: {} -image: pygame.Surface +StatBar(owner: Player) -ax: float -clip(x: float, y: float) +Spritesheet(filename: String) +setAmbient(intensity: -collide() -rect: pygame.Rect +update(time_passed: int) int) -ay: float +defineSprite(name: String, +Text(g: game, text: String, +vanish() -world: Map +setPos((x: float, y: float) rect: pygame.Rect, colorkey: +clear(s: size: int, colour: pygame.Surface, bg: -moveToPos: (float, float) int) pygame.Color, font: String, BoundingBox pygame.Surface) -turnToAngle: float +getSpriteImage(name: antialias: bool) OwnedObject -owner: Player +update() +MoveableSprite(bbox: String): pygame.Surface TankBrigade +setFont(f: String, size: int) -owner -totalHeight: int +draw(s: pygame.Rect, world: Map, +knownSprite(name: String): +TankBrigade() +setText(text: String) +OwnedObject(owner) -image: pygame.Surface pygame.Surface) containers: [pygame.Group]) bool +load() +place(x: float, y: float) +getOwner(): owner's Type -rect: pygame.Rect +addLight(l: Light) +setPos((x: float, y: float) +makeTiledSurface(tile: String, +center(x: float, y: float) +setOwner(owner) +BoundingBox(owner: +removeLight(l: Light) +getPos(): (float, float) size: (int, int)): +update(time_passed: int) +isMine(object): bool Player) +setAngle(alpha: float) pygame.Surface +vanish() 1 +getAngle(): float +vanish() +getAngleProjection(): (float, float) groups * +setSpeed(s: float) Projectile +getSpeed(): float Light has a -game: Game -game: Game +setTurnSpeed(s: float) Pickup Explosion Turret Tank -strength: int -radius: int +getTurnSpeed(): float -game: Game -turret_img: 1 1 -game: Game -game: Game -image: -source: Player +getBoundingBox(): pygame.Rect -spritesheet: Spritesheet pygame.Surface -body_img: pygame.Surface -reappearingTime: int pygame.Surface -intensity: int +updatePosition(time_position: int) -image: pygame.Surface -image: pygame.Surface -image: pygame.Surface -image: pygame.Surface -rect: pygame.Rect -image: pygame.Surface +turnTo(angle: float, turnSpeed: -rect: pygame.Rect -rect: pygame.Rect -rect: pygame.Rect -rect: pygame.Rect +Projectile(owner: -rect: pygame.Rect float) -magnitude: int -angle: float -imagechange: bool +Pickup(spsh: Spritesheet, g: Player, imgID: String, +Light(owner: Player) +moveTo(object: MoveableSprite) -time_passed: int -turnSpeed: float +Tank(game: Game, player: Game, imgID: String, pos: speed: float, strength: +get_bounding_rect(): +isNear((x: float, y: float)) (float, float)) -phase: int -turnToAngle: float Player, colour: int) int) pygame.Rect +isMovingToDestination(): bool -increment: int -imagechange: bool +setAngle(angle: float) +getPos(): (float, float) +update(time_passed: +update(time_passed: int) +getClosest(objects: +Explosion(g: Game, owner: +Turret(game: Game, +setTurretAngle(angle: float) +setActive(active: bool) int) +draw(s: pygame.Surface) [MoveableSprite]) Player, pos: (float, float), owner: Player, colour: +getTurretAngle(): float +isActive(): bool +vanish() +vanish() int) +getTurret(): Turret +update(time_passed: int) magnitude: int) 1 +explode() +getPos(): (float, float) +turnTurretTo(target: +vanish(reappear: bool) +explode() Tombstone +setAngle(angle: float) MoveableSprite) +affectPlayer(p: Player) +update(time_passed: int) causes 1 -image: pygame.Surface +getAngle(): float +refreshImage() +tick(time_passed: int) +vanish() -rect: pygame.Rect +setTurnSpeed(ts: float) +tick(time_passed: int) +TombStone(pos: (float, float), +getTurnSpeed(): float +update(time_passed: int) map: Map, containers: +turnTo(angle: float, ts: +vanish() [pygame.Group] float) +fire() FuelBarrel RepairShop Bullet +refreshImage() +isEnemy(other: +FuelBarrel(spsh: Spritesheet, +RepairShop(spsh: +Bullet(owner: Player) +update(time_passed: OwnedObject) g: Game, pos: (float, float)) Spritesheet, g: Game, pos: int) +isFriendly(other: +affectPlayer(p: Player) (float, float)) +fire() OwnedObject) +affectPlayer(p: Player)

Figure 5.11: Class diagram for Sprites Package 5.2 A Test Platform for Statechart AI 54

The largest package of the test platform is the sprites package. It contains all base classes for sprites and the actual sprites themselves. When using Pygame to render graphics, anything that should be able to move, scale or otherwise change its representation without altering its content, should be a subclass of pygame.Sprite. Furthermore, these subclasses are required to provide two attributes: an image attribute to use when rendering and a rect attribute to specify the position for the rendered image on the display surface. Let us now go over the classes in this package brieﬂy, to understand how they relate to the player and the ﬂow of the game in general.

MoveableSprite This base class should be inherited by any other class that requires its visual representation to be moveable across the game area. The class implements location, movement, velocity, rotation and rotational velocity. Furthermore, it also provides the functionality to move towards a certain location on the map or turn towards a certain angle. More location-based convenience methods are available to facilitate higher-level polling of the entities on the map, relative to this instance of the MoveableSprite class.

CollideableSprite This class inherits from MoveableSprite and adds collision and clipping functionality to the pre-existing functionality. Clipping means that the object has moved of the boundaries of the map, and is handled differently than collisions. A collision will cause projectiles to explode, but clipping simply discards the object.

OwnedObject This class serves as a base class for any sprite that is owned by a player. Any sprites for projectiles ﬁred by a player and its tank and turret sprites are owned by the player. This allows us to easily check the relation of one sprite to another, because through their owners, we are able to check if they have a friendly or hostile relation.

Turret The class representing a player’s turret. It in fact does not constitute a sprite by itself, but forms a sprite in combination with an instance of the Tank class described in the next paragraph. It is an OwnedObject, and can be created in one of several colours, which are used to signal its association to a player team. The most important methods are update and fire. The update method updates the visual representation of the turret if required. Just like with tick methods of other classes discussed before, this is called every time-slice. Note however, that there is an important distinction between tick and update methods. Whereas tick is responsible for advancing the time and internal state of objects, update only deals with the visual representation. The fire method will create a Bullet and fire it in the current direction the turret is facing.

Tank This is the embodiment of a player’s pawn, in the form of a tank. It is created for a single player and can be created in any of a number of available colours, used to signal the player’s team. It 5.2 A Test Platform for Statechart AI 55

is bot an OwnedObject and a CollideableSprite and renders itself overlayed with its turret to the screen. Note that the Tank class contains both a tick and update method. Again, the distinction is that tick will update internal state and update will update the visual representation. For this class, the tick method will decrease the fuel level of the tank’s owner if the tank is moving during the current time-slice. The update method will simply call refreshImage, which will change the image attribute if the tank’s orientation or location have changed since the previous time-slice.

Spritesheet This class serves to parse and provide sprite sheets. A sprite sheet consists of an image file and its configuration file, which specifies where in the image the sprites are contained and provides identifiers and transparency levels for each sprite. More information about these files can be found in section 5.2.2.2. Sprites can be retrieved from an instance of this class by calling getSpriteImage and providing the identifier of the desired sprite as a parameter. A convenience method to create a completely tiled surface of a certain sprite is also available in makeTiledSurface, which requires the identifier of the sprite to use when tiling and the size of the surface that should be created.

TankBrigade This is simply a Spritesheet whose methods have been overridden to simplify loading the default sprite sheet of the test platform. Using the load method causes all sprites deﬁned in this sprite sheet to load, after which they can be retrieved as discussed in the previous paragraph.

Pickup This is the base class for all items that can be placed on a map and subsequently picked up by players. It is created using a Spritesheet instance to retrieve its visual representation, an instance of the current Game, identiﬁer for the sprite to use and a position on the map in the form of a tuple of ﬂoating point values. Classes that inherit from this base class can override the tick method to alter the pickups behaviour over time and respawn it after it is picked up. The affectPlayer method should be overridden in order to have some sort of effect on the player that has picked up the item.

FuelBarrel and RepairShop These two classes both inherit from Pickup and respectively replenish the fuel level and the health level of the player that has picked them up.

Projectile Projectiles are ﬁred by instances of the Turret class. They inherit from CollideableSprite and will explode on impact with the map or other sprites. When a projectile explodes, it creates an instance of the Explosion class at its current position. Note that projectiles are visible on to the player logic and are not just an animation that is played before damage is caused. 5.2 A Test Platform for Statechart AI 56

Bullet Bullets inherit from Projectile and set values concerning explosion radius and damage caused. This is the actual projectile type that is created by turrets.

Explosion An explosion is, as previously mentioned, caused when a Projectile collides with the map or another sprite. Explosions have a certain blast radius and magnitude. This magnitude directly determines the damage caused to any player that is within the blast radius of the explosion. A nice animation is played when an explosion is created. Objects of this class typically exist for no more than 2 seconds, just enough time to display the animation before being destroyed.

Luminosity and Light These classes are not currently in use in the version of the test platform submitted with this thesis. Nevertheless, they are included in the source and provide a very nice 2D lighting system for pixel perfect detection of sprites in the field of vision of players. Light objects cast shadows using the walls of the map, so the vision of players can be obscured. A single instance of Luminosity should be created in the Game instance to group all the lights together and draw them to the screen. Both of these classes have been replaced by much more efficient logic, but could be used for debugging purposes, since it allows the user to see the field of vision of each player.

HUD Element: StatBar When a player spawns, an instance of this class is created. It will render two horizontal bars directly below the player. The top bar is green and shows the health value of the player, the bottom bar is blue and shows the fuel level.

HUD Elements: Text and BoundingBox These ﬁnal two classes allow text to be rendered to the display, which is used in e.g. the Round class to display the time left for the current round at the top of the screen. The BoundingBox class is for debugging purposes and can show the bounding box of a player.

Package: Maps 5.2 A Test Platform for Statechart AI 57

maps

Map -config: {} -game: Game -blueprint: pygame.Surface -walls: [pygame.Rect] -ambient: int -surface: pygame.Surface +spritesheet: Spritesheet +collisionMask: pygame.Mask +area: (int, int) +Map(game, mapFile) -createGraph() -determineWalls() +collision(sprite, findTargets=True): (bool, colliders) +getNodePos(node): (float, float) +getPosNode(pos): (int, int) +getWalls(): [pygame.Rect] +getWallMask(): pygame.Mask +haveLOS(): bool +loadPickups() +playersAt(pos, radius=1): [Player] +render(): pygame.Surface +spawn(player): (float, float) +tick(time_passed: int) +unload()

Figure 5.12: Class diagram for Maps Package

The Maps package contains any logic concerning the content and visual representation of the maps on which rounds are played. It currently consists of a single class: Map. The class is designed to be extensible and versatile in use. Maps are always created using an instance of the Game and a mapFile, which is a string pointing to the configuration file for the map. More information about the configuration file can be found in section 5.2.2.2. The internal representation of the obstacles within a map is kept in an instance of pygame.Mask. This is an efficient way of storing pixel-perfect obstacle locations. However, since pixel-perfect navigation or collision detection is very expensive in CPU-time, some additional representations are available through other methods. These representations are created from the original Mask representation when the map is created and can be freely used by NPC Components that deal with the layout of the map. The following is a list of all representations provided by the Map class. pygame.Mask A pixel-perfect representation of the obstacles in the map. Should only be used sparingly, since iterating over this representation is very expensive. Can be retrieved using the getWallMask method. list of pygame.Rect A rectangular approximation of the original mask, retrievable by calling thegetWalls method.. This uses the get_bounding_rects method of the pygame.Mask representation to generate rectangular areas that overlap the original obstacle mask. All obstacles will be contained within at least one rectangle, but open space in the map might also be overlayed due to the rectangular approximation. 5.2 A Test Platform for Statechart AI 58 networkx.Graph A graph representation of the open space in the map. This representation can be retrieved by calling the createGraph method on the Map instance. The graph vertices represent tiles of open space and the edges represent navigation paths between these open spaces. This is the most efficient way to process the map obstacle data and can be used in efficient path finding algorithms such as A*.

Package: Lib

lib gamemath +distance(x, y): float +distance((x0, y0), (x1, y1)): float +distanceSprite(sprite0, sprite1): float +angle((x0, y0), (x1, y1)): float +angleSprite(sprite0, sprite1): float +rotate(image, angle): pygame.Surface +tracePoint(x0, y0, x1, y1, l): float +intersects((x0, y0), (x1, y1)): bool +area(a, b, c): float +isOnLeft(a, b, c): bool +isOnRight(a, b, c): bool +isCollinear(a, b, c): bool

Figure 5.13: Class diagram for Lib Package

The lib – or library – package is meant as a utility package, providing functionality used throughout the whole project. When considering only the test platform, without the additions made in the following chapters, it consists only of the gamemath module, which provides functions that aid with mathematical calculations required in various areas of the game. Functions for calculating distances, angles and orientation are available.

Events

EventQueue -deque: deque -_tickid: bool internal +EventQueue() representation +push(type: string, params=None) +get(type: string) Event +getAll() 1 * -_type: string +clean() -_params -_tickid +getType(): string +getParams(): any

Figure 5.14: Class diagram for EventQueue and Event 5.2 A Test Platform for Statechart AI 59

The logic for handling events in the test platform is contained within the two classes shown above. It is created for use in any part of the test platform, but is mainly used in the component layers, as it has been described in section 5.1. An EventQueue can be created anywhere and does not require any parameters to be supplied to its constructor. Creating events is as simple as calling the push method with two parameters: type, which must be a string and represents the type (or name) of the event that should be created, and params, which can be any variable the user desires. The last parameter params is optional. Getting the ﬁrst occurrence of an event type can be done by calling the get method with the appropriate type parameter. All events currently on the queue can be retrieved with the getAll method. Finally, to discard all events on the queue, the clean method can be used. There is one important remark that must be made about the functionality of this class. It uses a boolean tickid attribute to identify during which time-slice events were pushed onto the queue. At the end of each time-slice, events from the previous time-slice that were not retrieved are discarded. This ensures conformity to Statecharts semantics, while also ensuring the queue cannot overﬂow. The EventQueue class uses instances of the Event class as an internal representation for events created using the push method. This class simply stores the type and params values provided for the event and also keeps track of the time-slice the event was created in using the tickid attribute.

Main entry point: Game

Game -settings: dict -sessionconﬁg: dict -headless: bool -clock: pygame.time.Clock -screen: pygame.Surface -schedule: dict -timepassed: int +players: pygame.sprite.OrderedUpdates +pickups: pygame.sprite.RenderUpdates +hud: pygame.sprite.RenderUpdates +luminosity: Luminosity +Game(conﬁg) +after(time, function) +runAfters(time_passed) +time(): int +start()

Figure 5.15: Class diagram for Game

The Game class represents the main entry point into the test platform. Instances of the class can only be created from a configuration file, which must be supplied as a command-line argument. Details 5.2 A Test Platform for Statechart AI 60 concerning the configuration file are available in section 5.2.2.2 and a complete description on how to run the game from the command-line is given in 5.2.3. An instance of this class can be started using the start method. The class provides access to the game’s notion of time through the time method, which will return the number of milliseconds passed since the start of the game. Note that the value of time may be either real or simulated time, depending on the current command-line parameters. To have the game execute a certain function after a set amount of time has passed, users can supply this function and the requested delay in milliseconds to the after function. The function will then be executed in the time-slice where the provided delay has been reached, implying that the function might be executed a maximum of a full time-slice length later than the originally requested delay interval. To manually force execution of the functions scheduled with the after function, users can call the runAfters method with a time_passed parameter, which should be the number of milliseconds passed since the previous time-slice. Calling this method with a value of zero will never execute any function, since the game logic will have executed the functions scheduled for that time interval during the previous time-slice. The Game class keeps track of several pygame.Group instances to order and subsequently render all entities with a visual representation. The different groups are found in the attributes players, pickups, hud and luminosity. Note that the entities contained within these groups are only visual representations and no logic concerning internal state of these entities is directly run by the Game class. Logic dealing with internal state of the players is run by the Session class. The distinction of updating internal state and visual representation is made clear through the function names: tick updates internal state every time-slice and update updates visual representation.

5.2.2.2 Data Required by the Test Platform

The test platform requires a large collection of data and conﬁguration ﬁles in order to run. This section provides a comprehensive overview of all data required, together with a description of all data formats.

Configuration Data Format Before listing all required configuration files, we should discuss the data format in which they are created. Very early on, at the start of the implementation of the test platform, I decided upon using the old, but more recently popularised JSON1JavaScript Object Notation, more information available at http://www.json.org file format to store configuration data. It is a very simple data format, but allows for a very succinct description of complex data. Additionally, the JSON format is supported by the standard Python json package and de-serialises to Python dictionaries and lists in a very natural way. The notation of the configuration files and the Python syntax for the de-serialised data is very similar. JSON uses universal data structures to provide a very coherent and cross-platform file format. JSON files consist of objects, which may contain an arbitrary number of name, value pairs. These values may be another object or an array of values. An example JSON file can be seen in listing 5.1. For a complete description of the file format, please refer to the official page for the file format at http:// www.json.org/. 5.2 A Test Platform for Statechart AI 61

1 { 2 "firstName": "John", 3 "lastName" : "Smith", 4 " age " : 25 , 5 "address" : 6 { 7 "streetAddress": "21 2nd Street", 8 "city" :"NewYork", 9 "state" :"NY", 10 "postalCode" : "10021" 11 } , 12 " phoneNumber " : 13 [ 14 { 15 " t y p e " : " home " , 16 " number " : "212 555 −1234" 17 } , 18 { 19 " t y p e " : " f a x " , 20 " number " : "646 555 −4567" 21 } 22 ] 23 } Listing 5.1: JSON describing a person’s personal record

Game Configuration The game itself requires a small configuration file to determine some parameters required to run. It currently only contains the resolution the game should run at. Changing this value will cause the game to run in the set value, but new maps will need to be created to take advantage of higher or be compatible with lower resolutions. There is just one parameter present in the root object of this configuration file: resolution. Its value needs to be a list of length two, with integer values. The file needs to be placed in the root folder of the test platform and be named game.cfg. Listing 5.2 shows the standard game configuration file used throughout this thesis. 1 { 2 "resolution" : [640, 480] 3 } Listing 5.2: Game configuration file game.cfg

Session Configuration Each time the game is run, a session is started. The game runs for as long as the session has maps to play. Sessions are configured in their own configuration file, and the path to 5.2 A Test Platform for Statechart AI 62

this file needs to be provided as a command-line parameter when starting the game. More information about command-line parameters is found in 5.2.3. Each session configuration contains three parameters: maps, roundsPerMap and players. An example session configuration is given in listing 5.3

maps This parameter should have an array value. Each item in the array needs to be the path to a map configuration file. This path needs to be relative to the maps data directory and include the file extension.

roundsPerMap A simple parameter that speciﬁes the amount of rounds that need to be played every map. The value needs to be a strictly positive integer.

players This parameter serves to describe the players that should take part in the session. The value is an array of objects. The objects need to provide the following parameters:

name The player’s name, such that the player can be identified in the generated score files. The value should be a string. team The player’s team. Teams are integers and any integer value may serve to identify a team, no restrictions apply. config The player’s configuration name. A player configuration is a folder consisting of multiple data files, described in the paragraph Player Configuration. The value should be a string, representing the name of the player’s folder in the player data folder.

1 { 2 " maps " : 3 [ 4 "sand.cfg", 5 "interior.cfg", 6 "path.cfg", 7 "empty.cfg" 8 ] , 9 "roundsPerMap" : 3, 10 " p l a y e r s " : 11 [ 12 { 13 " name " : " P l a y e r A" , 14 " team " : 0 , 15 " c o n f i g " : " b a s i c " 16 } , 5.2 A Test Platform for Statechart AI 63

17 { 18 " name " : " P l a y e r B" , 19 " team " : 1 , 20 " c o n f i g " : " t e s t " 21 } , 22 { 23 " name " : " P l a y e r C" , 24 " team " : 2 , 25 " c o n f i g " : " r e s u l t " 26 } 27 ] 28 } Listing 5.3: Session conﬁguration ﬁle with 4 maps, 3 rounds per map and 3 players in 3 separate teams

Map Configuration Maps and all files required for their configuration should be placed in the data/maps directory of the test platform. At least two files are required to define a map, but more might be needed depending on the parameter values of the map configuration file. The map configuration file contains 5 parameters. An example map configuration file is given in listing 5.4, an example blueprint file can also be seen in figure 5.16.

sprites This parameter specifies the sprite sheet that should be used when rendering this map and any entities taking part in the game when this map is played. Its value should be a string, representing the path to the spritesheet configuration file.

blueprint A blueprint is the file that defines the obstacles on the map. Its value needs to be a string, representing the full filename of an image file in the same folder as the map configuration file. White pixels in the blueprint file will be interpreted as obstacles, while the pink areas represent open space.

ambient This parameter speciﬁes the ambient light level of the map. This can be used to correct the background image brightness level, or alter the way light sources affect the map when lights are used in the game. The value needs to be a positive integer in the interval [0,255].

image The image parameter’s value is again an object. It speciﬁes how the map should be rendered to the screen. The parameters that should be supplied in the object are: 5.2 A Test Platform for Statechart AI 64

type Either tiled or file. A tiled image is generated from sprites when the map is first loaded. A file image means the map should be rendered using a pre-existing image file. The choice of value for this parameter affects the subsequent parameters that are required. file Required only when a ’file’ image type was specified. The filename, including extension, of the image to use when rendering the map. This file should be present in the same folder as the map configuration file. background Required only when a ’tiled’ image type was specified. The name or identifier of the sprite that should be used as the background when rendering the map. The name specified here needs to be defined in the sprite sheet used by this map. walls Required only when a ’tiled’ image type was specified. The name or identifier of the sprite that should be used as the walls (or obstacles) when rendering a map. The name specified needs to be defined in the sprite sheet used by this map.

pickups The pickups parameter speciﬁes the types and locations of all pickups present on the map. Multiple entities of the same pickup type may be speciﬁed. This parameter’s value is an array of objects with the following parameters:

type The type of the pickup. In the current version of the test platform, allowed values are fuel and health. pos The position of the pickup on the map. The value must be an array of 2 integer values, representing (x, y) coordinates on the map.

1 { 2 "sprites": "tileset.cfg", 3 "blueprint": "interiormask.png", 4 "ambient": 150, 5 " image " : 6 { 7 "type":"tiled", 8 "background" : "dry dirt", 9 "walls": "cobblestones" 10 } , 11 " p i c k u p s " : 12 [ 13 { 14 " t y p e " : " f u e l " , 15 " pos " : [ 3 2 0 , 240] 16 } , 17 { 18 " t y p e " : " h e a l t h " , 5.2 A Test Platform for Statechart AI 65

19 " pos " : [ 4 9 4 , 120] 20 } , 21 { 22 " t y p e " : " h e a l t h " , 23 " pos " : [ 1 5 0 , 360] 24 } 25 ] 26 } Listing 5.4: Map conﬁguration ﬁle

Figure 5.16: interiormask.png, the blueprint ﬁle used in the example map conﬁguration above

Several maps are provided together with the test platform. The complete list of maps is as follows.

empty An empty map with two health pickups and one fuel pickup. This map allows testing of any behaviour without worrying about the interaction with obstacles on the map.

Figure 5.17: The empty map running in the test platform. 5.2 A Test Platform for Statechart AI 66 interior A simple map with some walls that create an interior space. This can be used for testing behaviour when it needs to interact with walls and ﬁnd its way around the map.

Figure 5.18: The interior map running in the test platform. path A more detailed map with more real-world scenario obstacles. It contains one health pickup and fuel pickup on opposite sides of the map.

Figure 5.19: The path map running in the test platform. sand Another detailed map with more realistic obstacles. The play area is a little smaller than the previous maps and a health pickup and fuel pickup are placed at together on the north end of the map. 5.2 A Test Platform for Statechart AI 67

Figure 5.20: The sand map running in the test platform.

Sprite Sheet Configuration Each map uses a sprite sheet to render itself. Tanks, turrets, projectiles and pickups all use sprites from this sprite sheet to provide their visual representations. A sprite sheet is in fact an image file containing various sprites overlayed on a colour that is usually interpreted as transparent. Sprite sheets for the test platform are simply placed in the data directory. They consist of two files: the configuration file and the image file. An example (trivial) configuration file is given in listing 5.5. The configuration file defines the sprites present in the image file and requires the following parameters. name The name of the sprite sheet as a string.

file The file name including extension of the image file for this sprite sheet. tilesize The size of each tile in the sprite sheet. key The colour that will be interpreted as transparent. The value should be a list of 3 integers, representing a colour in the RGB format. tiles This should have as a value an array of objects, where each object specifies everything needed to define a single sprite (or tile) in the sprite sheet. Each of these objects requires the following parameters.

name The name of this sprite, so that it can be retrieved by name at runtime. 5.2 A Test Platform for Statechart AI 68

x The x-coordinate on the image where this sprite can be found. Coordinates always point to the top left corner of a sprite. y The y-coordinate on the image where this sprite can be found. Coordinates always point to the top left corner of a sprite.

1 { 2 "name" : "Basic Tileset", 3 "file" : "tileset.png", 4 "tilesize" : 32, 5 "key" : [238, 0, 255], 6 " t i l e s " : [ 7 { 8 "name":"topleft wall", 9 " x " : 32 , 10 " y " : 32 11 } , 12 { 13 "name":"turret pink", 14 " x " : 98 , 15 " y " : 32 16 } 17 ] 18 } Listing 5.5: Spritesheet conﬁguration ﬁle

Player Configuration Each NPC requires a controller configuration file and component configuration files for each component used in the behaviour description. Furthermore, for each component model used by the components, there needs to be a component model file generated by the SCC compiler. A player configuration is a directory. The name of the directory serves to identify that player configuration in the session configuration file. Each such player configuration directory contains a components directory and a controller.cfg file. Let’s first take a look at the parameters required in the controller configuration file, before we take a look at what files are required to specify the various components used. A controller configuration is in fact a description of all layers and components belonging to these layers. Each parameter in the root object of the file has a name corresponding to the layer name. The value for each layer parameter is an object, containing as parameters the names of the components present in that layer. The values corresponding to these component names are again objects, with the following parameters to completely specify a single component: 5.2 A Test Platform for Statechart AI 69

type The class name of the component, such that it may be found and loaded at runtime.

config The name (including extension) of the component configuration file present in the player’s component directory.

arg Optional parameter. The arguments that need to be supplied to the component when it is ﬁrst created. Components do not get direct access to the complete controller, because this would allow a complete violation of the architecture as described in section 5.1. Instead, this parameter allows the user to specify a list of variables that need to be provided as parameters. The values accepted in this list are any reference to other components, in the form of layername.componentname. Any variable in the score of the base Component class is also allowed here, these include game (Game instance) and player (player instance). These arguments are required for e.g. the Radar component (described later in section 5.3.2), since it needs to be attached to some part of the tank under control by the player.

A short example for a controller conﬁguration ﬁle is given in listing 5.6 1 { 2 "first_layer" : 3 { 4 " h e a l t h " : 5 { 6 "type":"Health", 7 "config":"health_default.cfg" 8 } , 9 " r a d a r " : 10 { 11 " t y p e " : " Radar " , 12 "config":"radar_default.cfg", 13 "arg":["player.tank]" 14 } 15 } , 16 " s e c o n d _ l a y e r " : 17 { 18 " d e t e c t o r " : 19 { 20 "type":"InSightsDetector", 21 "config" : "insightsdetector_default.cfg", 22 "arg":["first_layer.radar"] 23 } 5.2 A Test Platform for Statechart AI 70

24 } 25 " t h i r d _ l a y e r " : 26 { 27 " p i l o t " : 28 { 29 " t y p e " : " P i l o t " , 30 "config":"pilot_default.cfg" 31 } 32 } , 33 " f o u r t h _ l a y e r " : 34 { 35 " mover " : 36 { 37 " t y p e " : " Mover " , 38 "config":"mover_default.cfg" 39 } , 40 " s h o o t e r " : 41 { 42 " t y p e " : " S h o o t e r " 43 "config":"shooter_default.cfg" 44 } 45 } 46 } Listing 5.6: Controller conﬁguration ﬁle

Now that we know how the controller configuration file looks like. Let’s look at the contents of the components directory required for each player configuration. The directory should contain for each component used in the behaviour description of the player, a component configuration file and a component model file. The component configuration file requires at least two parameters. For any extra value required by the component, an extra parameter is required. An example component configuration file is given in listing 5.8. It contains two extra values: maxHealth and lowHealth.

component The component’s class name as a string. This is also required in the component conﬁguration ﬁle, such that components may be loaded outside of the context of a complete player for debugging purposes.

model The component’s model class name. This should be a class generated by the SCC compiler, present in a python module of the same name as this class in the components directory of this player. 5.2 A Test Platform for Statechart AI 71

extra values Any extra values required by the component must be speciﬁed as separate parameters. The parameter name should be the value name expected by the component and the value can be anything allowed in the JSON syntax.

1 { 2 "component" : "Health", 3 "model" : "health_model", 4 "maxHealth" : 100, 5 "lowHealth" : 30 6 } Listing 5.7: Component conﬁguration ﬁle for a Health component

5.2.2.3 Dependencies

The test platform as it exists right now uses some third party packages to created all the desired functionality. This section lists these packages and their versions such that readers may be able to ﬁnd these later and run the test platform, should future versions of these packages prove to be incompatible with my code. I used Python 2.7.3 on a Mac OSX 10.7 machine to develop this software. I have tested it on various Linux distributions, but can offer no guarantee that the software will work as described on any platform. It should work ﬁne, provided the correct version of Python and the third party dependencies are used. The following third party packages are used by the test platform:

Pygame 1.9.1 is used as the 2D graphics library. It is required even when the test platform is executed in headless mode, since the library is also used to load images used as masks for the maps and collision meshes. Networkx 1.7 is used to create a graph representation of the maps. The graph representation is provided in order to aid the creation of pathﬁnding components.

5.2.3 Usage

The test platform can only be used if the required data has been provided. This data is described in section 5.2.2.2. One the required data is present, the test platform can be run from the command line. The test platform has been tested on a 32-bit machine, which means I cannot guarantee its functionality on 64-bit machines as of yet. However, if a 64-bit python environment and 64-bit versions of all dependencies are available, the test platform should function ﬁne on this platform too. The test platform’s game.py module serves as the main entry point into the test platform and should be run using Python 2.7.3 as follows: 5.2 A Test Platform for Statechart AI 72

Parameter Default Value Description -s, –session defaultsession.cfg The session configuration file to use during this execution of the test platform -o, –output scores.json Path to the file where the score output file should be written. -hl, –headless N/A If this parameter is provided, the test platform will run in headless mode, running simulated time as fast as possible.

1 python game.py

Listing 5.8: Component conﬁguration ﬁle for a Health component

When no command-line parameters are specified, the game will start in visual mode with a default session file located in the root directory of the test platform and named defaultsession.cfg. In order to run the test platform using your own session files, you will need to provide the appropriate command line parameter(s). The following is a list of all command-line parameters that can be set and their meaning.

As Test Platform

The test platform can of course be used to test any player behaviour you develop by placing the player configuration files as described in 5.2.2.2 in the data/players directory. By creating an appropriate session configuration file, the test platform can be used to check how the developed behaviour performs on a variety of maps.

As Analysis Platform

The test platform can be used as an analysis platform by providing the -hl parameter to run the test platform in headless mode. This will cause the game’s notion of time to become simulated. Time slices of 35 milliseconds will be used, but the game will not wait for the actual 35 milliseconds have passed. Instead, the game passes the 35 milliseconds as the time_passed parameter to the tick and update methods of the appropriate classes. When scheduling actions to be performed in the Statecharts formalism using the AFTER macro, the scheduled time will likewise be treated as simulated time. In essence, exactly the same scenarios can play out as when the test platform is not running in headless mode, but headless mode will run all logic as fast as your machine allows. In order to now analyse the performance of the created behaviour, you can use the generated scores output ﬁle. This ﬁle will provide several metrics for each player, per round that has been played. It is intended to be used for analysis and does not process the saved metrics in any way. The gathered data is simply extracted from the internal state of the test platform and saved to disk. 5.3 Creating NPC Behaviour Using Statecharts 73

5.3 Creating NPC Behaviour Using Statecharts

We now have everything that is required to start creating NPC behaviours and testing them using the test platform. The general design from 5.1 will guide us in designing the layers and components and the test platform developed in 5.2 provides us with the implementation of the design required to actually run the created behaviours.

5.3.1 Determining the layers and their function

We will develop a full description of the behaviour of an NPC that takes control of a tank in the test platform. To this end, we will adopt the eight layered approach also found in the related work by S. Blanch in [Bla08]. The layers, in order from the input to the output side, are as follows. We will also provide a short description for each layer below. For a more detailed description of these layers, please refer to section 4.1.1.1, which discusses the same layers in reference to their inception in the related work.

1. Sensors 2. Analyzers 3. Memorizers 4. Strategic Deciders 5. Tactical Deciders 6. Executors 7. Coordinators 8. Actuators

We should define the functionality of each layer before we create the components within. To this end, we have a clear picture of what the components in each layer should be responsible for, and what functionality they will need to delegate to components in subsequent layers. It is good practice to review the functionality of the layers and make sure no layers overlap in functionality or have too big of a responsibility. In both cases, the chosen number of layers probably does not translate to a natural description of the NPC behaviour. It is very important to take your time during this first step, as later modifications to the amount of layers might cause problems with the code and models you have developed for your components.

Sensors

The sensors layer is solely responsible for polling the game and player states, such that subsequent layers may act upon this information. In essence, this layer forms the input to the behaviour description of the NPC. Information such as a player’s health might be retrieved using a Health component. 5.3 Creating NPC Behaviour Using Statecharts 74

Analyzers

Analyzer layer components interpret the data gathered by the sensors layer and perform additional computation to better understand the current state of the game. Analyzers may interpret radar sensor data and provide information about what, if any, is in the line of sight of each radar.

Memorizers

This layer is responsible for storing information retrieved during a time-slice and making it available again in the future. For example, when a player moves about a map, a memorizer component could be responsible for remembering where the player has seen obstacles. That way, the player will gradually get to know the map and be able to navigate it better over time.

Strategic Deciders

Strategic deciders interpret the state of the game, which was retrieved, analysed and remembered by the previous three layers, and decide upon a high level goal. This layer signiﬁes the switch between handling retrieving information about the environment and signalling reactions to this information. High level goals could be exploring the environment, engaging a group of enemies, ﬁnding pickups, etc.

Tactical Deciders

Responsible for determining how the high level goals decided upon in the previous layer might be executed. There is typically one tactical decider component for each high level goal that can be decided upon. Of course, this is no explicit rule and the user is free to create as many tactical decider components as they wish. When the high level goal was to explore the map, a tactical decider component might generate a location to move towards (without dealing with the path towards this destination or any other more concrete decisions).

Executors

This layer has the responsibility of translating the events generated by the tactical deciders layer into events that can be interpreted by the actuators or coordinators. This means that this layer must decide upon low level, short term goals. To continue with the previous example, when a tactical decider has decided that the player should move towards some location, an executor component might attempt to ﬁnd a path to this destination, generating events for every waypoint on the path towards that destination until it has been reached.

Coordinators

Since there may be multiple high level goals decided upon by the strategic and/or tactical deciders, the executors might generate conflicting or non-efficient events. This layer is responsible for solving that problem and generating the most efficient events for the execution of the goals set forth by the previous layers. 5.3 Creating NPC Behaviour Using Statecharts 75

Actuators

Finally, the actuators are responsible for actually executing the actions requested by the events generated by the previous layers. An actuator component could e.g. be the motor of a tank, which can move forward or turn left or right.

5.3.2 Sensor components

5.3.2.1 Health

Health +maxHealth: int +lowHealth: int +isBelowPct(value: ﬂoat): bool +isBelow(value: int): bool

Figure 5.21: Health Component Class

Figure 5.22: Health Component Model

The Health sensor component takes care of polling the player for its health value. When we design the Health component class, we must make sure all methods are available to make this possible. Be- cause each component has access to a player attribute, the health component class does not require any additional work to be able to poll for the player’s health value. This can be done using the getH- ealth method from the Player class. The component class does provide two new attributes, namely 5.3 Creating NPC Behaviour Using Statecharts 76 maxHealth which denotes the maximum health value for the player and the lowHealth value which denotes a threshold for when the component believes the player’s health to be ’low’. The class also provides two methods – isBelowPct and isBelow to compare the player’s health value to some other value. The component model consists of the compound state Health, which contains all other states in the model. The default state is HighHealth, which denotes that the player’s health is high enough. Each second, the transition from HighHealth to LowHealth will attempt to fire, but it will only actually fire when its guard evaluates as true. The guard checks if the player’s health is below the value self.lowHealth. If that is the case, the transition is taken and the event HEALTH_LOW is pushed. When in the state LowHealth, the transition to HighHealth will attempt to fire each second, but it will only succeed if the player’s health is equal to its maximal health value, as can be seen in the guard condition. When the transition is taken, the event HEALTH_MAX is pushed.

5.3.2.2 Fuel

Fuel +maxFuel: int +lowFuel: int +isBelowPct(value: ﬂoat): bool +isBelow(value: int): bool

Figure 5.23: Fuel Component Class

Figure 5.24: Fuel Component Model 5.3 Creating NPC Behaviour Using Statecharts 77

The Fuel component functions very similarly to the Health component. The only notable difference is that it of course polls for a different attribute of the player attribute. The maxFuel and lowFuel attributes provide the same function as their Health component counterparts. The exact same methods isBelowPct and isBelow are again made available to check if the fuel values is below some other value. The Fuel component model also consists of two states, now HighFuel and LowFuel. When the fuel level decreases below self.lowFuel, the transition from the state representing a high fuel level to the state representing a low fuel level is taken. The opposite transition is taken when the player’s fuel level reaches self.maxFuel.

5.3.2.3 Radar

Radar +range: int +fov: ﬂoat +fovr: ﬂoat +attachedTo: MoveableObject -visibles(sprites): [pygame.Sprite] +scanEnemies(): bool +scanFriendlies(): bool +scanNodes(): bool +scanWalls(): bool +scanPickups(): bool

Figure 5.25: Radar Component Class 5.3 Creating NPC Behaviour Using Statecharts 78

Figure 5.26: Radar Component Model 5.3 Creating NPC Behaviour Using Statecharts 79

The Radar component allows players to visually experience the map and its denizens. A radar must always be attached to some part of the player’s tank. There are two distincts parts of the tank: the tank body and the tank turret. Whichever part the radar is attached to, it will be available as the attribute attachedTo. The radar component also contains two extra values: range which specifies the maximum distance from which the player can see objects and fov which specifies the field of vision provided by the radar (in degrees, fovr provides the same value in radians). The radar component class provides several methods to scan the map for enemies, friendlies, nodes, walls and pickups. The Radar component model consists of four orthogonal components, each responsible for scanning for a distinct type of entity: enemies, friends, pickups or walls. The orthogonal components for enemies, friends and pickups consist of two states. The default state, which signifies that no object of that type is in view. Each second, the transition to the other state scans the field of vision and if it detects any object of the appropriate type in view, the transition is taken and an appropriate event is pushed. The exact opposite of this is performed every second and the default state will again be reached when no more entities of the appropriate type are seen. The orthogonal component responsible for scanning for walls on the map only consists of a single state. This is because we do not have to take any action if no walls are visible, but we need to update the internal representation of the map is a wall is visible. Therefore, we only need a single state with a transition to itself if walls can be seen. This component is used twice in most tanks. Once to create a Tank Radar and again to create the Turret Radar. The differences between both instances of this component lie in the extra values for range and field of view, and of course the value of the attachedTo attribute. This way, we use the exact same code and model, fully leveraging the code re-use made possible by the Statecharts powered behaviour description.

5.3.3 Analyser Components

5.3.3.1 InSightsDetector

InSightsDetector +radar: Radar +tolerance: ﬂoat -somethingInFront(sprites: [pygame.Sprite]): pygame.Sprite +tankInFront(sprites: [pygame.Sprite]): Tank +projectileInFront(sprites: [pygame.Sprite]): Projectile +enemyInFront(): Tank or Projectile +enemyTankInFront(): Tank +enemyProjectileInFront(): Projectile +friendInFront(): Tank or Projectile +friendlyTankInFront(): Tank +friendlyProjectileInFront(): Projectile +pickupInFront(): Pickup

Figure 5.27: InSightsDetector Component Class 5.3 Creating NPC Behaviour Using Statecharts 80

Figure 5.28: InSightsDetector Component Model

This analyser layer component is responsible for signalling subsequent layers that enemies have been sighted directly in front of some radar. It is meant to be used with the Turret Radar such that when this component detects an enemy, the turret must be aimed at the enemy, meaning that the turret can be ﬁred. The InSightsDetector component class requires a Radar component in order to be able to detect enemies. It is also conﬁgured with a tolerance attribute, which denotes the angle that is deemed to be the area directly in front of the radar. This is usually set to some low value like 3 degrees. The class further provides an array of methods to detect enemies, friends or pickups that can be used in the component model. The component model consists of two states in the main compound InSightsDetector state. The default state entered when the player is started is Seeking. This means the detector is waiting for enemies to appear in front of its radar. Each second, the transition to the state EnemyInSights checks its guard, the enemyTankInFront method of the component class. If an enemy tank is actually in front of the radar, given to provided tolerance, the transition is taken. The transition from this state to the Seeking state has a guard of not self.enemyTankInFront, so the default state will again be reached when an enemy is no longer in front of the radar. The EnemyInSights state has a transition from and to itself, which will be taken every second if an enemy is in front of the radar. The EnemyInSights state is entered by two different transitions, so it has an enter action to push the INSIGHTSDETEC- TOR_ENEMYTANK event. 5.3 Creating NPC Behaviour Using Statecharts 81

5.3.4 Memoriser Components

5.3.4.1 Player Tracker

PlayerTracker +radars: [Radar] +nrTrackedEnemies(): int +nrTrackedFriends(): int +getTrackedFriends(): Tank +getTrackedEnemies(): Tank +getClosestTrackedFriend(): Tank +getClosestTrackedEnemy(): Tank +refreshEnemyPositions(): int +refreshFriendPositions(): int

Figure 5.29: PlayerTracker Component Class

Figure 5.30: PlayerTracker Component Model 5.3 Creating NPC Behaviour Using Statecharts 82

The PlayerTracker component is responsible for keeping track of all the players detected by the radars that are supplied to it. Usually all radars present on the tank are supplied. The component class provides method to get the number of tracked friends and enemies and retrieve the closest friend or enemy. Furthermore, two methods are supplied to refresh the positions of friends and enemies. These need to be called whenever the radars should be polled for the presence of friends or enemies. The component model consists of the large PlayerTracker compound state, in which two orthogonal components are found. The topmost orthogonal component is responsible for tracking enemies and the bottom orthogonal component for tracking friends. Each consists of two states, where the default state is the one denoting that no enemies or friends are currently tracked. As soon as the event RADAR_ENEMY or RADAR_FRIENDLY is received, the transition to the second state is taken and the event PLAYERTRACKER_ENEMY, respectively PLAYERTRACKER_FRIEND is pushed. When in the ’tracking’ state, each half second, the appropriate method to refresh the positions is called. When it returns a value of zero, this signiﬁes that no more enemies or friends are tracked and the transition back to the default state is taken. The transition also causes the event PLAYERTRACKER_NOENEMY or PLAYERTRACKER_NOFRIEND to be pushed.

5.3.4.2 Map Discoverer

MapDiscoverer +radars: [Radar] -createMapGraph() +getNodePos(node): pos +getPosNode(pos): node +getRandomNode() node +updatePickupPositions() +updateWalls(): int +knowFuelPosition(): bool +knowRepairPosition(): bool +repairShops(): [RepairShop] +fuelBarrels(): [FuelBarrel]

Figure 5.31: MapDiscoverer Component Class

Figure 5.32: MapDiscoverer Component Model 5.3 Creating NPC Behaviour Using Statecharts 83

The MapDiscoverer component makes it possible for the player to gradually learn about the map. It saves all obstacles encountered and the pickups that have been seen, so that the player can move towards them when required. Like the PlayerTracker component, this component also uses radars to be able to see the environment. Usually it is provided with all radars on the tank. The component class provides the methods updateWalls and updatePickups to respectively update the positions of walls and pickups from the current ﬁeld of vision of the radars. More methods are provided to check if the positions of the various pickup types are known, retrieve all known pickups and to retrieve a random node from the map. The component model consists of only one state: Discover. Two transitions go from and to this state and are triggered by events pushed by the Radar components. The transition triggered by RADAR_PICKUP calls the method refreshPickupPositions method, which checks if there new pickups are discovered and it pushed the appropriate event. The other transition has a guard of self.updateWalls() > 0, which means the transition is taken only when a new wall has been discovered. If that’s the case, the event MAP_FOUNDWALLS is pushed.

5.3.5 Strategic Decider Components

5.3.5.1 Pilot

Figure 5.33: Pilot Component Model 5.3 Creating NPC Behaviour Using Statecharts 84

Pilot

Figure 5.34: Pilot Component Class

As the only strategic decider component, the Pilot represents – as the name implies – the decision processes of the tank’s pilot. It decides on the highest level objectives that need to be completed. As such, it does not poll the game or player for any values and only responds to and pushes events. The component class thus has no attributes or methods that need to be accessible to the component model. The Pilot component model consists of a large compound state, Moving. When the model is in this state, the player can move about the map freely without worrying about the fuel level. When the game is started (after the START event is received), the Moving state is entered, inside this state compound state Fighting is the default. Within this state, the Explore basic state is the default. When this state is active, the player is free to explore the map. The enter action of the Explore state pushes the event EXPLORE. The exit action pushes the event STOP_EXPLORE. When the event PLAYERTRACKER_ENEMY is received, the transition to the state Engage is taken. This state’s enter action pushes the event ENGAGE to the layer’s event queue. The state’s exit action pushes the event DISENGAGE. The transition back to the Explore state is taken when the event PLAYERTRACKER_NOENEMY is received. Upon receiving the event HEALTH_LOW, the compound state Fighting is left and the state StayAlive is entered, the transition from this state back to Fighting is taken when the event HEALTH_MAX is received. Finally, when the FUEL_LOW is received, the transition from the large compound state Moving to the state StayFueled is taken. When FUEL_MAX is received, a transition back to the deep history state of the Moving compound state is taken. This means that when the fuel level is replenished, the player resumes any strategy that was previously decided upon. On the other hand, when the health level is replenished, the player starts exploring, since the history state is not used and the Explore state is the default in the Fighting compound state.

5.3.6 Tactical Decider Components

5.3.6.1 Explorer

Explorer +mapdiscoverer: MapDiscoverer +findNewDestination(): (float, float) +destinationExists(): bool +reachedDestination(): bool

Figure 5.35: Explorer Component Class 5.3 Creating NPC Behaviour Using Statecharts 85

Figure 5.36: Explorer Component Model

The Explorer component is responsible for exploring the map, in order to learn about its layout and hopefully find the other players taking part in the game. The component class requires a MapDiscov- erer component in order to retrieve information about the layout of the map as it is learned by that component. The component class also provides some methods for the component model. The findDes- tination method will find a new location on the map to move towards. It uses the location of obstacles to find an open space on the map and returns the location as a tuple of floats. The destinationExists method returns whether or not the current location is still an open space, as this might no longer be the case if the MapDiscoverer has discovered new obstacles on the map. Finally, the reachedDestination method returns true if the current destination has been reached. The component model starts off by entering the NotExploring state. When the EXPLORE event is received after it is generated by the Pilot, the compound state Explore is entered and the event SWEEP is pushed to start sweeping the surrounding area with a radar. When the event STOP_EXPLORE is received, the transition from Explore to NotExploring is taken and the events STOP_MOVING and STOP_SWEEP are pushed. The default state of the Explore compound state is NoDestination, which signifies that the map should be explored but there is currently no destination to move towards. The transition from NoDestination to MovingToDestination is taken immediately after this default state is entered. It finds a new destination and pushes the event MOVE_TO with as parameter the destination found. The transition in the other direction has a guard which evaluates to true if the destination has been reached or the destination no longer exists. Each seconds, an attempt is made to take this transition. When it is eventually taken, the event STOP_MOVING is pushed. 5.3 Creating NPC Behaviour Using Statecharts 86

5.3.6.2 Engager

Engager +mapdiscoverer: MapDiscoverer +playertracker: PlayerTracker +currentTarget: Tank +keepAtDistance: int +chooseTarget() +distanceToTarget(): ﬂoat +reachedDestination(): bool

Figure 5.37: Engager Component Class

Figure 5.38: Engager Component Model

The Engager component is responsible for attacking any enemy that can be seen. It requires information concerning the current map and the players that have been encountered. To that end, the 5.3 Creating NPC Behaviour Using Statecharts 87 component requires an instance of both the MapDiscoverer and the PlayerTracker component. The current target is saved in the currentTarget attribute of the class. The extra configuration value keep- AtDistance is also provided, which denotes the distance which should be kept between the player and the current target. The method chooseTarget finds the closest enemy to the player and saves it in the currentTarget attribute. Using the distanceToTarget method, the distance to the current target can be retrieved. Finally, the reachedDestination method returns true when the player The component model enters the large compound state Engager after the game is started. It has one basic, default state Idle. There is one transition from this state to the compound state Engaging which is taken when the event ENGAGE is received. The Engaging state is left when the event DISENGAGE is received. The compound state Engaging consists of two orthogonal components. The first and topmost orthogonal state in the model is responsible for aiming at and shooting the current target. The default state of this orthogonal state is AimAt, which is re-entered each time the event PLAYERTRACKER_ENEMY _MOVED is received. The state has an enter action which pushes the event AIM_AT with as parameter the position of the current target. When the event INSIGHTS_ENEMYTANK is received, we know that the current target is in the sights of our turret and the Shoot state is entered. Its enter action pushes the event SHOOT. After one second, the transition back to state AimAt is taken. The second orthogonal state, displayed at the bottom of the model, is responsible for following the current target when it moves. It contains only one state, Follow. The state has two transitions to itself which it will attempt to fire every half second. One transition has a guard that evaluates to true if the distance to the target is smaller than the keepAtDistance attribute specifies. If that transition is taken, the event STOP_MOVING is pushed. The other transition’s guard evaluates to true if the distance is larger than the keep-at distance, taking this transition causes the event MOVE_TO ,with as parameter the current position of the target, to be pushed.

5.3.6.3 Stay Alive

StayAlive +mapdiscoverer: MapDiscoverer +shop: RepairShop +closestRepairShop(): RepairShop +reachedDestination(): bool

Figure 5.39: StayAlive Component Class 5.3 Creating NPC Behaviour Using Statecharts 88

Figure 5.40: StayAlive Component Model

The StayAlive component is responsible for finding a health pickup when the player’s health has reached a low value. It attempts to move towards a health pickup if one has already been discovered. If no health pickup is known, it will move about to map in an attempt to find one. Because the pickup location information is gathered by the MapDiscoverer component, this component needs a reference to such an instance to gain access to these positions. If a health pickup is known and is currently being moved towards, it is saved as the attribute shop. The method closestRe- pairShop attempts to find the closest health pickup to the player and the reachedDestination method is able to signal if the current destination (the health pickup or a location where it is searching for one) has been reached. When the game is started, the component model enters the large StayAlive compound state, which has a single default state Idle. As long as this state is active, the component does nothing. If the event STAYALIVE is received, the transition to the compound state Heal is taken. When the event STOP_STAYALIVE is received, the opposite transition back into the idle state is taken. The compound state Heal again has one, default, state FindRepair. This state denotes that a healing pickup should be found so that it can be used. There are two transitions leaving from this state. If the shop attribute is set, the transition to the state MoveToRepair is taken. The enter action for this state sets the destination and pushes the event MOVE_TO with as parameter the position of the health pickup. The transition from FindRepair to SearchForRepair is taken only when the shop attribute has not already been set. 5.3 Creating NPC Behaviour Using Statecharts 89

This state’s enter simply ﬁnds a random location on the map and moves towards it by pushing the event MOVE_TO with as parameter the randomly chosen position. Each second, the transition to the MoveToRepair is attempted. It will be taken if its guard evaluates as true, which will only be the case if the map discoverer knows the location of at least one health pickup. When the transition is taken, the closest of the known health pickups is saved in the shop attribute. Finally, each half second, the transition from the compound state Moving to FindRepair is attempted. It will only be taken if its guard, which checks if the current destination is reached, succeeds.

5.3.6.4 Stay Fueled

StayFueled +mapdiscoverer: MapDiscoverer +barrel: FuelBarrel +closestFuelBarrel(): FuelBarrel +reachedDestination(): bool

Figure 5.41: StayFueled Component Class

Figure 5.42: StayFueled Component Model 5.3 Creating NPC Behaviour Using Statecharts 90

The StayFueled component implements the same behaviour and provides the same data as the Stay- Alive component. The only difference is that this component checks, searches and moves towards fuel pickups. We could not use the same component class or model, since the class types, events and method calls differ, but the principles are exactly the same. The reader can refer to the description of the StayAlive component for an explanation of how this component functions, since its inner workings do not differ from this component.

5.3.7 Executor Components

5.3.7.1 Movement Executor

MovementExecutor +mapdiscoverer: MapDiscoverer +currentPath: [(float, float)] +currentTarget: (float, float) +findPath(pos) +haveWaypoint(): bool +getNextWaypoint(): (float, float) +atWaypoint(): bool

Figure 5.43: MovementExecutor Component Class

Figure 5.44: MovementExecutor Component Model 5.3 Creating NPC Behaviour Using Statecharts 91

This component is responsible for executing the various movement-related objectives. Since it deals with movement, the component class requires a reference to a MapDiscoverer component. The class furthermore provides access to the current destination through the attribute currentTarget and the current path the player is on through the attribute currentPath. Some methods to create and check paths are provided. Method findPath will create a path and save it when a destination position is supplied to it. Methods haveWaypoint and atWaypoint perform checks on the path as their names imply. Finally the method getNextWaypoint retrieves the next waypoint along the path to the current destination. The component model starts off in the Idle state. This state is left and the compound state HavePath is entered when the event MOVE_TO is received. The transition causes the findPath method to be called with the parameters of the received event. When the HavePath state is entered, its default state FollowingPath becomes active. When there is at least one more waypoint along the path, the transition to the state MoveAlongPath is taken and the event TANK_MOVE_TO is generated with as parameter the next waypoint on the path. When the tank is at this next waypoint, the event TANK_ARRIVED is pushed and the transition back to FollowingPath is taken. While in this state, if the event MAP_ FOUNDWALLS is received, a new call to findPath is made, because the previous path might no longer be possible due to the discovery of new walls on the map. When either the event STOP_MOVING is received or no more waypoints are available on the path, a transition back to the Idle state is taken and the event TANK_ARRIVED is pushed.

5.3.7.2 Aim Executor

AimExecutor +direction: float +currentTarget: float +findDirection()

Figure 5.45: AimExecutor Component Class 5.3 Creating NPC Behaviour Using Statecharts 92

Figure 5.46: AimExecutor Component Model

The AimExecutor component takes care of translating high-level aiming-related objectives to lower level events. Its component class is quite simple, it contains only two attributes and a single method. The direction component contains the current direction the turret is facing at. The currentTarget contains the target direction to aim towards. By using the method ﬁndDirection, the direction the turret should aim at is retrieved using the current target value. The component model starts the game in the compound state AimExecutor, which consists of two states. Its default state, Aiming, is initially active and denotes that the turret should aim at a target. It is re-entered each time the event AIM_AT is received, which causes the event TURRET_AIM to be pushed with as parameter the value of the direction attribute. When the event SWEEP is received, the transition to the state Sweeping is taken. This state denotes that the turret should rotate in circles, such that the turret radar is able to view the complete surroundings of the tank. Every three seconds, the Sweeping state is re-entered through a self-transition so that the sweeping motion will continue. There are two ways this state can be left, either the event STOP_SWEEP or the event AIM_AT is received. In both cases, the state Aiming becomes active again and normal aiming behaviour resumes. 5.3 Creating NPC Behaviour Using Statecharts 93

5.3.8 Coordinator Components

5.3.8.1 Steering Coordinator

SteeringCoordinator +direction: ﬂoat +shouldTurnLeft(): bool +shouldTurnRight(): bool +isStraightAhead(): bool +isLeftOfTank(pos): bool +isRightOfTank(pos): bool +isInFrontOfTank(pos): bool +isBehindTank(pos): bool

Figure 5.47: SteeringCoordinator Component Class

Figure 5.48: SteeringCoordinator Component Model 5.3 Creating NPC Behaviour Using Statecharts 94

The Steering Coordinator is responsible for translating the higher-level events generated by the Move- mentExecutor into events that are efﬁcient and can be directly acted upon by the actuators. It determines how the actual movement goals can be achieved through turning and forward or backwards movement. In short, it coordinates the different possible movements the tank can make to achieve the determined goals. The component class keeps track of the current direction in the likewise named attribute. Several methods are provided to check if the tank should turn left, right or not at all for a given target position. The component model starts off the game in the compound state Steeringcoordinator and its default state Stopped. When the event TANK_MOVE_TO is received, the transition to the compound state OnTheMove is taken. The transition in the opposite direction is taken when the event TANK_ARRIVED is received. The default state in OnTheMove is the state DetermineHeading, which as its name implies, will determine the direction the tank must face in order to reach its destination. If the destination is straight ahead, the transition to the state MoveTowards is taken, which has an enter action that pushes the event TANK_FORWARD. If the destination is not in front of the tank, the transition from Deter- mineHeading to TurnDirection is taken. This state will determine through its outbound transitions to the Turning state the direction that the tank should turn to. Using the appropriate method calls in the transition guards, one of the two outbound transitions is taken and one of the events TANK_LEFT or TANK_RIGHT is pushed. Each 5 milliseconds, the transition from state Turning to MoveTowards will evaluate its guard, which checks if the destination is now in front of the tank. If it is, the transition is taken. Finally, if at any point while moving the event TANK_MOVE_TO is received, the compound state OnTheMove is re-entered in order to determine a new heading for the new destination.

5.3.8.2 Aiming Coordinator

AimingCoordinator +currentTarget: ﬂoat +shouldAimLeft(): bool +shouldAimRight(): bool +isStraightAhead(): bool

Figure 5.49: AimingCoordinator Component Class 5.3 Creating NPC Behaviour Using Statecharts 95

Figure 5.50: AimingCoordinator Component Model

The AimingCoordinator component provides the same functionality for aiming as the SteeringCoor- dinator does for moving. It will determine the best turn direction for the currently requested aiming direction. The component class keeps track of the current target direction in the attribute current- Target. It provides some methods to determine is turning should be done to the left or right or if no turning is required at all when the current direction is straight ahead. The component model starts off in the Aiming compound state and its default state is NewDi- rection. This state has three outbound transitions, one for each of the possible scenarios: the target direction means the turret should turn left, it means the turret should turn to the right or the turret should not turn at all. In the latter case, the state Aimed is entered and no further actions must be undertaken. If turning is required, one of the states TurnLeft or TurnRight is taken, which have an enter action that respectively pushes the events TURRET_LEFT or TURRET_RIGHT with as parameter the target direction. Both of the turning states have an outbound transition to the Aimed state. Each 0.2 seconds, the respective transition’s guard is checked, which will succeed if the target direction is now in front of the turret or the turret has turned a little bit too far in the determined direction. Finally, the whole compound state Aiming is re-entered when the event TURRET_AIM is received and the attribute currentTarget is then set the to parameter of this event. 5.3 Creating NPC Behaviour Using Statecharts 96

5.3.9 Actuator Components

5.3.9.1 Motor

Motor +speed: float +turnSpeed: float +forward() +backward() +idle() +left(angle: float) +right(angle: float) +straight()

Figure 5.51: Motor Component Class

Figure 5.52: Motor Component Model

The Motor component is responsible for calling the appropriate methods on the player’s tank in order to make it move. The component class contains two preset attributes, namely the speed at which the 5.3 Creating NPC Behaviour Using Statecharts 97 motor should be run and the turnSpeed at which the tank is able to turn. All methods provided by the class make the tank move, turn or stop movement and turning. Methods forward, backward and idle provide functionality to make the motor move. The methods left, right and straight make the tank turn (or stop turning). The component model consists of a single large compound state Motor which itself contains two orthogonal states. The leftmost orthogonal state is responsible for movement and the rightmost for turning. The movement orthogonal state starts off in the default state Idle. When either the event TANK_FORWARD or TANK_BACKWARD is received, this default state is left for respectively states Forward and Backward. From these two states, two outbound transitions exist. One transition leads back to the Idle state and is triggered when receiving the event TANK_STOP. The other transition leads to the state that denotes movement in the opposite direction and is triggered when the appropriate event is received as can be seen in the model ﬁgure above. Finally, the turning orthogonal state consists of a single state, ReadyToTurn. There are two self-transitions for this state, which are triggered by one of TANK_LEFT or TANK_RIGHT. When these transitions is taken, they call the appropriate methods on the component class with as parameter the event parameter that was received when the transition was triggered.

5.3.9.2 Turret

Turret +turnSpeed: float +left(angle: float) +right(angle: float) +straight() +shoot()

Figure 5.53: Turret Component Class 5.3 Creating NPC Behaviour Using Statecharts 98

Figure 5.54: Turret Component Model

The Turret component is responsible for aiming and shooting using the turret under the player’s control. The component class keeps track of one preset value, the turnSpeed at which the turret is able to turn. It furthermore provides methods to turn the turret left, right and make it stop turning through method straight. Finally, it provides the method shoot in order to ﬁre the player’s turret. The component model consists of one large compound state Turret which is made up of two orthogonal states. The topmost orthogonal state is responsible for turning the turret. Its default and only state is Straight. When the event TURRET_LEFT is received, a self-transition is taken and performs the action of calling the method left with the parameter provided with the event that triggered the transition. When the event TURRET_RIGHT is received, another self-transition is taken that calls the right method with the event parameter. The bottom orthogonal state is responsible for shooting the turret. It also contains only one state, CanShoot. It has a self-transition on the event SHOOT, which causes the method shoot to be called, ﬁring the turret. 6 Transformation of Statechart AI

The complete design for a layered architecture of components using Statecharts models to describe the behaviour of an NPC in a game has now been determined. On top of this, a full-ﬂedged test platform was created which can be used to test the created NPC behaviour visually or in an automated way for analysis purposes. Now that this new way of developing NPC behaviour is available, it is a good idea to explore the advantages offered by this approach further. The main advantage of the Statecharts modelling approach is that we can transform the created behaviour in several ways. Firstly, we can vary parameters to the various components which will cause variants of the base behaviour to emerge. This approach does not vary the behaviour itself, it does not alter logic inside the component class nor does it change the component model in any way. It does, however, alter the value of some thresholds or other parameters which are used in the decision making processes. Secondly, we can alter the composition of the components inside of their layers. When we develop a whole array of components, we do not always need to use all components to create a valid NPC. The more components we develop, the more different compositions of these components we can create, which will cause very different behaviours to emerge from the same base collection of components. Finally, we can transform the component models themselves using model transformations. This will affect the core of the behaviour descriptions and perhaps cause entirely new decisions to be made, which the developer did not necessarily intend. In what follows we will explore how why we would like each of these methods for transformation, what their main use is and how we should implement them. At the end of this section the concrete implementation as an addition to the existing test platform is discussed.

6.1 Parameter Transformation

Parameter transformation allows us to create variants of a single NPC behaviour description. When we parameterise the components, the decisions made by these components will (partly) depend upon the values of these parameters. In the previous section, we already introduced extra values to components. These values can be retrieved from parameters supplied with the component conﬁguration ﬁles. Introducing such parameters gives us a relatively easy, low overhead mechanism for transforming 6.1 Parameter Transformation 100 the components. The values for these parameters can be generated from probability distributions such as the popular gaussian or normal distribution. The use and a possible method for implementation was previously discussed in the chapter handling related work. Section 4.2.1.1 discusses an implementation of such a parameterisation of components. Let’s now take a look at how this is implemented in our test platform.

6.1.1 Implementation

The Python standard library includes the random package which provides functions to retrieve samples from a wide variety of probability distributions. The probability distributions offered by the test platform are directly linked to the functions provided by this standard Python package. The following probability distributions have been implemented. The required values for each probability distributions are given as well.

Distribution Type name Required values Description Random from randrange start, stop, step Start and stop respectively specify range the start and stop value of the random range. Step denotes the step size when the range is generated. A random entry in the generated range is returned. Choice from se- choice seq Seq must be a list of values. A sin- quence gle value randomly chosen from the sequence is returned. Uniform uniform a, b A and b respectively denote the start and stop values for the interval. A random value in this interval is returned. Triangular triangular low, high, mode Returns a random value n such that low <= n <= high, with the specified mode. Beta betavariate alpha, beta The beta distribution, alpha and beta parameters should both be strictly positive. Exponential expovariate lambd Lambd (lambda) is the inverse of the desired mean value and should not be zero. A value between zero and positive infinity is returned if lambd is positive, a value between negative infinity and zero when lambd is negative. 6.2 Component Structure Transformation 101

Gamma gammavariate alpha, beta This is the gamma distribution, not to be confused with the gamma function. Alpha and beta should both be strictly positive. Gaussian (or nor- gauss mu, sigma The gaussian distribution where mu mal) is the mean and sigma the standard deviation. Sigma should be strictly positive. Log normal lognormvariate mu, sigma The log normal distribution. When taking a natural logarithm of this function, the normal distribution with mean mu and standard deviation sigma is found. Sigma should be strictly positive. Normal normalvariate mu, sigma The normal distribution, equivalent to the gaussian distribution above, but a different implementation. von Mises vonmisesvariate mu, kappa The von mises distribution where mu is the mean angle expressed in radians. Mu should have a value between 0 and 2π. Kappa is the con- centration parameter, which must be positive. Pareto paretovariate alpha The Pareto distribution where alpha denotes the shape parameter. Weibull weibullvariate alpha, beta This is the Weibull distribution where alpha denotes the scale parameter and beta the shape parameter.

Table 6.1: All the probability distributions offered in the implementation of parameter transformation of the test platform

6.2 Component Structure Transformation

The first method to actually transform the behaviour description of NPCs themselves is implemented through component structure transformation. When a developer creates a wide array of components that each belong to some layer, we can start choosing components to fill up each layer and evaluate the performance of NPCs with this new structure of components powering their behaviour. This approach becomes more and more powerful in generating different NPC behaviours the larger the base collection of components is. It is important to have a good strategy to structure the compo- 6.2 Component Structure Transformation 102 nents in order to create valid configurations. Some components require references to others or might even be required for the whole NPC to be able to function. Take for example the Health sensor component. When this component is not present in the behaviour description, the NPC will never regard their own health in any decision that is made. This means the NPC will never attempt to replenish his health or flee other players when their health is full. The developer might not like this kind of behaviour in any generated NPC behaviour description and should have the ability to designate that component as required.

6.2.1 Implementation

The component structure transformation method has been implemented using a type of tournament to iteratively evaluate NPC descriptions and improve them. The developer is able to specify for each component whether or not it is required. If it is, it will always be included in the NPC behaviour description. Components that are not required are subject to a random sampling step from the base collection of all components. For each layer, we first add all required components. Subsequently, we add a random number of components, with a maximum of two components added to a layer in each iteration. To ensure that each layer contains at least one component, the random selection will select a minimum of one component for each layer that does contain any required components. When we have chosen components randomly, we should check each added component for its dependencies. This is required because the components may specify arguments that need to be supplied to them. These arguments may also be of the form layername.componentname to require that a specific component from the overall behaviour description is provided to it. Using the specified list of arguments for the component, we are able to check if that component was already added to the behaviour description. When it was not previously added, we must add it now or the component will not be able to be created. We now have a complete, valid behaviour description for an NPC. The developer can now specify how many of such behaviour descriptions should be generated as the base pool. This pool is divided into groups of a certain size. These groups are then each used to create a session for the test platform where each NPC in the group belongs to its own team. The test platform is run for each group of the current iteration. When the test platform has finished with a single group, the generated scores file is analysed according to a certain heuristic (described below). The value returned by this heuristic determines the relative performance of the NPCs compared against each other. The generation algorithm then finds the three best NPC according to that heuristic and merges them to create a new NPC behaviour description that consists of all components of these best three NPCs. At the end of each iteration, we end up with a new pool of NPC behaviour descriptions. The next iteration then again divides these new NPCs into groups to restart the whole process. This continues until a single NPC behaviour description is found. The output of this component configuration transformation method is an NPC behaviour description that resulted from merging the best NPC component configurations according to the supplied heuristic. This behaviour description can then be the subject of parameter transformation to generate variants of this behaviour. 6.3 Statecharts Model Transformation 103

6.3 Statecharts Model Transformation

The above transformation methods only vary values or the overall structure of the components in the NPC behaviour descriptions. The Statecharts model is never touched, which means the core behaviour description does not change. We can now leverage model transformations to alter this behaviour directly. In the extreme case, we could create a very large library of model transformations that are able to build complete component models from scratch. However, for the objectives of this thesis, we will use model transformations to subtly alter pre-created component models to observe new behaviour. The behaviour that results from this type of transformations might be unexpected by the developer, but can always be reduced to the result of combining a library of transformation rules with a heuristic that rewards certain behaviours and penalises others.

6.3.1 Implementation

I have implemented Statecharts model transformation of component models using the T-Core package developed by Eugene Syriani. More information regarding this package can be found at Eugene’s homepage at http://cs.ua.edu/~syriani/Research.aspx

Components

In order to be able to run transformation rules developed using T-Core on the component models, these models need to be compiled into the Himesis graph representation for models. When developing component models using the DChartsV3 formalism in AToM3, we can load the additional formalism HimesisCompiler to gain access to a Himesis compilation algorithm. When the component model is completed, pressing the compile to Himesis button will result in a ﬁle named Himesis.py to be generated. This ﬁle will now serve as the component model representation used by the transformation platform.

Transformation Rules

To create transformation rules for the transformation platform, we need to load the ramified versions of the DChartsV3 formalism and the MotifRule formalism together. Using the MotifRule formalism, we create a left-hand side and right-hand side for the transformation rule. We can then fill in both sides of the transformation rule with Statecharts entities to create the actual transformation rule. The MotifRule formalism further provides the ability to specify NAC’s for transformation rules. An NAC is a pattern that, when found, indicates that the transformation rule should NOT be applied. An arbitrary number of NAC’s may be created. When the transformation rule is complete, we must again load the HimesisCompiler formalism to compile the transformation rule to Himesis. The generated files should be placed in their own directory, with the same name as the one defined for the transformation rule. This allows the developer to use a single name to load all required files for a single 6.3 Statecharts Model Transformation 104 transformation rule.

Transformation Rule Application Strategy

The created transformation rules can be used as atomic rules or as sequential rules. Atomic rules are applied only once, on the first match of the left-hand side of the transformation rule to the component model. Sequential rules on the other hand are applied as long as matches are found in the component model. When the transformation platform generated component configurations, before a component is added, it will attempt to apply a number of transformation rules to the component model. It will respect the order in which they have been described in the player seed file, but a random subset of the declared transformation rules will be used. Due to this random sampling of a subset, perhaps no transformation rules are applied at all, or maybe all of the declared rules are applied. It is the responsibility of the heuristic to reward good components and thus good component models with a higher score for the player in which they are placed. By sampling the transformation rules randomly, we avoid the huge search space that would result from attempting to apply each and every permutation of the declared transformation rules, but we retain the ability to find component models that are consistently rewarded with higher scores by the heuristic.

6.3.2 Example Transformation Rules

For the results section of this thesis, a few example transformation rules were developed. We will go over these transformation rules, but only in the results section will we come back to how often they are chosen in the complete transformation process.

Compound State Reset

When a player is taking part in the game, the various component models will be in some combination of states. By introducing a self-transition on compound states of the component models which is triggered in a certain time-interval, we can effectively reset a part of the behaviour of the player. This will cause the player to, as they say, look at the situation with a fresh set of eyes. We expect this to be beneficial in a number of areas, e.g. when fleeing an enemy only to drive straight to another enemy that was not previously tracked. Resetting a compound state that deals with tracking players or fleeing, this behaviour could potentially be altered in a positive way. The transformation rule is implemented by placing a compound state on the LHS and adding a self-transition to this state on the RHS, with a trigger of AFTER(5).

Transition Delay

Several transitions do not have a trigger, but only a guard condition. We can delay the ﬁring of these transitions by introducing a trigger to the transition that delays when the transition is attempted. 6.3 Statecharts Model Transformation 105

Figure 6.1: Component Reset Rule

This could allow the player some more time before a decision is made, which might be useful in some scenarios. However, introducing a delay in decision making could also have a negative impact, as it might slow the player down and faster players might gain the upper hand. The transformation rule is implemented by placing two states in the LHS linked by a transition with an empty trigger ﬁeld. The RHS then contains both states and the transition, but the trigger ﬁeld now contains AFTER(1), giving this transition a delay of one second before its guard is evaluated.

Random Transition

We can create a transformation rule that introduces a random transition from one state to another. We should trigger this transition with a slight delay such that the other transitions also get a certain time interval in which they may be taken. If no other transitions can be taken in that time interval, this transformation rule effectively overrides the normal decision making process of the player. This could be useful, for example when the random transition gets placed between a state denoting ﬂeeing behaviour to a state denoting behaviour to pick up some health replenishing item. On the other hand, it has potentially very negative effects too, by negating decisions that need to be taken in order to survive for as long as possible. We will leave it up to the transformation platform to determine if this transformation rule will end up being used in the resulting player behaviour description(s). 6.3 Statecharts Model Transformation 106

Figure 6.2: Transition Delay Rule

Figure 6.3: Random Transition Rule

This transformation rule is implemented by placing two states in the LHS who are not linked with transitions. The RHS then contains both these states with a new transition from one state to the other. We trigger this transition with AFTER(1) to allow other transitions a short interval to get triggered. 6.4 Transformation Platform 107

6.4 Transformation Platform

The above methods for transforming the NPC behaviour descriptions have been implemented in a transformation platform. The various transformation methods are executed as described above. In order to use the transformation platform, it needs to be provided with a player seed file and component definition files, discussed in section 6.4.3. The design and implementation of the transformation is relatively simple. However, due to using the Himesis graph representation for component models, a new compilation algorithm to the DES file format accepted by SCC needed to be written. All new classes have been introduced as extensions to the pre-existing test platform, making it an all-in-one package for testing and generating NPC behaviour descriptions.

6.4.1 Goals

The primary of the transformation platform was to explore the advantages offered by the Statecharts modelling of NPC behaviour, through employing various transformation methods to alter, improve and vary the created NPC behaviours. Another goal for the transformation platform was to make it possible to write components and determine the layers in which they should be placed, but leaving it up to an automated generation algorithm to determine which components are required and which do not need to be present in the NPC behaviour description. This allows a developer to create a wide array of components with a lot of varying functionality. By supplying the transformation platform with a heuristic (described in 6.4.4), it can evaluate the performance of NPC behaviour descriptions and decide which descriptions were better than others, with respect to the provided heuristic. This way, the developer has complete control over the kind of NPC behaviours that are generated, by constructing a heuristic that rewards the desired behaviour. Both of the above goals have been achieved. The transformation methods are completely implemented and it is possible to generate valid NPC behaviour descriptions from a seed ﬁle, collection of component deﬁnitions and models and some supplied heuristic. This short exploration into one of the various advantages offered by this approach to NPC behaviour creation has shown that it is a very powerful approach, but a lot more can be done to leverage all advantages offered by Statecharts and the proposed component architecture. This is discussed in the Future Work section at the end of this thesis.

6.4.2 Design

As was previously mentioned, the transformation platform was developed as an extension to the test platform. Several classes have been added to the lib package to allow for the parsing of the newly required data files (discussed in 6.4.3) and to generate the DES files that are required by our Statecharts compiler SCC. Furthermore, a new entry point into the platform was created in the Generate class. Let’s take a look at these new additions one by one. The class diagram can be seen in figure 6.4. 6.4 Transformation Platform 108

ComponentParameter

This class serves to parse the distribution name and the required arguments and generate samples from the requested distributions. It implements the transformation method described in section 6.1 Parameter Transformation. Instances of the class are constructed using the parsed definition retrieved from a component definition file. Retrieving the appropriate distribution from the Python random package is performed using the method parseDistributionSpecifics. The method create will sample from the distribution and return the sampled value.

ComponentDeﬁnition

Component definition files are represented in the transformation platform using the Component- Definition class. Its constructor requires the path to the definition file in order to create an instance of the class. The method setModel allows the transformation platform to assign a specific Himesis graph component model to a component definition. Finally, the method createConfig creates a component configuration file as it was discussed in the previous chapter’s section 5.2.2.2.

ControllerConﬁg

This class represents a player controller’s configuration of layers and components contained within these layers. The class is constructed from a list of layer names and an identifier. This identifier is required in order to be able to export the controller configuration and all components it contains in a complete player configuration directory. The method addComponent allows the addition of components to the controller configuration, this is the method used in the Generator to fill in the behaviour description with components. Finally, the method createConfig is responsible for writing the controller configuration to a file.

DESGenerator

The class responsible to translate Himesis graph representations of component models to the DES ﬁle format accepted by the Statecharts compiler SCC. It uses a lightweight AST representation of the DChartsV3 AToM3 formalism to generate the DES format. The classes DChart, Composite, Orthogo- nal, Basic, History and Hyperedge together make up the AST representation. The DESGenerator class is constructed from a graph representation of a component model. Its method parse will translate the graph to the AST representation. The method generate is responsible for generating the actual DES ﬁle.

Generator

The main entry point and impolementation of the transformation platform. This class combines the previous classes to create a full-ﬂedged NPC behaviour generator. Instances of this class are created 6.4 Transformation Platform 109

from a configuration that is specified at the command-line. The method addDependencies uses a complete controller configuration and component definition to find the dependencies of the specified component definition. These dependencies are then added to the provided controller. The method group splits a list of generated player controller configurations into groups of the provided size n. Method parseSeed is responsible for parsing the player seed file. Transformations of a component model is performed by the method transformModel on the supplied component definition. The generateBasePool method is responsible for generating the base pool of player controller configurations which starts of the generation process. Finally, generation is started using the appropriately named start method.

6.4.3 Required Data

The transformation platform requires some data before it can do its job. The user must supply a player seed file and in the components directory, the user needs to place component definition files and all created component models. Let’s go over these files in order to understand how to create them.

Player Seed File

The player seed ﬁle speciﬁes all information that is required in order to start the behaviour generation process. Listing 6.1 shows an example player seed. 1 { 2 " game " : 3 { 4 "maps" : [ "sand.cfg", "empty.cfg" ], 5 "roundsPerMap":3 6 } , 7 " p l a y e r " : 8 { 9 "layers" : [ "sensors", "analyzers", "memorizers", " strategic_deciders", "tactical_deciders", "executors ", "coordinators", "actuators"], 10 "components" : [ "aimexecutor_definition.cfg", " aimingcoordinator_definition.cfg", " engager_definition.cfg", 11 "explorer_definition.cfg"," fuel_definition.cfg", " health_definition.cfg", " insightsdetector_definition .cfg", 12 "mapdiscoverer_definition.cfg"," motor_definition.cfg", " movementexecutor_definition.cfg", " 6.4 Transformation Platform 110

pilot_definition.cfg", 13 "playertracker_definition.cfg"," stayalive_definition.cfg", " stayfueled_definition.cfg", " steeringcoordinator_definition .cfg ", 14 "turret_definition.cfg"," tankradar_definition.cfg", " turretradar_definition.cfg" ] 15 } , 16 " t r a n s f o r m a t i o n " : 17 { 18 " r u l e s " : 19 { 20 " a r u l e s " : [ ] , 21 "srules":["composite_reset"] 22 } , 23 " b a s e P o o l " : 64 , 24 " g r o u p S i z e " : 8 , 25 "heuristic" : "basicheuristic" 26 } 27 } Listing 6.1: Example Player Seed File

The first information that needs to be supplied in this JSON file is the game object. This object serves to configure the sessions that will be run in the test platform. The parameter maps should be a list of map configuration files and the parameter roundsPerMap should be a strictly positive integer that specifies the number of rounds that need to be played on each map. Next, we need to supply all information required to create player configurations in the player object. The parameter layers should provide an ordered list of strings to declare the order and names of all layers for which the created components have been developed. The following parameter, components is an unordered list of component definition files. Finally, the transformation object specifies all information required to determine which transformations should be performed and how many iterations the transformation platform should perform. The child rules object contains two parameters: arules is an ordered list of atomic transformation rules that have been placed in the transformationrules directory of the transformation platform’s data directory. srules is an ordered list of sequential transformation rules. The order in which the rules are specified shows their precedence, this will be respected by the transformation platform. The basePool parameter denotes the size of the base pool of player configurations which starts off the tournament- based transformation algorithm. The following parameter, groupSize, specifies the size of groups of players that should be pitted against each other in the test platform. Finally, the heuristic parame- 6.4 Transformation Platform 111

ter should denote the class and package name of the heuristic to be used to evaluate generated NPC behaviour descriptions.

Component Deﬁnition Files

Because the transformation platform generates component configuration files, we now need to supply component definition files from which these concrete configurations can be generated. Listing 6.3 shows such a definition file. 1 { 2 "layer" : "sensors", 3 "name" : "fuel", 4 "component" : "Fuel", 5 "model_source" : "fuel", 6 "events_accepted" : [], 7 " e v e n t s _ p r o v i d e d " : [ "FUEL_LOW" , "FUEL_MAX" ] , 8 "required" : 1, 9 "parameters" : 10 [ 11 { 12 " name " : " maxFuel " , 13 "distribution":"gauss", 14 "mu" : 100 , 15 " sigma " : 20 16 } , 17 { 18 " name " : " lowFuel " , 19 "distribution":"gauss", 20 "mu" : 30 , 21 " sigma " : 5 22 } 23 ] 24 } Listing 6.2: Example Component Definition File

This JSON ﬁle should contain at the following information. The layer parameter should denote the name of the layer to which the described component belongs. The name parameter denotes the name of the component. This name should be unique in its layer and may not be shared with any other components belonging to the same layer. The component parameter should denote the class name of this component, required for loading the component class logic. Next, the model_source parameter should denote the name of the directory in the data/components directory which contains the Himesis.py ﬁle generated after your component model has been compiled to Himesis. The parameters events_accepted and events_provided parameters should list the appropriate events. These are used 6.4 Transformation Platform 112 for bookkeeping purposes. The required parameter may denote the number of instances of this component that are required in every generated player controller, although in the current transformation platform, each component may only be present once. The parameters object is a list of objects which specify the various component parameters as described in 6.1. Each parameter list object should contain a name and distribution entry. Furthermore, the arguments required by the chosen distribution should be provided. For the names of these arguments, refer to table 6.1.

6.4.4 Heuristics

The transformation platform uses heuristics which are placed in the data/heuristics folder to evaluate the scores file generated by the test platform. The user should take great care to develop appropriate heuristics. A default heuristic is provided which emphasises dealing damage and penalises taking damage, while slightly rewarding finding pickups and greatly awarding the length of time the player was alive during each round. Finding an appropriate heuristic is not a trivial task. The first question a user should ask themselves is what kind of behaviour should be generated. The created heuristic should then reward metrics that result from this behaviour and penalise metrics that result from observing behaviour that does not correspond to the desires of the user. This balance is quite difficult indeed, but the question of What is good behaviour? is equally difficult. Should players be rewarded when dealing damage directly, or should making it possible to deal damage be rewarded? Perhaps both? Creating heuristics is often a trial-and-error process, since often behaviour is generated that was unexpected, but conforms to the provided heuristic. Heuristics in the transformation platform are implemented quite simply. Any class with a constructor that does not require any arguments in its constructor and contains a method calculate(score: Score) can be used as the heuristic for the transformation platform. No super class is provided, meaning the user is free to create their own organisation of heuristic classes, with the full power of Python at their disposal.

6.4.4.1 Dependencies

The transformation platform introduces dependencies on some new third party packages in order to perform the required transformations. Like with the test platform, these are provided here such that future readers are able to retrieve these packages and run the code created during this thesis. I used Python 2.7.3 on a Mac OSX 10.7 machine to develop this software. I have tested it on various Linux distributions, but can offer no guarantee that the software will work as described on any platform. It should work ﬁne, provided the correct version of Python and the third party dependencies are used. The following third party packages are used by the transformation platform:

T-Core is used to save the component models and transformation rules and subsequently apply them. iGraph is used by the T-Core package and the DESGenerator to handle the component models in 6.4 Transformation Platform 113

Parameter Default Value Description -s, –seed defaultseed.cfg The player seed ﬁle to use during this execution of the transformation platform -v, –variants 1 Number of variants of the generated behaviour to create. -o, –optimize N/A If this parameter is provided, the transformation platform will generate more variants than requested and only keep the best performers of all generated variants.

their Himesis graph representation.

6.4.5 Usage

The transformation platform can be used when the required data described above has been placed in their appropriate directories. Once this data is present, the transformation platform can be run from the command line. It is started by running the generate.py Python package, which accepts some command line parameters which are explained below. The transformation platform should be run using Python 2.7.3 as follows:

1 python generate.py

Listing 6.3: Example Component Deﬁnition File

When no command-line parameters are speciﬁed, the transformation platform will attempt to use the ﬁle defaultseed.cfg as the player seed. It will furthermore assume that only 1 variant of the generated behaviour is required and that optimisation is required. 6.4 Transformation Platform 114

lib

ComponentParameter DESGenerator -definition: {} -graph: igraph.Graph -name: String -dchart: DChart -distribution: String +DESGenerator(graph: 1 -dFunc: Function igraph.Graph) -dParams: [float] +parse() +ComponentParameter(definition: {}) +generate(filename: String) +parseDistributionSpecifics() +create(): float 1 +__str__() DChart Hyperedge History * parameterised by -graph: igraph.Graph -dchart: DChart -dchart: DChart -children: [] -parent -parent -hyperedges: [HyperEdge] -graph: igraph.Graph -graph: igraph.Graph 1 -allNodes: [] -vertex: int -vertex: int ComponentDefinition +DChart(graph: igraph.Graph) -source +History(dchart: DChart, parent, -sourceFile: String +generate(desfile: File) -destination graph: igraph.Graph, vertex: int) -definition: {} +Hyperedge(dchart: DChart, parent, +generateDeclaration(desfile: File, 1 1 -layer: String graph: igraph.Graph, vertex: int) prefix: String) -name: String * +generateTransition(desfile: File) +generateActions(desfile: File) -componentClass: Class * +getHierarchicName(): String -arg: [Any Type] * -modelSource: String -requiredNum: int * -eventsAccepted: [String] Basic -eventsProvided: [String] Orthogonal -dchart: DChart -parameters: [ComponentParameter] -dchart: DChart -parent +ComponentDefinition(definitionFile: -parent -graph: igraph.Graph String) contains -graph: igraph.Graph -vertex: int +setModel(model: igraph.Graph) -vertex: int * * -hyperedges: [HyperEdge] +createConfig(compdir: String) * 0..2 -children: [] 1 0..1 +Basic(dchart: DChart, parent, +__str__() +Orthogonal(dchart: DChart, parent, Composite graph: igraph.Graph, vertex: int) * graph: igraph.Graph, vertex: int) contains -dchart: DChart +generateDeclaration(desfile: File, +generateDeclaration(desfile: File, prefix: String) prefix: String) -parent -graph: igraph.Graph +generateActions(desfile: File) 1 controlled by +generateActions(desfile: File) +getHierarchicName(): String +getHierarchicName(): String -vertex: int ControllerConfig -children: [] -identifier: String -orthogonals: [Orthogonal] -layers: [String] -hyperedges: [HyperEdge] -components: {} +Composite(dchart: DChart, parent, +ControllerConfig(layers: [String], graph: igraph.Graph, vertex: int) identifier: String) +generateDeclaration(desfile: File, +addComponent(compDef: prefix: String) ComponentDefinition) +generateActions(desfile: File) +createConfig(name: String) +getHierarchicName(): String

usage

Generator -seed: {} -variantsToCreate: int -optimizeVariants: bool +Generator(configuration: String) +addDependencies(controller: ControllerConfig, component: ComponentDefinition) +group(n: int, iterable): [iterable Type] +parseSeed() +transformModel(compDef: ComponentDefinition) +generateBasePool() +start()

Figure 6.4: Transformation Platform Class Diagram 7 Results

In order to verify the usefulness of the work created in this thesis, we will perform a short analysis of results generated by the platform. First we will go over the conﬁguration used to achieve the results and then we’ll run the behaviours generated by the transformation platform through a couple test scenarios to see how they perform. Finally, we’ll discuss our ﬁndings and attempt to identify causes for any issues we found during the testing.

7.1 Platform Conﬁgurations

7.1.1 Test Platform

The session configuration file for the test platform is either created manually or created by the transformation platform. The transformation platform creates a session configuration with information derived from the seed file provided to it and the behaviours it is testing in that particular session. An example session configuration created by the transformation platform is shown in listing 7.1. 1 { 2 "maps": ["sand.cfg", "empty.cfg"], 3 "roundsPerMap": 3, 4 "players":[ 5 {"config":"56", 6 "name":"56", 7 "team":0}, 8 {"config":"57", 9 "name":"57", 10 " team " : 1} , 11 {" c o n f i g " : " 5 8 " , 12 " name " : " 5 8 " , 13 " team " : 2} , 14 {" c o n f i g " : " 5 9 " , 15 " name " : " 5 9 " , 16 " team " : 3} , 7.1 Platform Configurations 116

17 {" c o n f i g " : " 6 0 " , 18 " name " : " 6 0 " , 19 " team " : 4} , 20 {" c o n f i g " : " 6 1 " , 21 " name " : " 6 1 " , 22 " team " : 5} , 23 {" c o n f i g " : " 6 2 " , 24 " name " : " 6 2 " , 25 " team " : 6} , 26 {" c o n f i g " : " 6 3 " , 27 " name " : " 6 3 " , 28 " team " : 7} 29 ] 30 } Listing 7.1: Test Platform session conﬁguration created by the Transformation Platform

This session configuration was created from a seed file that specified a group size of 8 players and contained the map list and rounds per map information as it can be seen in this configuration file. Player configurations are named by the order they are generated in by the transformation platform. The game configuration file containing resolution information is never altered in any tests. For performance reasons, but also because the resources such as maps have been created for the default 640x480 resolution. Furthermore, the transformation platform always runs the test platform in headless mode and uses the default output file scores.json to evaluate the behaviours tested during each session.

7.1.2 Transformation Platform

The transformation platform is run using a single player seed file containing all the components discussed in section 5.3. The seed file used in the results discussed in the following section is given in listing 7.2. 1 { 2 " game " : 3 { 4 "maps" : [ "sand.cfg", "empty.cfg" ], 5 "roundsPerMap":3 6 } , 7 " p l a y e r " : 8 { 9 "layers" : [ "sensors", "analyzers", "memorizers", " strategic_deciders", "tactical_deciders", "executors ", "coordinators", "actuators"], 7.1 Platform Configurations 117

10 "components" : [ "aimexecutor_definition.cfg", " aimingcoordinator_definition.cfg", " engager_definition.cfg", 11 "explorer_definition.cfg"," fuel_definition.cfg", " health_definition.cfg", " insightsdetector_definition .cfg", 12 "mapdiscoverer_definition.cfg"," motor_definition.cfg", " movementexecutor_definition.cfg", " pilot_definition.cfg", 13 "playertracker_definition.cfg"," stayalive_definition.cfg", " stayfueled_definition.cfg", " steeringcoordinator_definition .cfg ", 14 "turret_definition.cfg"," tankradar_definition.cfg", " turretradar_definition.cfg" ] 15 } , 16 " t r a n s f o r m a t i o n " : 17 { 18 " r u l e s " : 19 { 20 "arules":["random_trains"], 21 "srules":["composite_reset"," transition_delay"] 22 } , 23 " b a s e P o o l " : 64 , 24 " g r o u p S i z e " : 8 , 25 "heuristic" : "basicheuristic" 26 } 27 } Listing 7.2: Transformation Platform seed ﬁle

Each component deﬁnition listed in the player object’s component list contains all information as discussed in section 6.4.3. The parameter distributions are almost always chosen as gaussian with distribution parameters chosen such that the parameters will contain values that are correct in the sense that they create working components (no out of bounds values), but the values might result in a negative or positive effect on the generated behaviour. The transformation platform will then select behaviours where the overall effect of the parameters and results of other transformations is beneﬁcial. 7.2 Test Platform Results 118

7.2 Test Platform Results

The test platform is run on my personal machine, which is a Macbook Air from late 2010. Plots showing the frames per second (FPS) and ticks per second (TPS) for headless mode on four different maps are provided below. Keep in mind that the FPS is limited to a maximum of 40.

Figure 7.1: Sand map rounds being played by 4 players

Frames Per Second in Non-Headless Mode Figure 7.1 shows 3 rounds being played by 4 players on the Sand map. This is a detailed map, which means performance is affected by the map as well as the AI’s calculations. Performance was stable at about 30 fps during the ﬁrst round, but the 2 subsequent rounds show better performance, which is due to the position of tanks and the complexity of their surrounding terrain. The fps drops visible on the graph are due to time delayed transitions causing calculations to be performed by the AI. This causes the ticks to take more time, lowering 7.2 Test Platform Results 119 the fps. During gameplay this is hardly noticeable, but it shows a deﬁnite and constant decrease in performance when plotted on this graph.

Figure 7.2: Interior map rounds being played by 4 players

The Interior map’s 3 rounds are plotted in ﬁgure 7.2. This map is much less detailed than the previous one and tops out at 40 fps during all rounds. The effect of timed AI calculations are more erratic and noticeable, however. The Interior map is created using a different technique compared to the Sand map, which seems to provide much more stable performance over the various areas of the map. The Path map rounds are plotted in ﬁgure 7.3. This map is once again made using a background image and detailed collision mask, like the Sand map. This can immediately be noticed by the jumps in performance between the three rounds. Various areas of the map cause differing performance. However, the affect on the performance caused by AI calculations is also affected. Round 1 shows much less constant drops in performance when compared with round 2 or 3. This is caused by two 7.2 Test Platform Results 120

Figure 7.3: Path map rounds being played by 4 players factors, one being the positions of the tanks compared to each other. When many tanks are in each other’s vicinity, the performance drops will be more erratic. It is also caused by the map itself, because some parts of the Path map have a very complex collision mask, while others do not, making the calculation of paths less time-consuming in some areas of the map. The Empty map serves as the base-line for determining the effect of map complexity on performance. Its FPS is shown in figure 7.4. All performance drops seen in this graph must be explained by either the coding of the engine itself or the AI calculations. We can see three very large drops in performance: one when the first round start, caused by after-effects of the map loading process. The second one occurs around the 420 seconds mark, when the new round is loaded. The third one around the 475 seconds mark, when the last round is started. Each drop in performance is followed by some more erratic drops in FPS, which must be caused by each tank getting a first look at the world through its components and figuring out how to move about. The start of the third round seems to be the most 7.2 Test Platform Results 121

Figure 7.4: Empty map rounds being played by 4 players erratic, this can be explained by the fact that during this round, most tanks were spawned closely together, causing more calculations to be performed then otherwise would have. As rounds progress and tanks die, the performance drops decrease until a quite stable FPS is reached, a few seconds before the winner is found and a new round starts.

Ticks Per Second in Headless Mode For the transformation platform, headless mode should complete the game rounds as fast as possible. To evaluate this and the performance effects of the AI calculations, we can take a look at the ticks per second in headless mode on the same maps as shown above. The tps of the Sand map is shown in ﬁgure 7.5. It clearly shows that as each round progresses, the tps rises. We can again clearly see drops in performance at regular intervals due to the timed transitions taken in the Statecharts models. From this graph we can deduce that the tps is at least equal 7.2 Test Platform Results 122

Figure 7.5: Sand map rounds being played by 4 players in headless mode to the maximal fps reached in non-headless mode, but rises quite high during a round. Headless mode is a definitely a huge aid when running the game for analytical purposes, and causes results to be available in acceptable time frames, even on weaker machines. The Interior map’s tps is plotted in figure 7.6. We can see that the tps varies more rapidly than it did on the Sand map, and on the whole is slightly higher than the tps on the Sand map. The increase in performance compared to non-headless mode is the same for both maps. The Path map’s tps is shown in figure 7.7. It again shows great similarity with the Sand map’s performance, due to the technique the map was created with. The increase in performance is again very good and no new issues regarding performance arise on this map. The Empty map’s tps is plotted in figure 7.8. Since there is no map complexity of any significance, we can use this plot as the baseline performance plot of the behaviour using Statecharts models. The 7.2 Test Platform Results 123

Figure 7.6: Interior map rounds being played by 4 players in headless mode behaviour I created is highly dependent on the amount of enemies left in the game, and shows very regular performance drops due to timed transitions. It would be better if these can be avoided in some way, possibly by more evenly distributing the calculations over different states.

Overall results The test platform is the very ﬁrst game engine I have created from scratch. It’s performance is good, but could be improved by more efﬁcient code in various areas. The fact that the performance depends heavily on the complexity of the map and the amount of players left in a round is something that is not desirable in a commercial product. Luckily, this issue is due to my coding of the engine and choices of the calculations made during each transition or action in a state. The proposed architecture for Statecharts behaviour works very good, provides an organised and re- usable code base and is easily expandable by creating more components or adding new functionality to existing components. 7.3 Transformation Platform Results 124

Figure 7.7: Path map rounds being played by 4 players in headless mode

7.3 Transformation Platform Results

The transformation platform was consistently able to generate behaviours that behaved in compliance with the heuristic that is supplied in the seed file. The heuristic used rewarded damaging other tanks and punished getting damaged, requiring pickups and being unable to move due to an empty fuel gauge. The resulting behaviours were not exactly as anticipated, but can be explained logically. Due to the random nature of the transformations, when we do not specify a seed for the random number generator(s), we always get slightly different results. During one run of the transformation platform, a behaviour was generated that resulted in tanks that were unable to move. Further inspection revealed a missing SteeringCoordinator, which the platform deemed not necessary, because tanks that did move, but did not have a firing capability yet passed by thanks that could fire but not move. This resulted in the stationary tanks to get better scores by the heuristic. This was only the case during 7.3 Transformation Platform Results 125

Figure 7.8: Empty map rounds being played by 4 players in headless mode some runs, but it highlights the exceptional sensitivity of the platform to the supplied heuristic. In essence, the heuristic speciﬁes the kind of behaviour that should be observed. The basic heuristic I have used did not reward moving, so moving behaviour was not always observed. All behaviours that were generated during this thesis are valid behaviours in the sense that they function within the limitations imposed on parameter values, component compositions and are valid models Statecharts formalism. The behaviour observed depends completely on the heuristic used, which is very important to generate behaviours that are applicable in various scenarios. I have so far only tested behaviours that conform to the heuristic described above, and have consistently gotten good results. When more effort is placed in the creation of better components and component models, smarter behaviours can be created. I have not focussed on creation optimal components in order to create amazingly performant behaviour, but prioritised the techniques required to create these from a set of components. Future development could focus on creating more complex components to observe 7.4 Testing Generated NPC Behaviour Descriptions 126 more interesting behaviours than described in this thesis. In the following section, I put the created behaviour through some tests to prove to the reader that indeed, the created behaviours perform as they should.

7.4 Testing Generated NPC Behaviour Descriptions

It is easy to observe that the created behaviour functions as desired, simply by starting the test platform with the default session file. One then observes four instances of the basic behaviour battling it out on four maps. However, to go into a bit more detail I created two test sessions where one instance of the basic behaviour is pitted against stationary opponents. The first session spawns a tank in the middle of a circle of stationary opponents to test if it does indeed start to attack the abundant supply of opponents it is confronted with. In the second session, I spawn a tank next to a path of stationary tanks to see if it goes for the first in line and follows the path to clear the whole map or if some other strategy is observed.

7.4.0.1 Circle Test Session

The circular test session can be started by supplying the file circlesession.cfg as the session configuration to the test platform. The session is played out on an empty map to avoid map complexity to influence the behaviour. Figure 7.9 shows the blue tank which is controlled by generated behaviour working to kill all tanks in the circle of green stationary tanks. Three rounds are played in this session, and each round we observe a slightly different strategy. The tank always turns slightly before observing and finding a target. Then, depending on the chosen target and if there is a route out of the circle, the tank either remains in the center and attacks a second target or it moves out of the circle to get some distance between itself and its target. The generated behaviour always tries to get some distance between itself and the enemy, so as soon as it is able to, it will move out of the (small) circle. From then on, the behaviour differs more each round. Depending on where the turret radar turns, it is possible that the tank might move away from the circle before seeing tanks on the other side and attacking them. However, most runs, the tank does indeed see each tank in the circle immediately and kills them. Sometimes sequentially, but sometimes it also picks the next tank on the circumference of the circle, because its Euclidean distance is smaller. In nearly all cases, the tank kills all tanks in the circle or is busy killing them before the round time runs out.

7.4.0.2 Path Test Session

The path test session can be started by supplying the file pathsession.cfg as the session configuration to the test platform. The session is again played out on the empty map to avoid map complexity to influence behaviour in any way. Figure 7.10 shows the blue tank – again controlled by generated behaviour – killing the green stationary tanks along the path. Just like in the previous test session, three rounds are played and each time the tank behaves slightly different. Most of the time the tank finds the first tank on the path and starts destroying it. It then moves along the path to destroy the next tanks. When it reaches the green tank on the upper left 7.4 Testing Generated NPC Behaviour Descriptions 127

Figure 7.9: Circle Test Session part of the path, the behaviour tends to diverge. Either it goes to destroy it or it goes for a green tank more to the south. When it moves to the southern tanks, it almost always continues along the path. When it chooses to destroy the tank in the upper left, the tank sometimes moves away from the path because it does not turn its turret to the southern tanks, whether or not it will continue killing tanks on the path then depends on whether or not the random pathﬁnding component chooses to move back in the western direction or not. When it does move its turret in the southern direction, it goes to the south and continues along the path, destroying tanks.

7.4.1 Interpreting These Results

Each run, we observe differing behaviour. Nearly all runs, the desired behaviour is observed. Since the objective of the transformation platform is to generate differing behaviour, I believe these obser- 7.4 Testing Generated NPC Behaviour Descriptions 128

Figure 7.10: Path Test Session vations conﬁrm that indeed, we have created NPC behaviour that does not follow a predeﬁned script to the letter. We must however take note that these observations are made on a single instance of generated behaviour, meaning that even in one instance of behaviour generated by the test platform, we will see variability in behaviour. We can reduce this variability by seeding the random number generator used in the component objects, however, the objective was never to end up with behaviour that performed its task exactly the same way each time. Since the tanks almost always succeed in their task – conforming to the heuristic with which their behaviour was generated – I believe the behaviour generation has succeeded. Future work could put the generated behaviour through much more tests, analysing performance and improving the transformation platform each time. 8 Conclusion

During this thesis I developed a complete architecture for creating NPC behaviour using Statecharts models. The architecture contains the design for a layer-based component structure and the definitions of all the concepts required to create Statecharts powered NPC behaviour. The behaviour needed to be tested in a controlled environment, for which purpose I created a test platform and all required files such that behaviour can be created for tanks in a Tank Wars style game. The test platform is highly configurable and can be run in visual and headless modes for analytical purposes. Besides creating the architecture to make Statecharts powered behaviour possible, I also developed methods to transform the behaviour and create new behaviour from a set of predefined components and layers. When provided with a heuristic, the created transformation platform can form behaviour that conforms to the heuristic and shows desired behaviour with varying parameters, component compositions and even varying Statecharts models through model transformation. The combination of a Statecharts powered behaviour as the core of the AI of an NPC and the creation of this behaviour through a transformation platform is completely new and this is therefore the first foray into this particular area of research. This implies future work is desirable to optimise, improve and expand upon the work in this thesis.

8.1 Real World Applicability

Statecharts NPC behaviour is highly applicable to commercial products. The architecture developed in this thesis was deliberately kept as general as possible in order to ensure its applicability in a variety of contexts (i.e. differing game engines). The architecture can be used in time-sliced and event-based environments and in any kind of gameplay scenario. It conforms to the state-based design for today’s modern AI, but formalises it using the Statecharts formalism.

8.2 Behaviour Modelling Advantages

By developing and working with the new architecture, test and transformation platforms. I have encountered several great advantages to this approach which are listed below. Apart from increased readability of the behaviour through the use of a modelling language, re-usability is introduced in a way that has never been seen before in game AI creation. However, some surprising other advantages 8.3 Future Work 130 were encountered which were not anticipated before my work on this thesis began.

• Abstracted behaviour components introduce a very modular way of creating NPC AI. The architecture in and of itself ensures a modular design of the behaviour which has an array of advantages when creating complex behavioiurs. • Readability is greatly improved thanks to the modular design and the use of a modelling language such as Statecharts. With a minimum knownledge of the API provided by the components and game engine, a developer can create new functionality in the Statecharts formalism which can be used by the transformation platform to generate completely new behaviours. • Re-usability is introduced by separating the behaviour in components. The transformation platform can re-use components in new behaviours, and while the resulting behaviour will differ, the basic components used to create it can be the same. • Transforming behaviour became possible by identifying the parts of the architecture that are independent of the core definition of the components. This means we can now automate (part of) the creation of NPC AI, greatly reducing work load on developers to create varying gameplay experiences. • By separating the components in a behaviour model and the API of the component itself, a clear line is drawn between the logic performed and the decisions made by the behaviour. This makes it very clear to the developer what is done at any point during gameplay, because the decision making process and the execution of logic can be debugged separately. • Creating new behaviour manually can be as easy as opening a Statecharts model, changing a few states and recompiling the model. There is no need to dive into code to find where some sort of action is undertaken. The Statecharts formalism makes it very clear where a certain decision is made, which increases developer efficiency and reduces the overhead of reading through long lines of code to find where a particular behaviour is defined. • The architecture in and of itself is quite simple, but very powerful. Learning to use it is not a difficult task and learning how to create components is as easy as learning the API of the game engine, the programming language used and the Statecharts formalism. No more knowledge is required. • As previously mentioned, the architecture was deliberately kept very general in order to ensure a maximal applicability to varying contexts. This means that this thesis can serve as the basis on which new NPC behaviour libraries can be created.

8.3 Future Work

Future work to be performed after this thesis can be found in all sorts of areas. The test and transformation platform itself can be improved by making the code more efﬁcient. However, they can also be expanded to create new functionality, traceability and making debugging of behaviour easier. These areas are not developer in detail in this thesis and future work is desired. 8.3 Future Work 131

Various concepts used in the transformation platform can be expanded upon. The parameter transformation could be expanded by introducing correctness of the parameter values, discarding values that are invalid or promoting better values through some sort of mechanism. This would remove the need to manually determine which parameter distribution values are good for use in the transformation platform. The component composition transformation algorithm could be improved by formalising the way the components are chosen during each step of the transformation platform. The current version choses them randomly each time, but more informed decisions could greatly improve the behaviour created. Finally, more transformation rules should be created to alter the Statecharts models, the way they are applied can be improved in the same way as the method for creating component compositions could be improved. Another area for future work is expanding the way heuristics are used to determine desired behaviour. The way heuristics are implemented in the current version is very basic, and limits the kind of behaviour that can be generated. Both the tracking of player scores and the usage of these scores in the heuristics can be greatly improved in order to create a much wider array of possible behaviours to be created by the transformation platform. In my opinion, the biggest opportunity for future work would be to create a version of the architecture that is applicable to at least one commercial product. This can serve to convince the industry that indeed, this new method for creating NPC behaviour is viable and has deﬁnite advantages. I am certain that this can be done within an acceptable time span and the possibilities for the further development of this architecture would be much improved when the industry takes note of this research. Bibliography

[Art05] Electronic Arts. EA tank wars. http://info.ea.com/company/company_tw.php, 2005.

[BDJ+03] Jean Bézivin, Grégoire Dupé, Frédéric Jouault, Gilles Pitette, and Jamal Eddine Rougui. First experiments with the atl model transformation language: Transforming xslt into xquery. In 2nd OOPSLA Workshop on Generative Techniques in the context of Model Driven Architecture, 2003.

[Bla08] Silvia Mur Blanch. Statecharts modelling of a robot’s behavior. Master’s thesis, Escola Tècnica Superior d’Enginyeria Electrònica I Informàtica La Salle, 2008.

[CC01] Craig Craig Cleaveland and J. Craig Cleaveland. Program Generators with XML and Java with CD-ROM. Prentice Hall PTR, Upper Saddle River, NJ, USA, 2001.

[CH03] Krzysztof Czarnecki and Simon Helsen. Classiﬁcation of model transformation approaches. In OOPSLA03 Workshop on Generative Techniques in the Context of Model- Driven Architecture, 2003.

[Cor12] Valve Corporation. Source engine. http://source.valvesoftware.com, 2012.

[DKV06] Alexandre Denault, Jörg Kienzle, and Hans Vangheluwe. Model-based design of game ai. 2nd International North-American Conference on Intelligent Games and Simulation, pages 67–71, September 2006.

[DKV11] Christopher Dragert, Jörg Kienzle, and Hans Vangheluwe. Generating extras: Procedural ai with statecharts. Technical Report SOCS-TR-2011.1, School of Computer Science, McGill University, March 2011.

[Fen03] Thomas Huining Feng. Scc (statechart compiler) documentation. http://msdl.cs. mcgill.ca/people/tfeng/uml/scc, November 2003.

[Fen04] Thomas Huining Feng. Dcharts, a formalism for modeling and simulation based design of reactive software systems. Master’s thesis, School of Computer Science, McGill University, February 2004.

[FHJ03] Daniel Fu, Ryan Houlette, and Randy Jensen. A visual environment for rapid behavior deﬁnition. In In Proc. Conf. on Behavior Representation in Modeling and Simulation, 2003. BIBLIOGRAPHY 133

[Gil04] Sunbir Gill. Visual ﬁnite state machine ai systems. Gamasutra, http://www. gamasutra.com/view/feature/2165/visual_finite_state_machine_ai_.php, November 2004.

[Har87] David Harel. Statecharts: a visual formalism for complex systems. Science of Computer Programming, 8(3):231 – 274, 1987.

[HN96] David Harel and Amnon Naamad. The statemate semantics of statecharts. ACM Trans. Softw. Eng. Methodol., 5(4):293–333, October 1996.

[Inc12a] Activision Publishing Inc. Call of duty®. http://www.callofduty.com/, 2012.

[Inc12b] Blizzard Entertainment Inc. World of warcraft. http://us.battle.net/wow/, 2012.

[Inc12c] Epic Games Inc. Game engine technology by unreal. http://www.unrealengine.com, 2008-2012.

[KDV07] Jörg Kienzle, Alexandre Denault, and Hans Vangheluwe. Model-based design of computer-controlled game character behavior. In Gregor Engels, Bill Opdyke, Douglas Schmidt, and Frank Weil, editors, ACM/IEEE 10th International Conference on Model Driven Engineering Languages and Systems, volume 4735 of Lecture Notes in Computer Science, pages 650–665. Springer Berlin / Heidelberg, 2007.

[Li08a] Kyle Li. Project report: Automatic behavior customization of non-player characters using players temperament. School of Computer Science, McGill University, Montreal, QC H3A 2T5, Canada, http://msdl.cs.mcgill.ca/people/hv/teaching/MSBDesign/ COMP763B2008/projects/repository/Yifan%20Li/ProjectReport.pdf, 2008.

[Li08b] Kyle Li. Reading report: Automatic customization of non-player characters using players temperament. School of Computer Science, McGill University, Montreal, QC H3A 2T5, Canada, http://msdl.cs.mcgill.ca/people/hv/teaching/MSBDesign/ COMP763B2008/projects/repository/Yifan%20Li/ReadingReport.pdf, 2008.

[NL12] Mathew Nelson and Flemming N. Larsen. Robocode. http://robocode.sourceforge. net, 2012.

[Omg08] Qvt Omg. Meta object facility ( mof ) 2 . 0 query / view / transformation speciﬁcation. Transformation, (April):1–230, 2008.

[Qui86] J. R. Quinlan. Induction of decision trees. Machine Learning, 1:81–106, 1986. 10.1007/BF00116251.

[Shi12] Pete Shinners. Pygame. http://pygame.org/, 2012.

[Sof08] Take-Two Interactive Software. Bioshock. http://www.2kgames.com/bioshock/, 2008. BIBLIOGRAPHY 134

[vRD01] Guido van Rossum and Fred Drake. Python reference manual. http://www.python.org, 2001.