AI for Warzone 2100

Masaryk University Faculty of Informatics

AI for Warzone 2100

Bachelor’s Thesis

Filip Ďuračka

Brno, Spring 2020

Masaryk University Faculty of Informatics

AI for Warzone 2100

Bachelor’s Thesis

Filip Ďuračka

Brno, Spring 2020

This is where a copy of the official signed thesis assignment and a copy ofthe Statement of an Author is located in the printed version of the document.

Declaration

Hereby I declare that this paper is my original authorial work, which I have worked out on my own. All sources, references, and literature used or excerpted during elaboration of this work are properly cited and listed in complete reference to the due source.

Filip Ďuračka

Advisor: RNDr. Mgr. Jaroslav Bayer

Acknowledgements

I‘d like to thank my advisor Jaroslav Bayer for all the support, and Warzone 2100 community for maintaining this amazing game for all those years.

iii Abstract

The goal of this work was to design and implement an AI for real- time strategy game Warzone 2100. As part of the design, two new features unique to this AI were implemented. First is creation of a graph representation of any given game map, obtained by analysing terrain and application of image processing algorithms. Second is a neural network, trained on data from 727 matches against other AIs, that guides the AI‘s battle plan.

iv Keywords

Artificial Intelligence, Game AI, Image Processing, Artificial Neural Networks, Warzone 2100, JavaScript

Contents

Introduction 1

1 Warzone 2100 3 1.1 Game mechanics ...... 3 1.2 Scripting language used for AIs ...... 6 1.3 Existing AIs ...... 6 1.3.1 Nexus AI ...... 6 1.3.2 NullBot ...... 7 1.3.3 Bonecrusher ...... 7 1.3.4 Cobra ...... 9

2 TenguAI 11 2.1 Map analysis ...... 11 2.1.1 Terrain analysis ...... 12 2.1.2 Image thinning ...... 13 2.1.3 Graph creation ...... 16 2.1.4 Graph representation and manipulation . . . . . 18 2.2 Cores ...... 20 2.2.1 Macromanagement cores ...... 21 2.2.2 Micromanagement cores ...... 24 2.3 Neural network for playstyle selection ...... 27

3 Conclusion 31

A Maps used for neural network training 33

B Statistics of matches before and after the use of neural network 37

Bibliography 39

vii

List of Tables

2.1 Visualisation of tile neighbourhood [8] 14 2.2 Match statistics during learning process, per opponent. 29 2.3 Match statistics with learned neural network, per opponent. 29 B.1 Match statistics during learning process, per map and game setting. 37 B.2 Match statistics with learned neural network, per map and game setting. 38

List of Figures

2.1 ’Picturisation’ of a map. Minimap (left) compared binary picture (right). Red denotes impassable tiles. 14 2.2 Comparison of binary image (left) to its thinned counterpart (right). 15 2.3 Thinned shape with nodes. Starting positions (green), resources (blue), crossroads (brown). 16 2.4 Final graph of the map. Starting positions (green), resources (blue), chokepoints (red), mundane nodes(black). 17 A.1 Small map with tight corridors and perfect early game chokepoint in the middle. 33 A.2 Water-separated continents with varying amounts of players. Ideal for testing analyser and cross-water attacks of the AI. 33 A.3 Sizeable, but tight map with hill-separated pathways. Good for concentrated defenses and macro movement. 34 A.4 Small but spacious map. Good for quick matches that do not require fancy movements. 34 A.5 Personal favourite. Tight, highly asymmetric starting positions. 35 A.6 Similar in concept to Monocot, but with more resources and corridors. 35

Introduction

The goal of this thesis is to create a new AI for Warzone 2100, that would be both more challenging and fun to play against, and deeper in its decision-making process than the alternatives. This is attempted by two systems no other AI in this game has. Firstly, giving the AI proper knowledge of the playing field by analysing the game map and creating a graph which can then be traversed in an arbitrary fashion. Secondly, equipping it with a neural network trained to select the best playstyle for the current state of the game. The first chapter introduces the game itself, explains the mechanics and presents AIs currently in use for multiplayer games, their strengths and weaknesses. The second chapter dissects design and implementation of my AI, what it does better, faster, what it does extra and how it affects an average game session. The end contains analysis of the results of pitting the AIs against each other and discussion about possible causes.

1 Warzone 2100

This chapter explains what the game is [1], what sets it apart from most other games in its genre and design of AIs most widely used for play.

1.1 Game mechanics

Warzone 2100 is a real-time strategy game set in a post-apocalyptic fu- ture. As such, the goal of the game is to destroy all hostile assets present on the map. To this end, players have to build a base of operations, secure income of resources and produce an army, while defending their own base against enemy attacks [2, 3]. The maps are made of square tiles organised into rectangular grid. Each tile can be occupied by a single player-built structure or environmental feature (e.g. trees, houses, miscellaneous ruins), with some structures requiring several tiles. This grid has no effect on units, which can roam freely across tiles. Maps can be up to 250 * 250 tiles in size. Building a base is done via special, defenseless builder unit available since the start of the game. Typical player base would consist of headquarters structure and some amount of factories, research facilities, generators and static defenses(turrets, bunkers and walls) positioned around the perimeter. Resource collection in the game is simplified compared to others in the genre. Instead of sending out defenseless and potentially expensive harvester unit, Warzone 2100 makes the player build a free(and defenseless) resource extractor on designated places on the map. These are, upon construction, automatically assigned to a power generator, which generates the game‘s only resource – power – at speed dependent on the amount of extractors assigned. Maximum number of these a single generator can serve is four, so players must take care in reserving enough space and power for multiple generators. Unlike most games in this genre, Warzone 2100 provides the player with an opportunity to design their own units, called droids. After building the headquarters, and only while this building is present on the map, the player gets access to a designer tool. Within this tool,

3 1. Warzone 2100 they can create a droid template by selecting components of three categories:

∙ body, affecting base health, horsepower, thermal and kinetic armour and turret capacity of the droid;

∙ propulsion, affecting speed, general mobility of the droid, type of traversable terrain and resistance(or vulnerability) to certain weapon types;

∙ turret, which is either a weapon, or a utility, i.e. constructor, radar, repair.

In this tool, player can design aircraft by selecting a VTOL (Verti- cal Take-off and Landing) propulsion. This changes a lot of properties of the droid. Firstly, and unsurprisingly, it gains the ability to fly over any terrain and becomes untargetable by many ground-based weapons. Secondly, it gains access to special VTOL only variants of droid weapons (and some new weapons of their own) that have limited ammunition supply. Lastly, it needs a special factory to be produced in, and rearming pads to repair and replenish the ever needed ammunition. Role of ground infantry is held by cyborg units. These are, unlike droids or VTOLs, not customisable, instead coming as template ready for production when necessary technology has been researched. Basic cyborg templates are unlocked together with certain droid weapons. Heavier, more durable and damaging variations are available to be researched separately after researching several high-tech droid weapons. Cyborgs follow their own armor upgrade research path separate from the droids and require special factory to be produced in. Related to cyborgs are flying unarmed transports capable of ferrying small amount of cyborgs across otherwise impassable terrain, with heavy cyborgs requiring their larger, more durable variant. Like many contemporary RTS games, Warzone 2100 features research opportunities to enhance capabilities of player‘s units. However, unlike most of these games, research here provides much more than upgrading stats. Within the very expansive technology tree, players can unlock new buildings, base defenses and droid components in addition to mentioned damage/armour/speed upgrades for all assets.

4 1. Warzone 2100

It is crucial for player‘s success to properly prioritise which research path to follow, when to keep following it and when to switch to another one [4]. Weapons are possibly the most important research items in the whole tech tree. They are separated into several categories. These determine what upgrades apply to the weapon in question, and modify damage to certain propulsions and structure types [5]. Weapons themselves deal either kinetic or thermal damage that gets mitigated by kinetic or thermal armour, respectively. Accuracy of the weapon also plays a role, with bullets being able to completely miss their target and hit something nearby, or detonate on terrain. Some weapons get mild bonus to accuracy when firing from within half of their maximum range, which combined with possibility of hitting something else makes proper unit placement in battle not negligible. There are multiple parameters that can be set before the start of each game. Aside from choosing a map for the game to take place on, the host can also alter starting power for all players(low, medium, high), starting bases(no base, basic structures only, defended base), structure limits(except defensive structures) and starting technology level(no tech, basic components available, advanced components available, everything researched). The game contains basic diplomacy system. Its type can be set in the lobby and can be one of:

∙ free for all, where every player stands for themselves;

∙ dynamic alliances, where players can form and break alliances at will during the course of the game;

∙ locked teams, where the alliances are decided during game setup;

∙ locked teams with shared research, which is self-explanatory.

When in alliance, players can choose to share map vision and send power or droids to an allied player. Last option during game setup is the presence of scavengers. These are AI-controlled, non-playable faction that uses unique units and structures. They are usually found guarding power sources around

5 1. Warzone 2100 the map, and may have a base with unit production capabilities in a central location on a map. Their destruction is not necessary for victory, but they do provide a sort of anti-rush protection and generally slow the early game flow in games when enabled. Last thing, there are two types of pick-ups available during the course of a game. First is an oil drum, which provides immediate bonus of 1000 energy upon pick up. Second one is an artifact, which some- times spawns from a destroyed factory. When picked up, it unlocks a component for research for the player, regardless of its dependen- cies. This component is always such that the owner of the factory has researched it, but the picking player has not. In case there is no such component, artifact simply does nothing.

1.2 Scripting language used for AIs

All AI scripting is done in so called "Warzone 2100 JavaScript Scripting API". This is, at least as far as we can tell, adhering to ECMAScript 5.1 specification. If the reader would find themselves scripting in thisAPI, caution should be taken to not attempt to use classes, "=>" operator and some quality of life functions and consult ECMA 5.1 specification1 and API documentation2 frequently.

1.3 Existing AIs

This section introduces popular AIs available for play.

1.3.1 Nexus AI Original AI made by Pumpkin Studios. Very simple in nature and slow to react. Testing matches against Nexus almost always resulted in its defeat, even when set on highest difficulty. Thus, Nexus will not be used as an opponent.

1. http://web.archive.org/web/20111103184035/http://www. ecma-international.org/publications/files/ECMA-ST/ECMA-262% 20edition%205.1%2C%20June%202011.pdf 2. https://github.com/Warzone2100/warzone2100/blob/master/doc/ Scripting.md

6 1. Warzone 2100

1.3.2 NullBot NullBot3 is an AI featuring dynamic unit design and research based on its opponent‘s choices. There are 4 major personalities available for this bot: standard, hover(optimised for watery maps), turtle(builds only defensive structures, no units but builders) and scavenger(takes control of scavenger NPC faction). Each of these has subpersonalities defined by what weapon types they focus on. Adaptive mechanism gathers enemy unit statistics - which components were used and guess their purpose. Based on this information, it attempts to build an army equipped to maximum effectiveness against that opponent. During design process, weapons are picked by their expected role and matched with body by its weight class. Nullbot is also able to cooperate with human player if they happen to be together in an alliance. By simple chat commands, player is able to request power or units from the bot.

1.3.3 Bonecrusher Non-cheating AI that favours aggressive expansion. Bonecrusher4 is usually regarded as the most able of the available AIs [6]. Bonecrusher‘s droid design is rather simple – if available, it always builds heavier bodies. Propulsion is always weighted to the heaviest one it can afford. It chooses weapons based on how much research went into their type, and from the most researched type it chooses one of x best options at random. It completely ignores anti-air turrets. AI‘s research follows hard-coded paths, each providing a specific benefit, be it upgrades to economy, armor or components. Weapon type the AI uses for a game, and thus its main focus, is contained in pre-defined profiles. These profiles are assigned at the beginning of the game either at random, or to supplement other AIs in the team. After this assignment, based on AI‘s difficulty, the profile‘s techs are researched in one of three ways.

3. https://github.com/haoNoQ/nullbot 4. https://github.com/EuPhobos/bonecrusher

7 1. Warzone 2100

∙ In strict order, as defined in the profile, being supplemented by upgrade paths;

∙ As ’smudged’, meaning each tech in a profile gets its own path, consisting of a single research;

∙ Random order, where the profile gets ’smudged’ and its tech order shuffled.

Smudging can cause certain technologies to be researched earlier than in strict mode, potentially altering AI‘s design choices. If the bot doesn’t consider itself rich, its VTOLs target enemy resource extractors, otherwise they bomb production facilities, or scout last known enemy positions. Repair units and jammers either chicken out towards nearest defensive structure when the army is dying or try to get close to hurt units. Partisans target:

1. enemy walls;

2. immediate threat to self;

3. immediate threat to ally if Bonecrusher considers itself rich;

If they are understrength, they run to repair units if available, else they scout unknown resources. In case partisans consider themselves in full strength, they harass resource locations, scout the map, target factories and builders, in this order. Bonecrusher takes care to get heavily wounded units repaired whenever possible. Cyborgs harass enemy resources when understrength, else they attack defenses and factories, but only if its own base is not under threat. If they cannot do anything better, they clear map features (e.g. trees, ruined miscellaneous buildings). If nothing of the above can be done, they join regular army. Regular droids first assemble at given point, avoiding combat when overwhelmed. They get called when under attack. The bot is characteristic in its aggressive early game resource acquisition and liberal use of defensive structures. It attempts to build defenses near every resource location, regardless of its current owner. Exact amount of these defenses scales with difficulty, but it always

8 1. Warzone 2100

results in well fortified resource extractor, regardless if bonecrusher got to the resource first or the defenses destroyed extractor already present. Much like Nullbot, Bonecrusher is capable of interaction with player via chat messages. Aside from simple greetings and smug trash-talk, it is capable of responding to requests for power, units, or to pursue a certain profile. Of course, player needs to be allied with the bot to use this feature.

1.3.4 Cobra Cobra5 is optimised for shared research games. It has personalities based on the combination of weapons it uses, some of which are considered unusable if the AI plays alone. Its unit design does not differentiate unit‘s purpose. All units are designed as either, attackers, VTOLs or system(builder, repair or sen- sor). It prefers to use heavy bodies when possible, tracks for propulsion and weapon based on its personality. From tactics standpoint, Cobra divides droids into ordinary attackers and artillery. Attackers, cyborgs and VTOLs attack nearest assets of an enemy perceived to be the biggest threat. Artillery acts similarly, but works closely with sensors to take full advantage of its range.

5. https://github.com/KJeff01/Cobra

2 TenguAI

This chapter analyses our AI, it‘s core components and performance in comparison to competition. Design philosophy for this AI is comprised of several points. Firstly, no-cheating policy (also know as basic decency). To maintain fair-play and truly test the capability of this AI, care is taken to avoid usual crutches used in AI design, mostly having access to information it‘s human counterpart would not have under identical circumstances. Secondly, independence from balance changes. In the spirit of low maintenance, all components and their relative comparisons are not to be assumed immutable. Any change of base values or addition/re- moval of components should not negatively impact the usability of the design process. Thirdly, to differentiate this AI from competitors and give it the technological edge, graph representation of the battlefield is to be created and utilised. No other Warzone 2100 AI has succeeded in (and perhaps even attempted) this task. As this is the most distinctive and important feature of this AI, it is discussed as very first. Fourthly, to stay more firmly within the field of AI, a neural network is trained to choose (preferably) optimal playstyle for any given game state the AI can find itself in. In light of these points, a completely new AI was built from scratch to accomodate the analysis and neural network.

2.1 Map analysis

If player is to make sound tactical and strategic decisions, knowledge of the playing field is paramount. Starting positions, locations of resources, chokepoints, environmental hazards, all of these are important factors to include in decision making process. In modern RTS games, all competitive AI designs have access to a graph representation of the map.[7] This graph is usually created by human map designer simultaneously with the map itself. War- zone 2100, being as old game as it is, has no such feature, and by having active map-making community, creating a graph for each map

11 2. TenguAI individually is not viable. Instead, we opted for automatic map analysis. The analysis takes place at the end of level loading sequence, in a state when environment is fully loaded but player objects have not yet been instantiated. There would be an option to perform this analysis at proper level start, but seeing this would freeze the game for the duration of the analysis process (which takes around 3-4 seconds on average), it was decided against. The analysis process can be divided into several stages: terrain analysis, shape thinning and graph creation.

2.1.1 Terrain analysis

During pre-processing, node objects are created for points of interest (further as POIs): player starting positions, resource locations and gateways. These gateways are places defined by the map creator, meant to help AIs identify easily defendable areas. Such perfect knowledge of the battlefield is not to be considered cheating, as players themselves have access to map preview before every match. There, these points (except gateways) are highlighted. Next, a general map topology is found by examining what propulsion can reach which POIs. Wheels are used as a base, land-only propulsion, hover as water-capable option. There are several purposes to this information. Firstly, analysis itself uses it to determine what it should consider traversable. Secondly, it limits the unit designer, as to not use water-incapable propulsion on a water-dominated map. Thirdly, it guides micro core‘s choice of unit compositions. The AI is capable of recognising these topologies:

∙ If all POIs can be reached by wheels, the map is declared ’land’ - designer is free to use all propulsions and any water present is considered not traversable;

∙ If all players can be reached by wheels but some resources only by hovercraft, ’semi-land’ - unit designer prefers to equip construction units with water-capable propulsions, water is traversable;

12 2. TenguAI

∙ If some players can only be reached by hovercraft, ’water’ - special micromanagement subroutines are activated and water is, again, considered a traversable tile;

∙ Otherwise, topology is declared ’vtol’ - only flying units will be used and all tiles are considered passable.

2.1.2 Image thinning To create representation of the map, knowledge of POIs alone does not suffice. We also need information about edges between them. This could be done by running simultaneous BFS searches from each POI, and connecting nodes who meet each other. This approach, however, omits information about crossroads, passageways, even whole alter- nate routes. Instead, we can interpret the map as an image. Images have shapes. And shapes can be minimised down to simple lines, along which we can place edges of the graph. Actual goal of this stage is to obtain medial axis transform, or its close approximation, of the shape bound by impassable terrain. Medial axis can be understood as skeleton of a shape. There are two main categories of algorithms obtaining medial axis - skeletonisation and thinning [8]. Skeletonisation algorithms aim to obtain a set of points which are equidistant from nearest boundaries. To find these points, all map tiles would get assigned their distance from the nearest edge and finding skeleton would become a task of finding all locally maximal tiles. This approach has been tried, but resulting skeletons were often disjointed and tremendous effort had to be made to piece them together [8]. Thinning consists of examining immediate neighbourhood of any given point in order to identify pixels that are, essentially, redundant to the shape, i.e. those lying on and edge of a circle. By removing such points, we are able to obtain an approximation of medial axis, insensitive to boundary noise. Compared to skeletonisation, thinning always results in fully connected shape, and lacking the extra join- ing overhead, performs faster. Needless to say, it was chosen as the preferred algorithm for this stage [8]. The map itself is traversed tile by tile, placing 1s on reachable tiles and 0s on unreachable ones. Exact definition of reachable depends

13 2. TenguAI on overall map topology. Potentially, the result can contain several disjointed areas of reachable space, representing continents and/or different elevation levels unreachable by preferred propulsion system. Interpreting this two-dimensional array as a binary image, a thinning algorithm is used to obtain medial axis.

Figure 2.1: ’Picturisation’ of a map. Minimap (left) compared binary picture (right). Red denotes impassable tiles.

Initially, two identical copies of the image are created. One of the is declared current and thinning iteration commences. For each tile, if a thinning condition is met, it is set to 0 (non-shape). This condition is evaluated on the other image, as changes done to the image may interfere with the algorithm. Referring to table 2.1, the condition is as follows: Table 2.1: Visualisation of tile neighbourhood [8]

P3 P2 P9

P4 P1 P8

P5 P6 P7

14 2. TenguAI

2 ≤ NZ(P1) ≤ 6 ∧ Z0(P1) = 1 ∧ P2 · P4 · P8 = 0 ∨ Z0(P2) ̸= 1 ∧ P2 · P4 · P6 = 0 ∨ Z0(P4) ̸= 1

where Z0(P1) is a number of zero to non-zero transitions on a path P2 → P3 → · · · → P9 → P2, and NZ(P1) is the number of non- zero neighbours of P1 [8]. In addition to this, one more condition is applied to except POIs and keep them within the shape regardless. This ensures natural formation of paths that connect them to the rest of thinned shape.

Figure 2.2: Comparison of binary image (left) to its thinned counterpart (right).

After whole image has been traversed, content of the modified image is copied into the helper image and process starts again, until no further change occurs in the image. At that point, the array is scanned one more time. All tiles that have more than three non-zero neighbours are declared as nodes. This results in clusters of nodes being created at locations of crossroads, which is going to be handled by the next step.

15 2. TenguAI

2.1.3 Graph creation In this step, obtained thinned shape is used to establish edges between nodes of the graph.

Figure 2.3: Thinned shape with nodes. Starting positions (green), resources (blue), crossroads (brown).

Queue of nodes to be processed is established, initially containing only node located at AI‘s starting position. From here, paths visible in thinned image are traversed in breadth first manner. Visited tiles are marked with ’-’ to prevent backtracking and potential cycle. If another node is encountered, it is pushed into queue of nodes to be processed, bidirectional edge is established between them and further traversal in this direction is stopped. This stop is trivial, because paths are always single pixel wide. Now, the algorithm gets to deal with the redundant nodes present in all crossroads. To preserve edges and graph integrity, these nodes are merged together. Merging here meaning all edges from node A are added into node B. No other attributes are affected and node Ais discarded afterwards. This is to be done until a single node remains in the cluster.

16 2. TenguAI

Starting again at the node representing AI‘s base, the graph is traversed via BFS. All connected nodes within manhattan distance of 1 are added into the cluster and searched for more nodes to add. After no more nodes can be added, the cluster is scanned for POI nodes. If any POIs were found within the cluster, all mundane nodes directly connected to any POI nodes are merged into all POI nodes they are connected to, repeating until cluster contains only POI nodes. If no POIs were present in the cluster, nodes from both sides of the cluster are merged into their immediate neighbour. As clusters are stored in arrays, the first and last nodes are merged into second and one-before-last nodes, respectively. This results in a single node roughly at the center of the cluster. At this point, all nodes that are left edgeless are removed. These are unreachable and could interfere with graph usage.

Figure 2.4: Final graph of the map. Starting positions (green), resources (blue), chokepoints (red), mundane nodes(black).

17 2. TenguAI

Upon close examination on some maps, edges can be found that seem to skip over a node. These are obviously unwanted and need to be removed, lest they interfere with pathfinding. These edges are identified by forming an obtuse triangle with smaller angles being less than 10 degrees. Algorithm identifies such triangles and removes the edge matching onto the longest edge of said triangle. As the last step, if the map topology is not ’land’, edges are assigned ’water’ property if they connect nodes divided by a body of water. These are identified by a localised version of thinning. Whole process, as described above, is applied to a minuscule section of the map. This section is a rectangle containing nodes of the edge in its opposing corners. Water is set as 0s, everything else as 1s. Thinning is then applied and if the nodes are not connected by the medial axis, edge is declared ’water’ and will be only pathed through for water capable propulsions.

2.1.4 Graph representation and manipulation With map analysed and graph created, its actual structure and use needs to be discussed. Nodes themselves contain information pertaining to the general area surrounding the node. Besides type of the node and edges, as discussed above, nodes keep track other strategically important information. LastSeen timestamp is used to send out scouting parties to obtain information about enemy presence (or lack thereof) around given coordinates. This information consists of the following: fortification, describing how much is the area fortified; owner, denoting last known controller of the area; chokepoint flag, identifying the node as well defendable; and hazard level, the size of enemy force known to be around the point. References to nodes themselves are kept in two objects. One is in two dimensional array on node‘s respective coordinates. This array is used to quickly fetch any given node at or nearby any given coordinates. Second one is in simple array, which is used for iteration over all nodes. Units, structures and even terrain features almost never correspond to exact node positions. Ergo, an algorithm has been devised to match any given coordinate to the nearest node. It works simply as follows:

18 2. TenguAI

starting at given coordinates, if node is present, return it. Else examine tiles 1 away from original point. This examines an edge of progressively larger rectangular portion of 2D array containing nodes, up to a distance equal to half of longer edge of the map. This limit ensures a node is always found. For pathfinding through the graph, two algorithms are used, based on information required. Any time the AI needs to find nearest x or path from A to B, these algorithms are used. For finding nearest x nodes satisfying a condition, a modified breadth first search (further as BFS) is used. In its base form, BFS is an uninformed (meaning no further information other than the graph itself is provided) search strategy that examines all immediate neighbours of a given node (regardless of distance), then neighbours of these neighbours and so on, until a match is found or it runs out of nodes [9]. First modification is the ability to look for nodes based on arbitrary conditions. Every time a node is processed, a predicate function is run to determine if the node is to be returned. Second modification is the option to return multiple matches instead ofjust a single node. This is done by filling an array with viable nodes until either a given amount is reached or the algorithm runs out of nodes. Third alteration is the implementation of a priority queue, making the search prefer to expand closer nodes. For pathfinding, AI uses A* algorithm. A* is by nature an informed search. This additional information is provided via heuristic function that alters preference values for every node. For every node to be expanded, A* computes the cost of expanding each node, in this case distance from source node, adds value given by heuristic function and then expands the cheapest node available. This is repeated until target node is reached and path taken to reach this node is minimal, as long as heuristic function satisfies two conditions: admissibility and consistency [9]. Admissibility means heuristic function will never evaluate cost to target node as greater than what it actually is. Example of this would be straight line distance, which AI partially uses. Consistent function adheres to triangle inequality - estimated cost from node A to target is not greater than cost to neighbour node B plus estimated cost from B to target.

19 2. TenguAI

As a heuristic, this AI uses an addition of straight line distance and hazard level of a node. This is to ensure that the safest path is found first, if the shortest one would result in unnecessary danger tounits. Periodically, AI scans area around all nodes for enemy contacts. If it finds anything, it saves the position and composition of enemy force or base located. As groups of enemy units tend to move around the map quite often, it also attempts to compute whether it has seen the group before. This is done by the following algorithm:

1. Note down current information about the group: owner, composition, current game time and map position;

2. Based on components of a single randomly chosen unit, compute its movements speed.

3. For each group G, compute speed required to move from position of original group to G‘s location in amount of time elapsed from the time G was last seen.

4. If this required speed is less than or equal to group‘s speed AND G is closest such group found, set G as most likely match.

5. If a group satisfying these conditions was found, update its info with current group.

Base location is much easier. Once a structure was revealed, it stays visible to the player until they explore the area again. With this knowledge, the AI simply scans the nodes for any visible structures present in the vicinity and notes down their location, amount and owner.

2.2 Cores

As with every RTS game, decision making process in Warzone 2100 can be divided into two separate, mutually supportive categories: macromanagement and micromanagement. For both of these, separate controller units - cores - were created in order to separate different aspects of play.

20 2. TenguAI

2.2.1 Macromanagement cores Macromanagement consists decisions affecting the flow of the game on global scope. Strategy, unit production, resource procurement, research and development all fall under this category. In the case of Warzone 2100, unit design is also considered part of macromanagement. Each of these aspects is represented by a core.

Main core This core is the heart of macromanagement decision-making process. It contains a neural network trained to output preferred strategy based on current and previous game state. This is further elaborated on in section 2.3. There are three strategies available to AI, based on what field it should try and gain/maintain advantage over the opponent. These strategies being production, defense and technology. Focusing on one of these, the AI changes processing order of other macro cores, essentially assigning them priority access to resource stockpile. With production focus, the AI attempts to gain numerical superi- ority. When focusing on defense, the AI fortifies its resource extractors and base/entrance to the base, if available. Research is preferring defensive structures and upgrades to their longevity. When attempting to gain technological advantage, the AI gives more value to researching new components and upgrades to unit health, armor and damage.

Engineering, Production and Research cores This trinity of cores represent basic strategic aspects common across most RTS games. The all include an object mapping each structure/unit type/research item to its perceived value to the AI. Communication between these cores is done via requests. When a core finds out it needs something falling under the jurisdiction of another core, it send a request stating what it needs. The other core than increments the value of requested item by an amount defined by the main core. This amount is defined by relevance of said item to

21 2. TenguAI the current plan. Any time the AI commits to a plan this value gets incremented by 1. Research and production cores are fairly simple both in their purpose and implementation. They manage their respective facilities and queue up items for research/production based on their perceived worth. Engineering core serves as hub for constructor units and manages the construction of base structures and defensive works. Constructor units are divided into 2 categories - main group and roamers. Main group stays within base perimeter and does most of the constructing. Roamers are dedicated to building resource extractors, progressively further away from the base. This results in rather aggressive early game expansion, very similar to Bonecrusher‘s (barring the defenses), and is deemed vital to establishing healthy resource income. In addition to structures having priority values, engineering core also keeps track of sites that have been requested to be fortified, again with priorities. From these, every time the AI decides to build defenses, the most pressing location is picked.

Design core Design of units is one of the defining aspects of Warzone 2100, and as such, has its own dedicated core. It keeps unit templates and sets of components available for use. The goal of the design process is to create a unit which deals maximum amount of damage to the opponent‘s army while simultaneously minimising damage it takes from the same force. Due to nature of the components, however, it is impossible to create a super-unit that would be good against everything. For this reason, the designer core creates several templates which focus on different types of enemies. The composition of these different units into squads is left for other cores to handle. Templates themselves are objects containing components making up the unit and its final properties, like damage, speed and health. The API does provide its own templates, but these were deemed insufficient as they lack most attributes required for proper design process.

22 2. TenguAI

Both droid and cyborg templates have three variants, depending on their preferred targets: anti-tank, effective against droids, preferring to wield missiles and cannons; anti-personnel against cyborgs with machine guns and flamethrowers and anti-structure using mortars and howitzers to deal with enemy emplacements. Every weapon in game has three attributes defining its purpose: impact class, impact type and damage modifier. These are usually bound together, but not always, which can cause some issues. Impact type is the type of damage a unit deals. It can be either heat, as in case of flamethrowers, which is mitigated by thermal armour, or kinetic, which can be prevented by simple armour. Impact class, or upgrade class for clarity, is the class of research upgrades the weapon benefits from. This does not take into account what type of damage the unit actually deals and improves all weapons in the class equally. Damage modifier is what actually matters when trying to maximise damage. All damage dealt to a target is multiplied by this value, which depends on propulsion or structure hardness any given target has. As mentioned, this does not always correspond to the impact class. Most notable case being class of rockets: while all benefiting from same research, some rocket weapons are considered anti-tank in the terms of their damage modifier, while others are marked as artillery. Majority of Warzone 2100‘s strategy is in army composition and ensuring you can destroy both enemy droids and cyborgs quickly, and diluting one‘s research with weapons of class one does not plan on upgrading, or upgrading weapons one does not use, is not only pointless but detrimental. Thus, the AI picks three impact classes to pursue, one for each unit type and one as anti-structure option. It does this by sorting all weapons possible to research into arrays corresponding to different impact classes. From these, one class per purpose is chosen at random, mainly to maintain variety in game style. As a sign of decency, the actual design process is not started if the Command Center has not been built. This is to adhere to the rule of human players being unable to open the designer window under the same circumstances, which AIs are not inherently bound to. Designer core keeps a template of the most numerous enemy unit encountered so far and bases its design choices on this template. Droids are designed by the following process:

23 2. TenguAI

1. From available bodies, such is chosen as to improve both health and armor/thermal armor (depending on enemies weapons);

2. From available propulsions, one is chosen to minimize damage taken from enemy weapons. In case the map is not universally reachable by land units, purely land-based propulsions are ig- nored;

3. From available turrets, turret capacity of chosen body id filled with a weapon with greatest damage against most common target.

For VTOLs, propulsion is pre-determined to be VTOL and so this step is skipped. For cyborgs, only weapon is being chosen, as they are hard-coded by the game to only take one specific propulsion type and one of two body types, hard-linked to the weapon. After design of a unit type is complete, designer sends research requests for new components, based on the same logic as stated above. Upgrades are also requested, with weapon upgrades being limited to chosen upgrade paths.

2.2.2 Micromanagement cores Micromanagement is the process of controlling individual units or groups of units, in order to achieve strategic objectives (i.e. destroy enemy base). Unit movement, their placement during combat and flanking all belong into this category. Cores responsible for micromanagement are divided based on the amount of units they control.

Sergeant core This is the basic formation of AI‘s army. It contains up to predetermined amount of units of specific type and using the same propulsion. This last condition was added in order to streamline the pathfinding process and secure that all units under single sergeant are able to reach their target. When the core takes fire from an enemy unit, a response rou- tine starts. First, the core looks around and obtains total amount of friendly and hostile units within certain radius. If the enemy has more

24 2. TenguAI units present, the core sends a reinforcement request to its superior. If granted, the core stays in combat, otherwise it attempts to retreat. Sergeant cores can be in one of 4 states - idle, moving, engaged and retreat. When idle, the core is willing to accept new assignments from superior core. If it detects that combined health of its units dropped below predetermined percentage, it attempts to move to nearest repair facility for repairs. When moving, the core is in transition towards given coordinates to perform assigned task. This task can be one of:

∙ attack - the core moves towards location actively engaging anything it meets;

∙ scout - the core moves towards given coordinates ignoring enemies, unless it is engaged first;

∙ hold - the core is sitting on top of a location, guarding against enemy attacks;

∙ retreat - the core requests a repair facility to be built and attempts to locate either repair facility or nearest friendly and fortified position and moves there to regroup. If no such position is found, it returns back to base;

∙ follow path - core carefully follows given path, acting similarly to scout order;

∙ disband - core moves all its units to nearest factory for recycling.

Once in combat, the core locates, for each unit, the nearest enemy said unit can effectively engage. Then, it orders the unit to attack and carefully moves it around to stay at the edge of its short weapons range. Within this range, the unit benefits from increased accuracy, and thus, higher damage potential. When retreating, the core is in panic and ignores all orders and attacks until its destination is reached. Any time a sergeant is completely wiped out during battle, it asks superior core to reevaluate its current unit type and assign a new one, based on current situation.

25 2. TenguAI

Colonel core Colonel core is the central hub for micromanagement. It coordinates sergeants and makes tactical decisions based on known enemy strength and location. This core‘s process can be divided into two main parts: threat evaluation and planning. Threat evaluation follows the following pattern: 1. All known enemy groups are assigned a threat value defined as groupSize/distanceFromOwnBase; 2. If the enemy has (globally) more cyborgs than droids, production priority of anti-personnel units is incremented. Same hap- pens for anti-tank units in case of prevalence of droids; 3. If the enemy has in total more units that AI‘s sergeants can currently manage, add a new sergeant of type most effective against most numerous enemy and weapon type; 4. If the most threatening group has overwhelming numbers (more than double of AI‘s total), find a path from the group‘s last known location and request fortification of chokepoints on this path; 5. Find path from AI‘s base to the base of focused enemy. If the enemy has more fortifications on any given node on this path than AI‘s anti-structure unit count, add new sergeant of anti- structure role. During planning, if on disjoint map, the AI tries to identify if it has any enemies left on its continent. If not, it instruct the designer to favour water-capable propulsions and orders the disbanding of all land-based sergeants, if an alternative, water-crossing design is available. Additionally, if the macro plan is to focus on defense, the AI requests fortifications to be built in its base, the entrance to its base,if available, and all currently controlled resource locations. Then, threats are ordered by their severity and individual idling sergeants are ordered around depending on currently selected micro plan. This plan can be one of attack, skirmish, de f end.

26 2. TenguAI

If the plan is to attack, the AI tries to engage strongest known enemy group or assault enemy base, if no such group is known. If a group is to be engaged, the AI first attempts to muster strong enough force to have a chance of victory. This is done by pooling sergeants together, until their combined unit count is at least that of the targeted enemy group. In case of skirmishing, the AI looks over nearest 5 known enemy derricks, picks one with weakest know enemy presence and pools sergeants together, until their strength is greater than perceived hazard around targeted derrick. If no derrick is found, it sends the sergeant against known, weak enemy group; condition being the sergeant must outnumber it at least 2:1. If all fails, the sergeant is sent to scout nodes that have not been seen for a considerable amount of time. When defending, the sergeant is sent to hold a fortified position lying on a shortest path that the most threatening enemy group can take to AI‘s base. If no such node exists, the sergeant holds entrance to the base, if it exists, else it loiters around the base itself. With orders prepared, colonel actually gives orders to sergeants who are in groups satisfying strength requirements for their targets. Any remaining sergeants are ordered to hold either entrance to base or base itself.

2.3 Neural network for playstyle selection

Neural networks are machine learning systems loosely based on the function of biological brain. Every such network contains layers of discrete units - neurons. Outputs from all neurons in any given layer are used as inputs for successive layer [10]. Every neuron takes a vector of weighted, real-valued inputs, sums them and outputs a result of application of its activation function on this sum. All neurons in a network have the same activation function [10]. Goal of the learning process is to determine values for all input weights, such that the final outputs match desired values [10].

27 2. TenguAI

For the purposes of this AI, it was decided ConvNetJS1 library was to be used. This has a rather simple reason - it was the only ECMA 5.1 compatible neural network library I was able to find. ConvNetJS offers several options when training neural networks. Of these, we are interested in method, activation function and L2 decay. Method specifies algorithm to use for automatic adjustment of the learning rate of the trained network. L2 decay represents how tightly is the network fitted onto training data. Generally, high L2 decay results in very general networks, often to detriment of their accuracy.[11] The training of this network was done on a set of 727 games played against several different AIs under different conditions on multiple maps, all further detailed in appendix A. Opponents were placed on a random starting position. During each game, following procedure was followed: 1. Pick a random plan from the list. Save current game state information. 2. After approximately 30 seconds, evaluate current game state and compare it to previous one. If an advantage in field chosen in step 1 was attained or improved, output previous state information with the plan ID. Else, pick a different plan at random. The idea behind this is - if the AI is able to maintain an advantage throughout the course of a match, it should result in a victory. Data from these games, in total 6701 entries, was divided into training and test sets by an 80/20 split. The training set was then fed into and evaluator script in an attempt to obtain the best parameters for final network. For methods, ’Adadelta’ and ’Adagrad’ were evaluated in combination with following: ∙ One of sigmoid, tanh and ReLU2 (rectifier) for activation function; ∙ L2 decay of values 0.1, 0.05, 0.01, 0.005, . . . , 0.0001; ∙ Up to three neuron layers with number of nodes in range [5, 50] in increments of 5.

1. https://cs.stanford.edu/people/karpathy/convnetjs/index.html 2. http://proceedings.mlr.press/v15/glorot11a/glorot11a.pdf

28 2. TenguAI

Within this script, training data were further split into training and evaluation sets, again by 80/20 ratio. Out of these options, the best accuracy was achieved by Adadelta3 with rectifier activation function on 0.0001 L2 decay with 2neuron layers of 40 and 5 neurons, respectively. Its accuracy was 88.71 % and thus these settings were used to train the final neural network. This network was tested on the yet unseen 20 % of data and its final accuracy was measured at 66 %. Then, another set of 558 games has been run, this time with the network managing macro plan selection. The results can be seen in table B.2, in comparison with pre-network statistics (table 2.2). Draw result was declared when the match was not over after two hours of playtime. Table 2.2: Match statistics during learning process, per opponent.

NullBot Bonecrusher Cobra Total Win 119 73 75 267 Loss 95 155 155 405 Draw 20 24 11 55 Win Rate 50.85 % 28.96 % 31.12 % 36.72 %

Table 2.3: Match statistics with learned neural network, per opponent.

NullBot Bonecrusher Cobra Total Win 88 49 54 191 Loss 89 101 122 312 Draw 19 32 4 55 Win Rate 44.90 % 26.92 % 30.00 % 34.23 %

From this data, it appears the inclusion of neural network had no to slightly detrimental effect on the AIs performance. This may be due to low accuracy of the network or poorly chosen variables chosen as input. However, seeing the win rate dropped by 2.49 %, it may

3. https://www.matthewzeiler.com/mattzeiler/adadelta.pdf

29 2. TenguAI also very well be simply circumstantial occurence - opponents simply happened to play better in this set of games. Regardless of the cause, success of the AI is mediocre at best.

30 3 Conclusion

After all is said and done, TenguAI comes out as mediocre to subpar in terms of its competitive ability. However, the working implementa- tions of both map analysis and neural network open doors for more developers to pick up the proverbial torch and include these subsystems in designs of their own, hopefully opening new door in AI capability within Warzone 2100. With regard to TenguAI and its possible extensions, more work can definitely be done on its micromanagment subsystems, target acquisition and sergeant coordination. It may also be worthwhile endeavor to implement second neural network, focused on selection of micromanagement plans. Another option to enhance the AI would be to develop a multi criteria optimisation system for unit designer, making priority list of attributes and designing units based on that. Finally, the AI was released to the community under CC0 license.

A Maps used for neural network training

Note: all pictures present are screenshots of minimaps within the game.

Figure A.1: Small map with tight corridors and perfect early game chokepoint in the middle.

Figure A.2: Water-separated continents with varying amounts of players. Ideal for testing analyser and cross-water attacks of the AI.

33 A. Maps used for neural network training

Figure A.3: Sizeable, but tight map with hill-separated pathways. Good for concentrated defenses and macro movement.

Figure A.4: Small but spacious map. Good for quick matches that do not require fancy movements.

34 A. Maps used for neural network training

Figure A.5: Personal favourite. Tight, highly asymmetric starting positions.

Figure A.6: Similar in concept to Monocot, but with more resources and corridors.

B Statistics of matches before and after the use of neural network

Table B.1: Match statistics during learning process, per map and game setting.

Monocot Fish Nets Manhattan Miza Maze Win 59 48 39 35 Loss 102 78 49 66 Draw 1 7 20 7 Win Rate 36.42 % 36.09 % 36.11 % 32.41 %

Mountain Rush Scavengers No Scavengers Win 31 55 146 121 Loss 59 51 184 221 Draw 18 2 33 22 Win Rate 28.70 % 50.93 % 40.22 % 33.24 %

37 B. Statistics of matches before and after the use of neural network Table B.2: Match statistics with learned neural network, per map and game setting.

Monocot Fish Nets Manhattan Miza Maze Win 19 24 37 45 Loss 37 42 46 58 Draw 0 4 25 5 Win Rate 33.93 % 34.29 % 36.11 % 34.26 %

Mountain Rush Scavengers No Scavengers Win 29 37 93 98 Loss 58 71 151 161 Draw 21 0 35 20 Win Rate 26.85 % 34.26 % 33.33 % 35.13 %

38 Bibliography

1. WZ2100 Guide: Early-game multiplayer strategy [online] [visited on 2019-11-11]. Available from: http : / / betaguide . wz2100 . net/earlygamestrategy.html. 2. Introduction to Warzone 2100 [online] [visited on 2019-11-11]. Available from: http://betaguide.wz2100.net/Introduction. html#. 3. A History of Real-time Strategy Games [online] [visited on 2019-11-08]. Available from: https://web.archive.org/web/ 20110427052656/http://gamespot.com/gamespot/features/ all/real_time/. 4. WZ2100 Guide: Research tree [online] [visited on 2019-11-11]. Available from: http://betaguide.wz2100.net/Research.php? tree=1. 5. WZ2100 Guide: Weapon modifiers [online] [visited on 2019-11-11]. Available from: http://betaguide.wz2100.net/weapons.php? damageModifiers=1. 6. Bonecrusher thread on WZ2100 forum [online] [visited on 2019-11- 14]. Available from: http://forums.wz2100.net/viewtopic. php?f=49&t=12758. 7. MILLINGTON, Ian; FUNGE, John David. Artificial Intelligence for Games. 2nd ed. Burlington: Morgan Kaufmann Publishers, 2009. ISBN 978-0-12-374731-0. 8. JAIN, Anil K. Fundamentals of Digital Image Processing. Eaglewood Cliffs: Prentice Hall, Inc., 1989. ISBN 0-13-336165-9. 9. RUSSEL, Stuart J.; NORVIG, Peter. Artificial Intelligence: a modern approach. 3rd ed. Upper Saddle River: Prentice Hall, Inc., 2010. ISBN 978-0-13-207148-2. 10. MITCHELL, Tom M. Machine Learning. Boston: McGraw-Hill, 1997. ISBN 0070428077. 11. ConvNetJS - Deep learning in your browser. Available also from: https : / / cs . stanford . edu / people / karpathy / convnetjs / docs.html.