DEGREE PROJECT IN COMPUTER SCIENCE AND ENGINEERING, SECOND CYCLE, 30 CREDITS STOCKHOLM, SWEDEN 2017 Creating Human-like AI Movement in Games Using Imitation Learning CASPER RENMAN KTH ROYAL INSTITUTE OF TECHNOLOGY SCHOOL OF COMPUTER SCIENCE AND COMMUNICATION Creating Human-like AI Movement in Games Using Imitation Learning May 31, 2017 CASPER RENMAN Master’s Thesis in Computer Science School of Computer Science and Communication (CSC) Royal Institute of Technology, Stockholm Swedish Title: Imitation Learning som verktyg för att skapa människolik rörelse för AI-karaktärer i spel Principal: Kristoffer Benjaminsson, Fast Travel Games Supervisor: Christopher Peters Examiner: Olov Engwall iii Abstract The way characters move and behave in computer and video games are impor- tant factors in their believability, which has an impact on the player’s experi- ence. This project explores Imitation Learning using limited amounts of data as an approach to creating human-like AI behaviour in games, and through a user study investigates what factors determine if a character is human-like, when observed through the characters first-person perspective. The idea is to create or shape AI behaviour by recording one’s own actions. The implemented framework uses a Nearest Neighbour algorithm with a KD-tree as the policy which maps a state to an action. Results showed that the chosen approach was able to create human-like AI behaviour while respecting the performance constraints of a modern 3D game. iv Sammanfattning Sättet karaktärer rör sig och beter sig på i dator- och tvspel är viktiga faktorer i deras trovärdighet, som i sin tur har en inverkan på spelarens upplevelse. Det här projektet utforskar Imitation Learning med begränsad mängd data som ett tillvägagångssätt för att skapa människolik rörelse för AI-karaktärer i spel, och utforskar genom en användarstudie vilka faktorer som avgör om en karaktär är människolik, när karaktären observeras genom dess förstapersonsperspektiv. Iden är att skapa eller forma AI-beteende genom att spela in sina egna hand- lingar. Det implementerade ramverket använder en Nearest Neighbour-algoritm med ett KD-tree som den policy som kopplar ett tillstånd till en handling. Re- sultaten visade att det valda tillvägagångssättet lyckades skapa människolikt AI-beteende samtidigt som det respekterar beräkningskomplexitetsrestriktio- ner som ett modernt 3D-spel har. Contents Contents v 1 Introduction 1 1.1 Artificial Intelligence in games . 1 1.1.1 Imitation Learning . 2 1.1.2 Human-likeness . 3 1.2 Objective . 3 1.3 Limitations . 4 1.4 Report outline . 4 2 Background 5 2.1 Imitation Learning . 5 2.1.1 Policy . 5 2.1.2 Demonstration . 5 2.1.3 State representation . 6 2.1.4 Policy creation . 6 2.1.5 Data collection . 6 2.1.6 Demonstration dataset limitations . 7 2.2 Related work . 7 2.2.1 Summary and state of the art . 10 2.3 Performance in games . 11 2.4 Measuring believability of AI . 11 2.4.1 Turing test-approach . 11 2.4.2 Automated similarity test . 12 2.5 Conclusion . 12 3 Implementation 13 3.1 Setting . 13 3.2 Method motivation . 13 3.3 Implementation . 14 3.3.1 Summary . 14 3.3.2 Recording movement and state representation . 15 3.3.3 Playing back movement . 16 v vi CONTENTS 3.3.4 Policy . 17 3.3.5 Feature extraction . 17 3.3.6 Avoiding static obstacles . 17 3.3.7 Avoiding dynamic obstacles . 19 3.3.8 KD-tree . 21 3.3.9 Discretizing the environment . 24 3.3.10 Additional details . 27 3.3.11 Storing data . 28 3.3.12 Optimization and measuring performance . 29 3.4 Overall implementation . 29 4 Evaluation 31 4.1 User study . 31 4.1.1 The set-up . 32 4.1.2 Participants . 34 4.1.3 Stimuli . 34 4.1.4 Procedure . 34 4.1.5 Hypothesis . 36 4.2 Results . 36 4.2.1 User study . 37 4.2.2 Imitation agent performance . 40 4.3 Discussion . 41 4.3.1 The imitation agent . 41 4.3.2 The user study . 41 4.3.3 Creating non-human-like behaviour . 43 4.3.4 Performance in relation to games . 43 4.3.5 Ethical aspects . 43 5 Conclusions 45 5.1 Future work . 46 5.1.1 Use outside of games . 47 Bibliography 49 Chapter 1 Introduction This chapter gives a brief overview of Artificial Intelligence in games, Imitation Learning and human-likeness. It also presents the objective, limitations and the outline of the project. 1.1 Artificial Intelligence in games Computer and video games produce more and more complex virtual worlds. This in- troduces new challenges for the characters controlled by Artificial Intelligence (AI), also known as agents [20] or NPC’s (Non-Player Characters), meaning characters that are not being controlled by a human player. The way characters move and behave in computer and video games are important factors in their believability, which has an impact on the player’s experience. Being able to interact with NPC’s in meaningful ways and feel that they belong in the world is important [4]. In Virtual Reality (VR) this is even more important, as the gaming experience is even more immersive. The goal of many games’ AI is more or less the same as attempts to beat the Turing test - to create believable intelligence [12]. A popular genre in computer and video games is First-person shooter (FPS). In an FPS game the player experiences the game through the eyes of the character the player is controlling, also known as a first-person perspective. Typically a player is at most able to see the hands and arms of the character the player is controlling. The player can however see the whole bodies of characters of other players’ characters or NPC’s (Non-Player Characters). This is visualized in Figure 1.1. 1 2 CHAPTER 1. INTRODUCTION Figure 1.1: An example first-person perspective game scenario, seen from the eyes of the character that the player controls. The blue and red characters are NPC’s. AI in games is traditionally based on Finite State Machines (FSM), Behaviour Trees (BT) or other hand-coded techniques [27]. In these techniques, a programmer needs to explicitly define rules for what an agent should do in different situations. An example of such a rule could be: "if the character’s health is low and the character sees a hostile character, the character should flee". These techniques work in the sense that the agent is able to execute tasks and adapt its behaviour to its situation, but the result is predictable and static [11]. For example, if a player sees an NPC react to a situation the same way it did in an earlier similar situation, the player can be quite sure that the NPC will probably always react like that given a similar situation. In 2006, Orkin [17] said: “in the early generations of shooters, such as Shogo (1998) players were happy if the A.I. noticed them at all and started attacking. Today, players expect more realism, to complement the realism of the physics and lighting in the environments”. In order to get more realism and unpredictability in order to increase the entertainment for the player, it would perhaps be a good approach for agents to imitate human behaviour. 1.1.1 Imitation Learning Imitation Learning (IL) is a technique where the agent learns from examples, or demonstrations, provided by a teacher [1]. IL is a form of Machine Learning (ML). ML has been defined as the “field of study that gives computers the ability to learn without being explicitly programmed” [14]. Unlike Reinforcement Learning algorithms, IL does not require a reward function to be specified. Instead, an IL algorithm observes a teacher perform a task and learns a policy that imitates the teacher, with the purpose of generalizing to unseen data [28]. IL is regarded as a promising technique for creating human-like artificial agents [3]. Some approaches have shown to be able to develop agents with good performance in non-trivial tasks using limited amounts of data and computational resources [3]. It is a technique 1.2. OBJECTIVE 3 which also can be used to dynamically change game play to adapt to different players based on their play style and skill [7]. 1.1.2 Human-likeness Shaker et al. [24] describe character believability, which says that an agent is believ- able if someone who observes it believes that the agent is a human being. Player believability on the other hand, says that the agent is believable if someone observ- ing the agent believes that it is a human controlling it. It is player believability that is meant by human-like in this project. 1.2 Objective The primary goal of this project is to describe a method for creating human-like agent movement using IL with limited amounts of data. The idea is to create an agent by recording one’s own actions, shaping it with desired behaviours. Most related works in the field of IL in games want to create competitive AI, meaning AI that is good at beating the game. This is not the case in this project. The goal is to create AI that lets an agent imitate a demonstrating human well, while respecting the performance requirements of a modern 3D game. A hope is that this will lead to a more unpredictable and human-like agent which in turn could lead to better entertainment for a player playing the game. Lee et al. [9] say that human-like agent behaviour leads to a raised emotional involvement of the player, which increases the players immersion in the game.