GRUE: An Architecture for Agents in Games and Other Real-Time Environments Elizabeth Gordon, BS, MSc. Thesis submitted to the University of Nottingham for the degree of Doctor of Philosophy March 2005 Abstract This thesis presents an architecture, which we call GRUE, for intelligent agents in real-time dynamic worlds. Such environments require agents to be able to flexibly adjust their behaviour to take into account changes in the environment or other agents’ actions. Our architecture is based on work done in robotics (Nilsson, 1994; Benson and Nilsson, 1995; Benson, 1996), which also deals with complex, dynamic environments. Our work focuses on goal arbitration, the method used by the agent to choose an appropriate goal for the current situation, and to re-evaluate when the situation changes. In the process, we have also developed a method for representing items in the environment, which we call resources, in terms of their properties. This allows the agent to specify a needed object in terms of required properties and use available objects with appropriate properties interchangeably. We show that the GRUE architecture can be used successfully in both a typical AI test bed and a commercial game environment. In addition, we have undertaken to experimentally test the effects of the features included in our architecture by comparing agents using the standard GRUE architecture to agents with one or more features removed and find that these features do improve the performance of the agent where expected. ii Acknowledgments First of all I would like to thank Sony Computer Entertainment Europe, who provided the majority of my funding, and express my gratitude to Ian Wright and Dave Ranyard who made it possible. I would also like to thank the School of Computer Science for providing the remainder of my funding. Next I would like to express my gratitude to my supervisor, Brian Logan, for all his help in getting to this point. I would also like to thank Nick Hawes, for allowing me to use his Sim Agent code for connecting to Unreal Tournament, and Mike Lees, who allowed me to use his Tileworld implementation. Thanks also to Neil Madden and Mike Lees for help proofreading this document. Finally, thanks to Jamie Strachan for proofreading above and beyond the call of duty, as well as help, support, advice, and generally putting up with me. iii iv Contents List of Figures vi List of Tables viii 1 Introduction 1 1.1 The Problem Addressed 5 1.2 Summary of Work Done 8 1.3 Applications to Other Problem Domains 12 1.4 Overview of Thesis 16 2 Approaches to Agent Development 17 2.1 Behaviour-Based Architectures 19 2.2 BDI Architectures 22 2.3 The Cognitive Modeling Approach: Soar 23 2.4 The Cognition and Affect Project 26 2.5 Teleo-reactive Programs and the Teleo-reactive Architecture 29 2.6 Summary 34 3 The GRUE Architecture 36 3.1 Benson and Nilsson’s Architecture 36 3.2 GRUE: A New Architecture 43 3.3 Correcting Deficiencies of the Architecture Design 67 3.4 Comparing GRUE to Similar Systems 68 4 Implementation 73 4.1 Sim Agent 74 4.2 Implementation of GRUE Components 75 4.3 The GRUE Tileworld Agent 82 4.4 The GRUE Unreal Tournament Agent 96 4.5 Properties of TRPs 110 5 Evaluation of the GRUE architecture 112 5.1 The Tileworld Environment and Experimental Results 116 5.2 The Unreal Tournament Environment 158 CONTENTS v 6 Extensions to the GRUE Architecture 182 6.1 Flaws in the GRUE architecture 182 6.2 Garbage Collection 184 6.3 Task Suspension 185 6.4 Combining Actions 186 6.5 Optimizing the Architecture 187 6.6 Dynamic Filter adjustment 189 6.7 Meta-Management 190 6.8 Extensible Agents 191 6.9 Addition of a Reactive Layer 192 6.10 Planning 192 6.11 Learning 193 7 Conclusions 195 7.1 Implementation of GRUE Agents 197 7.2 Summary of Experimental Results 200 7.3 Changes to the Architecture Description 207 7.4 Limitations of GRUE 209 7.5 Wider Applications of Resource-Based Goal Arbitration and GRUE 211 Appendices A Summary of Architecture Components and Data Structures 213 B Resources used in Tileworld 218 C Resources used in Unreal Tournament 221 D Code for the Components of GRUE 225 D.1 File: grue components.p 225 D.2 File: grue binding.p 231 D.3 File: grue property utilities.p 242 D.4 File: grue number utilities.p 244 D.5 File: grue utilities.p 253 E Tileworld Specific Code 264 E.1 Tileworld Goal Generators 264 E.2 Tileworld Actions 268 E.3 Get Tile TRP 273 E.4 Fill Hole TRP 282 E.5 Avoid Obstacle TRP 292 CONTENTS vi F Unreal Tournament Specific Code 299 F.1 Unreal Tournament Goal Generators 299 F.2 The go to TRP 315 F.3 Other TRPs 324 G T-Tests for the Tileworld Agent 347 References 352 vii List of Figures 3.1Pseudocodeteleo-reactiveprogramsforaPac-Managent........... 39 3.2Ruleswithdisjunctions.............................. 42 3.3GRUE....................................... 44 3.4 Graph of utility for the value range #:5–>10:#................ 53 3.5 Graph of utility for the value range #|5–>10:#................ 53 3.6 Graph of utility for the value range #|5–>:#................. 54 3.7 Graph of utility for the value range #:<–5|# ................. 54 3.8AGRUETRPforaPac-Managent....................... 60 4.1PseudocodefortheBindingAlgorithm..................... 78 4.2TheCTF-simpleUnrealTournamentmap.................... 101 4.3RelativePrioritiesforGoalsinUnrealTournament............... 104 5.1TheTileworld................................... 117 5.2 Comparison of All Goals agent and the GRUE agent when N=70 and Density=10.................................... 122 5.3 Comparison of All Goals agent and the GRUE agent when N=10 and Density=10.................................... 123 5.4 Comparison of No Updates agent and the GRUE agent when N=70 and Density=100................................... 124 5.5ErrorcaseinTileworldwithincorrectpriorities................ 127 5.6 Comparison of Constant Priorities agent and the GRUE agent when N=70andDensity=100............................. 128 5.7 Comparison of All Goals Constant Priorities agent and the GRUE agentwhenN=70andDensity=10....................... 130 5.8 Comparison of All Goals Constant Priorities agent and the GRUE agentwhenN=10andDensity=10....................... 131 5.9 Comparison of Deleted Properties agent and the GRUE agent when N=100andDensity=40............................. 134 5.10 Comparison of Required Properties agent and the GRUE agent when N=100andDensity=70............................. 135 5.11 The GRUE agent compared to agents without preferred properties for N=100.136 LIST OF FIGURES viii 5.12 The GRUE agent compared to agents without numerical features with density setto40...................................... 137 5.13 Comparison of No Divisible agent and the GRUE agent when N=70 and Density=40.................................... 138 5.14 Comparison of No Ranges agent and the GRUE agent when N=70 and Density=40.................................... 139 5.15 Comparison of No Numerical agent and the GRUE agent when N=100 andDensity=40.................................. 140 5.16 GRUE agent compared to agents with binding features disabled with density setto40...................................... 142 5.17 Comparison of Non-exclusive agent, No Persistence agent and the GRUE agentwhenN=40andDensity=40....................... 143 5.18 Comparison of Non-exclusive agent, No Persistence agent and the GRUE agentwhenN=70andDensity=40....................... 144 5.19CaseinTileworldwheredisablingexclusivebindingsisbeneficial...... 146 5.20 GRUE compared to an agent with parallel execution of actions disabled with densitysetto40.................................. 148 5.21 Comparison of an agent with parallel execution of actions disabled and the GRUEagentwhenN=40andDensity=40................... 149 5.22 Comparison of an agent with parallel execution of actions disabled and the GRUEagentwhenN=70andDensity=40................... 150 5.23UnrealTournament................................ 158 5.24TheCTF-simpleUnrealTournamentmap.................... 159 5.25RelativePrioritiesforGoalsinUnrealTournament............... 160 5.26 Boxplot comparison for several agents in Unreal Tournament with no opponent166 5.27BoxplotcomparisonforseveralagentsinUnrealTournament........ 177 5.28 Boxplot comparison showing the difference in score between modified agents andthestandardGRUEagentinUnrealTournament............ 178 ix List of Tables 5.1ScoresforthestandardGRUEagentinTileworld............... 120 5.2ProcessortimetakenbythestandardGRUEagentinTileworld....... 121 5.3 Performance of an agent with all goals generated in all situations in Tileworld 122 5.4 Average processor time used by an agent with all goals generated in all sit- uationsinTileworld............................... 123 5.5 Performance of an agent with dynamic updating of goal priorities disabled . 124 5.6 Average processor time used by an agent with dynamic updating of goal prioritiesdisabledinTileworld......................... 125 5.7 Performance of an agent with constant goal priorities in Tileworld . 126 5.8 Average processor time used by an agent with constant goal priorities in Tileworld..................................... 127 5.9 Performance of an agent with goals generated in all situations and constant goal priorities in Tileworld . .................... 130 5.10 Average processor time used by an agent with goals generated in all situations and constant goal priorities in Tileworld .................... 131 5.11 Performance of an agent with preferred properties deleted in Tileworld . 133 5.12 Performance of an agent with preferred properties changed to required