Novelty Generation Framework for AI Agents in Angry Birds Style Physics Games Chathura Gamage Vimukthini Pinto Cheng Xue School of Computing School of Computing School of Computing The Australian National University The Australian National University The Australian National University Canberra, Australia Canberra, Australia Canberra, Australia [email protected] [email protected] [email protected] Matthew Stephenson Peng Zhang Jochen Renz Dpt. of Data Science and Knowledge Engineering School of Computing School of Computing Maastricht University The Australian National University The Australian National University Maastricht, The Netherlands Canberra, Australia Canberra, Australia [email protected] [email protected] [email protected] Abstract—Handling novel situations is a critical capability of to inject novelty and conduct controlled experiments [1], Artificial Intelligence (AI) agents when working in open-world we need test environments that simulate realistic novelties. physical environments. To develop and evaluate these agents, Physics simulation games, video games where the game world we need realistic and meaningful novelties, that is, novelties that are detectable and learnable. However, there is a lack of simulates real-world physics, offer simplified and controlled research in the area of creating novelties for AI agents in physical environments for developing and testing AI agents [3]. There- environments. Physics-based video games are popular among AI fore, physics simulation games are ideal platforms to introduce researchers due to the ability to create realistic and controllable realistic novelties and conduct controlled experiments of open- physical environments. In this paper, we present a systematic world learning systems. novelty generation framework for physics-based video games. This framework allows the user to define a specific objective when While novelty generation has been investigated in domains generating novel content that ensures detectability. We instantiate such as Monopoly [4] and Polycraft [5], to the best of our the proposed framework for the video game Angry Birds and knowledge, there are no existing automated approaches to gen- conduct experiments to show that the generated novel content is erate novelties in physics-based environments. In this paper, consistent with the user-defined objectives. Furthermore, we use a we present the first systematic novelty generation framework reinforcement learning agent to experiment with the learnability of the generated novel content. for physics-based environments. The framework generates Index Terms—open-world learning, novelty generation, physics meaningful novelties for AI agents. We consider a novelty based video games, Angry Birds to be meaningful if the novelty is detectable and learnable. The detectability and learnability are to ensure that agents I. INTRODUCTION can find and adapt to the novelty and improve performance Detecting and reacting to novel and unforeseen situations over tasks. The pipeline of the proposed framework is shown is a key feature of human intelligence and is still a challenge in Fig. 1. The framework consists of two components which for modern AI systems. Open-world learning is an emerging can, (1) generate novel game objects for a given user-defined research area that attempts to address this challenge [1]. objective and (2) inject generated novel objects to the game With the increasing reliance on autonomous systems operating content preserving the compatibility of the content for the in open-world environments, such as self-driving vehicles, game. We use Angry Birds as the test environment to introduce underwater exploration robots, and drone swarms, embedding our framework as it is a popular physics-based game for novelty detection and novelty adaption capabilities to AI is developing AI agents, with the long-running AI competition becoming more important. For example, a trained self-driving as part of the IJCAI conference [6]. car knows to stop at the red stop signs, but it may fail to The rest of the paper is structured as follows. We start by respond if it approaches a blue stop sign, which is not included providing the definition of novelty in the context of AI and dis- in its training data [2]. cuss related work in the areas of AI agent experimentation and To proceed with research in open-world learning, we need video games. Next, we present the overview of the proposed agents that can detect and adapt to novelty and environments novelty generation framework followed by the application to where novelty can be easily introduced in a controllable Angry Birds with the tests used to generate requested novel way. As real-world environments offer limited opportunities content. We then present examples of generated novel content from our framework followed by experiments with AI agents 978-1-6654-3886-5/21/$31.00 ©2021 Crown in Angry Birds to evaluate if the generated novel content meets Fig. 2: A level from the Angry Birds game. that can be used as drugs [10]. As open-world learning is Fig. 1: Overview of the proposed novelty generation frame- an emerging research area [1], [8], there is only a limited work. The Novel Object Generator generates novel game ob- number of prior work in generating novelties for AI agents. jects according to a Novelty Request satisfying a user-defined As discussed previously, Monopoly [4] and Polycraft [5] Novelty Objective. The Novelty Injector places the generated domains have been used to create novelties for open-world novel objects in game content ensuring the compatibility of agents. However, their approaches to novelty generation are the content in the context of the game. not publicly available. In the context of content generation for video games, Pro- the user-defined requirements. cedural Content Generation through Quality Diversity (PCG- QD), is a sub-branch of search-based procedural content gener- II. BACKGROUND AND RELATED WORK ation [11], which generates a substantial number of high qual- A. What is Novelty? ity varied artifacts. In PCG-QD, the underlying mechanism is Consider an unmanned underwater vehicle trained for near- evolutionary computation for multidimensional optimization. surface coastal areas. The vehicle has expertise for missions For the divergent search, evolution is not optimized using a in the trained setting. But when it enters a region with new fitness function which satisfies the eventual goal, but rewarded types of rocks or an environment with extremely hot water, the for the diversity of the solutions. This concept has been termed vehicle may fail to respond as the vehicle has not experienced by various researchers as increasing generality [12], surprise these situations before [1]. “Novelty” is described as situations [13], curiosity [14], or novelty [15]. PCG-QD has been used in that violate implicit or explicit assumptions about the agents, video games to generate 3D objects [16], weapons [17], and the environment, or their interactions [7]. Following this, [1] dungeons [18]. One of the major drawbacks of PCG-QD is and [8] explain different types of novelties that may occur that it has to explore multiple solutions in a multi-dimensional in open-world environments. In [5] novelty is explained as a space which can affect the performance of the algorithm. relative property to an agent’s past experience and cognitive Also, it is strenuous to find diversity from all the solutions capabilities. When an agent encounters an entity, if the agent obtained across the search space [19]. In our proposed work, cannot recall the entity from prior experience, or the agent the diversity that is introduced to the game from the novel cannot infer the entity through cognition, the encountered content is guaranteed through a series of tests. entity is considered novel for the agent. C. Angry Birds Following the novelty categorization in [5], we define two types of parameters available in the game objects of physics- Angry Birds is a physics-based puzzle game in which the based games as appearance parameters and physics parame- player shoots birds from a slingshot at pigs. These pigs are ters. Appearance parameters are the shape, size, and colour often covered by different physical structures and the goal is of an object which affects the visual appearance of the object. to kill all the pigs using the provided birds. A sample Angry Physics parameters are the mass, friction, bounciness, and etc., Birds level is shown in Fig. 2. Instead of the real Angry Birds which determine the physics characteristics of the object. In game environment, we used an open-source research clone of this paper, we consider two categories of novel objects which Angry Birds developed in Unity by Ferreira [20]. The main can be encountered in physics-based games, objects that are available in this game can be split into four • Category 1: Objects with new appearance parameter types: birds (red, blue, yellow, white, and black), blocks (ice, values to the already encountered objects in the game. wood, and stone), pigs, and explosives. Compared to already encountered objects, these objects While Angry Birds seems simple and easy to play for do not have additional parameters and they may or may human players, it is particularly challenging for AI agents not have the same physics parameter values. due to its large and effectively continuous action space, as well as unknown consequences for actions in advance [21]. • Category 2: Objects with the same appearance parameter values to the already encountered objects in the game, but A successful AI agent not only needs to learn the physical different physics parameter values. properties of game objects to correctly predict the outcome of an action but also needs to choose the desired action from the B. Related Work action space. To achieve the same or better performance as There is prior work on novelty generation such as generating compared to the best human players, agents need to address new types of symbols for digits [9] and designing molecules a number of issues from different areas of AI.