Transfer of Driving Behaviors Across Different Racing Games
Total Page：16
File Type：pdf, Size：1020Kb
Transfer of Driving Behaviors Across Different Racing Games Luigi Cardamone, Antonio Caiazzo, Daniele Loiacono and Pier Luca Lanzi Abstract— Transfer learning might be a promising approach known as Transfer Learning (TL) [26], [23]. In this paper, to boost the learning of non-player characters’ behaviors by we show how transfer learning can be applied to exploit the exploiting some existing knowledge available from a different knowledge gathered in one source domain for accelerating game. In this paper, we investigate how to transfer driving behaviors from The Open Racing Car Simulator (TORCS) to the learning in a target domain. We take into account two VDrift, which are two well known open-source racing games racing games with many differences related to the physics featuring rather different physics engines and game dynam- engine and the game dynamics: TORCS and VDrift. In ics. We focus on a neuroevolution learning framework based particular we focus on a driving behavior represented by a on NEAT and compare three different methods of transfer neural network and evolved using neuroevolution. We study learning: (i) transfer of the learned behaviors; (ii) transfer of the learning process; (iii) transfer of both the behaviors how the knowledge already available for TORCS can be used and the process. Our experimental analysis suggests that all to accelerate the learning of a driving behavior for VDrift. In the proposed methods of transfer learning might be effectively general, transfer learning, involves the transfer of policies or applied to boost the learning of driving behaviors in VDrift functions that represent a particular behavior. In this work, by exploiting the knowledge previously learned in TORCS. In we use the term transfer learning in a broader sense since we particular, transferring both learned behaviors and learning process appears to be the best trade-off between the final study also how it is possible transfer the fitness function and performance and the computational cost. the evaluation mechanism across different domains. In our experimental analysis we compare three different approaches I. INTRODUCTION for transfer learning: (i) transfer of the learned behaviors, by In the recent years, the computer games market is growing copying the existing behaviors from the source domain to fast both in the number of published titles and in the the target domain; (ii) transfer of the learning process, by complexity of the games. To boost the game development it is replicating in the target domain the evaluation mechanism possible to use third-party packages like graphics engines and and the fitness designed for the source domain; (iii) transfer physics engines. In this context, it became important to have of both the behaviors and the process, by replicating the some techniques to speed-up also the development of the evolutionary process of the source domain and by performing Artificial Intelligence (AI) of the game, i.e., the behaviors of an incremental learning using the source behavior as a seed. the non-player characters (NPCs). There is the need of some The results shows that all the three approaches work and the sort of pluggable AI, i.e., a set of algorithms independent adaptation technique has the best trade-off in term of driving from the specific game and from its representation, that can performance and evolution time. This paper is organized work on different games to achieve some common tasks (e.g. as follows: in Section II, we review the related works navigating a 3D environment, driving, aiming, etc.). about racing games and transfer learning; in Section III, A first step towards a pluggable AI has been done with the we present the neuroevolutionary approach used; in Section General Game Playing (GGP) competition [10]. In this com- IV, we describe the racing games used for the experiments; petition an agent has to play an unknown game described in in Section V, we describe the sensors and the actuators of first order logic. Although good results have been achieved, the driver; in Section VI, we present three approaches for the agents can only play games that are deterministic and applying transfer learning and the results achieved; finally in with perfect information (like board games), where it is easy Section VII, we present the conclusions of the work. to simulate the future states of the game given the description of the rules. These conditions usually do not hold in many II. RELATED WORK computer games where the agent has to control continuous Some of the early works on racing games were performed outputs and the future game states cannot be predicted. by Togelius et al. using a simple java-based 2D car racing A different approach for boosting the development of the game. In [27], [28], Togelius et al. investigated several AI of NPCs is to re-use an existing AI already developed schema of sensory information (e.g., first person based, third in a similar game. This type of problem is part of the area person based, etc.) and studied the generalization capabilities of the evolved controllers. Luigi Cardamone ([email protected]), Antonio Caiazzo (anto- More recently, Cardamone et al. [6], [5] applied neu- [email protected]), Daniele Loiacono ([email protected]) and Pier Luca Lanzi ([email protected]) are with the Dipartimento di roevolution to evolve a driving and an overtaking behavior, Elettronica e Informazione, Politecnico di Milano, Milano, Italy. Pier Luca in a sophisticated 3D car racing simulator (TORCS). The Lanzi ([email protected]) is also member of the Illinois Genetic Al- overtaking issue was consider also in [18] using a modular gorithm Laboratory (IlliGAL), University of Illinois at Urbana Champaign, Urbana, IL 61801, USA. fuzzy architecture and in [15] using Reinforcement Learning. In [9] evolutionary computation is applied to find the optimal III. NEAT racing line between the shortest path and the maximum In this study, we focused on Neuroevolution with Aug- curvature path of the track. Several works, like [16] [7] [29], menting Topology or NEAT [20], one of the most suc- applied imitation learning both to learn from human data cessful and widely applied neuroevolution approach. NEAT and to generate believable drivers, i.e., drivers with a human is specifically designed to evolve neural networks without driving-style. In [31] and in [8], genetic algorithms were assuming any a priori knowledge on the optimal topology applied to optimize the car setup using respectively the games nor on the type of connections (e.g., simple or recurrent Formula One Challenge and TORCS. Some of the most connections). NEAT is based on three main ideas. First, recent works [19], [3], [17] on racing games originate from in NEAT the evolutionary search starts from a network the submission to the Simulated Car Racing Championship topology as simple as possible, i.e. a fully connected network [14], an international competition organized by Loiacono et with only the input and the output layers. Complex struc- al. where the goal is to develop the best driver for TORCS tures emerge during the evolutionary process and survive able to race alone and against the other competitors. only when useful. Second, NEAT deals with the problem Finally, the computational intelligence techniques have of recombining networks with different structures through been also applied to some commercial racing games. In an historical marking mechanism. Whenever a structural Colin McRae Rally 2.0 (Codemasters) a neural network mutation occurs, a unique innovation number is assigned is used to drive a rally car, thus avoiding the need to to the gene representing this innovation. This mechanism handcraft a large and complex set of rules [11]: a feedforward is then used to perform the recombination and to identify multilayer neural network has been trained to follow the similarities between the networks without the need of a ideal trajectory, while the other behaviors ranging from the complex and expensive topological analysis. Third, NEAT opponent overtaking to the crash recovery are programmed. protects the structural innovations through the mechanism In Forza Motorsport (Microsoft) the player, can train his own of speciation. The competition to survive does not happen drivatars, i.e., a controller that learns the player’s driving in the whole population but it is restricted to niches where style and that can take his place in the races. all the networks have a similar topology. Therefore, when a In this work, we exploit the knowledge available in one new topology emerges, NEAT has enough time to optimize it. game (TORCS) to accelerate the learning in an another game Again, the historical marking is used to identify the niches by (VDrift). This type of problem belongs to the area known as using the innovation numbers of the genes for measuring the Transfer Learning (TL) [26], [23]. This area studies how the similarities between different topologies. To prevent a single knowledge learned on a source task can be exploited to speed species from taking over the population, the networks in the up the learning on a new target task. Several methods of same niche share their fitness. Then, in the next generation transfer learning have been specifically devised to be applied more resources will be allocated to the species with greater in Reinforcement Learning (RL) or Temporal Difference fitness. (TD) learning [21]. Such methods either focus on the transfer of value functions (e.g., [22], [24]) or experience instances IV. OPEN SOURCE RACING GAMES (e.g., [13]). Only few works in the literature (see [23] for In this section we review all the open-source car racing an overview) focused on transferring the learned policies games currently available. We focus only on open-source and, thus, can be applied to the methods that perform a racing games because the possibility of modifying the source search in the policy space (like NEAT [20]).