Autoencoder and Evolutionary Algorithm for Level Generation in Lode Runner

Sarjak Thakkar, Changxing Cao, Lifan Wang, Tae Jong Choi, Julian Togelius New York University New York, USA {tsarjak, cc5766, lw2435, tc3045, julian.togelius}@nyu.edu

Abstract— Procedural content generation can be used to Runner. In addition, we hse a multi-population evo- create arbitrarily large amounts of game levels automati- lutionary algorithm that contains patch-wise crossover cally, but traditionally the PCG algorithms needed to be and latent variable mutation operators, which can re- developed or adapted for each game manually. Procedural Content Generation via Machine Learning (PCGML) har- tain some interesting features of original levels while nesses the power of machine learning to semi-automate the inserting less predictable variations. Our experimental development of PCG solutions, training on existing game results showed that the proposed method is able to content so as to create new content from the trained models. create diverse levels. Besides, we found out that vari- One of the machine learning techniques that have been ational autoencoders prone to generate more complex suggested for this purpose is the autoencoder. However, very limited work has been done to explore the potential levels than vanilla autoencoders. of autoencoders for PCGML. In this paper, we train au- toencoders on levels for the platform game Lode Runner, II.RELATED WORK and use them to generate levels. Compared to previous In the past, Summerville et al. used LSTMs on the work, we use a multi-channel approach to represent content in full fidelity, and we compare standard and variational string representation of the map for PCG [3]. Snodgrass autoencoders. We also evolve the values of the hidden layer et al. suggested learning statistical patterns from exist- of trained autoencoders in order to find levels with desired ing levels generated by humans, and use those patterns properties. for training Markov Chains and for generating new levels from such Markov Chains [4]. In 2016, Font et al. I.INTRODUCTION suggested using evolutionary methods for generating levels by using two-phase level encoding [5]. In 2017, Procedural content generation (PCG) techniques are Samuel Alvernaz et al. trained an autoencoder to create used for game content for several reasons, including a comparatively low-dimensional representation of the introducing variation to the player, enabling particular environment observation, and then use CMA-ES to game aesthetics, and automating some parts of game train neural network controllers acting on this input development. Most PCG algorithms need to be devel- data [6]. The bottom line of all these approaches is that oped or tailored to each game. In contrast, PCGML uses different representations of the same game combined machine learning to generate content, using models with different algorithms achieve varying results. trained on existing content [1]. On the other hand, Lode Runner has a much larger One machine learning method that has only been variety of chunks, and Lode Runner maps rely heavily cursorily investigated for PCGML is the autoencoder, on puzzle-like structures and mazes in the level de- a type of neural network used for feature compres- sign [7]. However, our approach of using variational sion. It was recently proposed to use autoencoders autoencoder to learn the complex underlying structure for tasks such as level generation, level repairing, and of Lode Runner levels, coupled with an evolutionary level recognition [2]. However, that study really only algorithm to increase the playability score of the levels, scratched the surface. Among other things, the autoen- generated good results. Notably, we find most of the coder in that study using a single channel for all types raw generated levels from autoencoder as not playable of tiles, severely limiting the usefulness of the method. until the evolution step is carried out. We will show In this paper, we build on the recent research on more details of the technique we used in the following using autoencoders for PCG [2]. We propose a multi- sections. channel encoding for autoencoders to learn the struc- ture and properties of the levels of Lode Runner. III.OUR APPROACH Concretely, the proposed method uses a five channel We selected a game called Lode Runner in order encoding to separately learn the patterns of the bricks, to evaluate our approach. We used the multi-channel ladders, ropes, enemies, golds of the levels of Lode encoding for the 150 original levels and trained an autoencoder using that data. The autoencoder is used 978-1-7281-1884-0/19/$31.00 c 2019 IEEE to generate a pool of 10,000 levels by inputting ran- Parent Level 1 Parent Level 2 dom seeds to the decoder. We evolved these levels by XXX OOO passing them through the evolutionary process. XXX OOO XXX OOO A. Multi-Channel Encoding Crossover While selecting the format of level data to be used XXX for an autoencoder, the primary goal was to make the XXX computation for encoding and decoding more logically XOO feasible and reduce as much complexity as possible. In order to achieve this, we divided each level into various blocks that represent entities such as bricks, Mutate Variational Latent Space Variational ladders, ropes, etc. By doing so, we reduced each level Encoder Decoder to a small size of 22x32 blocks. This representation can be considered as a 2D string, where each character of the string represents a block of a level (Fig. 1). For example, we used ’b’ for a brick and ’-’ for a rope. The Second Generation Level string representation enables our autoencoder to learn the underlying structure of a level. Fig. 3: Evolutionary Process of Proposed Method Each character of a string is then later converted to a binary encoding for separate channels. In other words, Diederik et al. showed that the posterior inference can the proposed method uses separate channels for bricks, be made especially efficient by fitting an approximate ladders, ropes, etc. Each channel has 1 at the position inference mode [8]. The variational autoencoder is more depending on whether the particular block exists in advanced as it creates a distribution for each latent a level, and 0 otherwise. This representation can be variable [9]. Hence, the variational autoencoder is able compared to the RGB channels of a color image. We to reproduce images with more variations, compared to used five channels for the proposed method. Fig. 2 the vanilla autoencoder. We employed the variational shows a brick and ladder encodings of a level. autoencoder in the proposed method. One is to produce a large dataset that is up to 10,000 images as a global gene pool before the evolutionary process; the other is used in the evolutionary process to perform mutation on the latent representation of the level. C. Evolutionary Process After creating a global gene pool, the proposed method uses the evolutionary operators such as crossover and mutation to generate playable levels. For the level generation in Lode Runner, the proposed method uses the specially designed crossover and mu- Fig. 1: Example of String Representation of Level tation operators, fitness function, and multi-population structure, which will be explained in the following subsections. 1) Crossover and Mutation Operators: In order to retain some interesting features of original levels, we applied a patch-wise crossover. As illustrated in Fig. 3, the parent images are divided into nine patches, and a child image inherits the first seven patches from parent 1 and the last two from parent 2. For the mutation, the proposed method takes the child image, passes it through the autoencoder, and randomly mutate its Fig. 2: Bricks (Red) and Ladders (Green) Encoding latent variable. The parameters for the crossover and mutation operators can be found in Table I. Fig. 3 illustrates the process of the mutation operator. B. Variational Autoencoder 2) Fitness Function: This section describes the fitness Autoencoders have been used a variety of fields function of the proposed method, which is consists of such as images, sounds, music. The representation playability and connectivity. of any data in the latent variable of an autoencoder • Playability: We employed a path-finding algorithm can be used to reconstruct the original data from it. (PFA) to check if all the golds of a generated level TABLE I: Used Control Parameters a brick, it keeps falling until it reaches a rope, ladder or brick. We employed A* algorithm as a PFA of the Crossover Mutation Population Number of Rate Rate Size Generations proposed method to find the shortest path for collecting 0.78 0.1 32 150 all golds. Fig 5 is the solution path of an original level from A* algorithm. An extension of this approach, where the PFA tries to reach every block of the level helps to determine the connectivity of the level. can be collected. If they can be collected, the level is playable; otherwise, not playable. • Connectivity: The connectivity measure evaluates the percentage of coordinate points in a generated level that can be reached. The proposed method checks the playability measure first, and only playable levels are chosen to evolve. After that, the proposed method applies the connec- tivity measure in order to evaluate the fitness of each playable level. In other words, the more blocks are Fig. 5: Shortest Path from A* Algorithm connected, the higher fitness scores are returned. We employed a PFA to check what proportion of the coordinate in a level is reachable. IV. RESULTS 3) Multi-Population Structure: Since we faced the A. Results of Proposed Method with Variational Autoen- lack of variations in generated levels from one evo- coder lution iteration, we applied a multi-population struc- Fig. 6(a) consists of four randomly chosen levels ture to avoid the premature convergence. The pro- out of 10,000 generated levels from the variational posed method maintains ten populations separately autoencoder (before evolution) with the parameters as and evolves ten unique and better image sets, which described in Table II. are then passed onto the next iteration. Since each of the ten populations contains its unique local characteristics, TABLE II: Parameters of Variational Autoencoder combining them creates more diverse image sets with several different characteristics revealed. Fig 4 shows Training Noise Learning Number of the multi-population structure. Data Dimension Rate Epochs 300 16 1e-3 1000

From Fig. 6(a) we can observe the variational autoen- coder results. In these results, the left-upper one is not valid because all of the golds cannot be collected. As we mentioned above, only playable levels are used to generate child levels. Fig. 7(b) consists of two randomly chosen results after the evolutionary process combined with the variational autoencoder. Since the vanilla and variational autoencoders generate levels with varying densities, we decided to use them separately to gen- erate dense or sparse levels as hard and easy levels, shown in the next. B. Results of Proposed Method with Vanilla Autoencoder Fig. 4: Multi-Population Structure of Proposed Method Fig. 7(a) consists of two randomly chosen results af- ter the evolutionary process combined with the vanilla autoencoder, which are sparse and easy for rookies. D. Path Finding Algorithm From Fig. 7(b), we can see that the results generated A PFA in the proposed method is to check the with variational autoencoder are dense and hard for playability and connectivity of generated levels. Based rookies. on the rules regarding the player movement in Lode 1) Level Differences Between Variational and Vanilla Au- Runner, we developed an algorithm, which acts as a toencoders: Compared to a vanilla autoencoder, a vari- playing agent. Some of those rules include - the player ational autoencoder adds regularization constraints to can move only up or down on the ladder; the player the encoding process by forcing the generated implicit can dig the bricks; if the player falls off the edge of vector to follow a standard normal distribution. On (a) Sample Levels Generated by Vanilla Autoencoder

(a) Variational Autoencoder with Gaussian Noise

(b) Sample Levels Generated by Variational Autoencoder

Fig. 7: Results After Evolutionary Process

mutation, which helps to retain some interesting fea- tures of original levels while inserting less predictable variations. Our experimental results showed that the (b) Vanilla Autoencoder with Uniform Noise proposed method can create large amounts of signifi- cantly different playable levels. We also discussed the Fig. 6: Results of Variational and Vanilla Autoencoders difference in generated levels of the variational and vanilla autoencoders. For future work, we will apply the proposed method to other games such as Super applying evolutionary algorithm on variational autoen- Mario Bros and Mario Kart. coder, generated levels grow more branches of details with small mutations in the latent space, eventually REFERENCES making generations different. On the other hand, the [1] A. Summerville, S. Snodgrass, M. Guzdial, C. Holmgard,˚ A. K. vanilla autoencoder does not regularize the implicit Hoover, A. Isaksen, A. Nealen, and J. Togelius, “Procedural con- vector, as a result of which, the mutation results in tent generation via machine learning (pcgml),” IEEE Transactions lesser details. on Games, vol. 10, no. 3, pp. 257–270, 2018. [2] R. Jain, A. Isaksen, C. Holmgard,˚ and J. Togelius, “Autoencoders for level generation, repair, and recognition.” C. Similarity of Generated Levels to Original Levels [3] A. J. Summerville and M. Mateas, “Super mario as a string: We compared all of the 330 generated levels with the Platformer level generation via lstms,” 2016. [4] S. Snodgrass and S. Ontan˜on,´ “Experiments in map generation original levels of the game, block by block (ignoring using markov chains.” the blank space) to calculate the similarity proportion. [5] J. M. Font, R. Izquierdo, D. Manrique, and J. Togelius, “Con- Table III shows the mean similarity and the number of strained level generation through grammar-based evolutionary algorithms,” in European Conference on the Applications of Evolu- generated levels that are over 30% and 50% similar to tionary Computation. Springer, 2016, pp. 558–573. any of the existing levels of the game, respectively. [6] S. Alvernaz and J. Togelius, “Autoencoder-augmented neuroevo- lution for visual doom playing,” in 2017 IEEE Conference on Computational Intelligence and Games (CIG), Aug 2017, pp. 1–8. TABLE III: Similarity of Generated Levels [7] S. Snodgrass and S. Ontanon, “A hierarchical mdmc approach to Mean Num Levels Num Levels 2d video game map generation,” in Eleventh Artificial Intelligence Similarity >30% Similarity >50% Similarity and Interactive Digital Entertainment Conference, 2015. Variational 25.36% 73 / 330 3 / 330 [8] D. P. Kingma and M. Welling, “Auto-Encoding Variational Vanilla 27.05% 130 / 330 38 / 330 Bayes,” Dec 2013, p. arXiv:1312.6114. [9] I. Gulrajani, K. Kumar, F. Ahmed, A. A. Taiga, F. Visin, D. Vazquez, and A. Courville, “PixelVAE: A Latent Variable Model for Natural Images,” arXiv e-prints, p. arXiv:1611.05013, V. CONCLUSIONS Nov 2016. In this paper, we proposed a multi-channel encoding for an autoencoder with a multi-population evolu- tionary algorithm to generate levels for Lode Run- ner. We used patch-wise crossover and latent variable