Neural Computing and Applications (2021) 33:19–37 https://doi.org/10.1007/s00521-020-05383-8 (0123456789().,-volV)(0123456789().,- volV) S. I : NEURAL NETWORKS IN ART, SOUND AND DESIGN Deep learning for procedural content generation 1 2 3 2,4 2,5,6 Jialin Liu • Sam Snodgrass • Ahmed Khalifa • Sebastian Risi • Georgios N. Yannakakis • Julian Togelius2,3 Received: 14 May 2020 / Accepted: 23 September 2020 / Published online: 8 October 2020 Ó Springer-Verlag London Ltd., part of Springer Nature 2020 Abstract Procedural content generation in video games has a long history. Existing procedural content generation methods, such as search-based, solver-based, rule-based and grammar-based methods have been applied to various content types such as levels, maps, character models, and textures. A research field centered on content generation in games has existed for more than a decade. More recently, deep learning has powered a remarkable range of inventions in content production, which are applicable to games. While some cutting-edge deep learning methods are applied on their own, others are applied in combination with more traditional methods, or in an interactive setting. This article surveys the various deep learning methods that have been applied to generate game content directly or indirectly, discusses deep learning methods that could be used for content generation purposes but are rarely used today, and envisages some limitations and potential future directions of deep learning for procedural content generation. Keywords Procedural content generation Á Game design Á Deep learning Á Machine learning Á Computational and artificial intelligence 1 Introduction & Julian Togelius Deep learning has powered a remarkable range of inven- [email protected]; [email protected] tions in content production in recent years, including new Jialin Liu methods for generating audio, images, 3D objects, network [email protected] layouts, and other content types across a range of domains. Sam Snodgrass It stands to reason that many of these inventions would be [email protected] applicable to games. In particular, modern video games Ahmed Khalifa require large quantities of high-definition media, which [email protected] could potentially be generated through deep learning Sebastian Risi approaches. For example, promising recent methods for [email protected] generating photo-realistic faces could be used for character Georgios N. Yannakakis creation in games. [email protected] At the same time, video games have a long tradition of procedural content generation (PCG) [132], where some 1 Guangdong Provincial Key Laboratory of Brain-inspired forms of game content have been generated algorithmically Intelligent Computation, Department of Computer Science and Engineering, Southern University of Science and for a long time; the history of digital PCG in games Technology, Shenzhen, China stretches back four decades. In the last decade and a half, 2 Modl.ai, Copenhagen, Denmark we have additionally seen a research community spring up around challenges posed by game content generation 3 New York University, New York, USA [16, 93, 112, 129, 133, 134, 148]. This research community 4 IT University of Copenhagen, Copenhagen, Denmark has applied methods from core computer science, such as 5 Institute of Digital Games, University of Malta, Msida, Malta grammar expansion [22]; AI, such as constraint solving 6 Technical University of Crete, Chania, Greece [115] and evolutionary computation [7, 133]; and graphics, 123 20 Neural Computing and Applications (2021) 33:19–37 such as fractal noise [24]. But only in the last few years and surveys in Sects. 2.1 and 2.2. Section 2.3 describes our have we seen a real effort to bring the tools of deep paper selection methodology. learning to game content generation. Deep learning brings new opportunities and leads to 2.1 Related work exciting advances in PCG, such as generative adversarial networks (GANs) [32], deep variational autoencoders A number of books and surveys of PCG with different (VAEs) [63] and long short-term memory (LSTM) focuses and aims have been published in the past two [34, 45]. However, those methods for other generative or decade [16, 93, 112, 129, 133, 134, 148]. The two text- creative purposes are not always applicable to games and books for PCG [112] and Game AI [148] cover the search- need certain adaptations due to the functionality criteria of based methods, solver-based methods, constructive gener- different game content. Methods for generating images ation methods (such as cellular automata and grammar- (e.g., generative networks) can be used to generate image- based methods), fractals, noise, and ad-hoc methods for like game content (e.g., level maps, landscapes, and generating diverse game content. De Kegel and Haahr [16] sprites). However, the generated levels should be playable reviewed the PCG methods for eleven categories of puz- and require specific gameplay skill-depth. The generated zles, but few work based on deep learning has been sprites should imply specific character or emotion, as well reported. The article by Togelius et al. reviews the search- as coherence within the game. Training reliable models based PCG methods, defined as using meta-heuristics to requires a necessary amount and quality of data, while the search in a predefined content space, not necessarily rep- available data of content and playing experience for most resented by the same format of the content itself, and games are limited. Careful consideration and sophisticated automatically generate new content [133]. The search is led design of adaptation techniques are requisites for applying by a fitness or evaluation function which measures the deep learning methods to generate game content. quality or playability of the generated content. The expe- It is important to note that content generation has uses rience-driven PCG framework [147] largely adopts a outside of designing and developing games for humans to search-based approach and reviews ways in which algo- experience. In addition to creating content in games meant rithms can generate content for adjusting the player expe- for humans to play, content generation can also play a rience. Most of the reviewed search-based methods in both crucial role in creating generalizable game-based and survey papers rely on evolutionary algorithms. In this game-like benchmarks for reinforcement learning and article, we also cover some search-based methods which other forms of AI [26, 136]. cooperated with deep learning methods for generating This article surveys the various approaches that have content. The most famous example may be latent variable been taken to generate game content with deep learning evolution [5]. Risi and Togelius [93] focuses on PCG for and also discusses methods proposed from within deep applications in Reinforcement Learning (RL), while the learning research that could be used for PCG purposes. work based on RL methods reviewed in this article mainly First, we give an overview of types of game content that used RL agents to play the generated levels, which indi- could conceivably be generated by deep learning, including rectly served as content evaluators. Khalifa et al. [62] the particular constraints and affordances of each content models the level generation as an iterative process that one type and examples of such applications (if they exist), needs to edit the levels to meet certain requirements or followed by an overview of applicable deep learning achieve some specific goals. RL agents need to learn to methods. generate levels through this iterative process. The study of Summerville et al. [129], published in 2018, reviews the PCG via Machine Learning (PCGML) methods, building 2 Scope of the review on e.g. Markov chains (e.g., [118–120, 131, 152]), n-grams (e.g., [14]), and Bayes nets (e.g.,[37]), whereas This article discusses the use of deep learning (DL) we will focus exclusively on deep learning in this article. methods, here defined as neural networks with at least two layers and some nonlinearity [33], for game content gen- 2.2 Novelty of the review eration. We take an inclusive view of games as any games a human would conceivably play, including board games, The differences between the current article and the card games, and any type of video games, such as arcade PCGML survey [129] is that (i) our article focuses on DL- games, role-playing games, first-person shooters, puzzle based methods, defined at the beginning of Sect. 2 games, and many others. Several other surveys and over- (although other techniques will be mentioned for contrast); views of PCG in games already exist. Here, we delineate (ii) our article surveys more types of game content, such as the scope of our article by comparing it to existing books narrative text and graphical textures; (iii) we also discuss 123 Neural Computing and Applications (2021) 33:19–37 21 applications of deep learning to support PCG, such as for of random pixels just because the wheels on the car don’t content quality prediction; and (iv) our survey is written match. In contrast, when generating a game level, if the more than three years after the PCGML survey was first final door has no matching key the level is unplayable; the submitted and two years after it was published, during level’s utility as content is not just slightly diminished, but which time an avalanche of new work in the field has essentially zero (unless manually repaired). Making a appeared. neural network learn to produce only functional content is During the two years after the publication of [129], PCG often a tall task, and is one of the core challenges of using via deep learning has been growing quickly and a signifi- deep learning for PCG. Not all types of game content have cant number of papers and articles have been published. the same extent of functional constraints however, and The trend was mainly set by latent variable evolution [5]in some offer affordances that may make content generation 2018.