The Thirty-Fifth AAAI Conference on Artificial Intelligence (AAAI-21) Applied Machine Learning for Games: A Graduate School Course Yilei Zeng, Aayush Shah, Jameson Thai, Michael Zyda University of Southern California fyilei.zeng, aayushsh, jamesont, [email protected] Abstract research-oriented, industry-oriented or patent-oriented di- rections. The projects’ difficulties are also dynamically ad- The game industry is moving into an era where old-style justable towards different students’ learning curve or prior game engines are being replaced by re-engineered systems experiences in machine learning. In this class, we intend with embedded machine learning technologies for the opera- tion, analysis and understanding of game play. In this paper, to encourage further research into different gaming areas we describe our machine learning course designed for gradu- by requiring students to work on a semester-long research ate students interested in applying recent advances of deep project in groups of up to 8. Students work on incorporat- learning and reinforcement learning towards gaming. This ing deep learning and reinforcement learning techniques in course serves as a bridge to foster interdisciplinary collab- different aspects of game-play creation, simulation, or spec- oration among graduate schools and does not require prior tating. These projects are completely driven by the student experience designing or building games. Graduate students along any direction they wish to explore. Giving the students enrolled in this course apply different fields of machine learn- an intrinsic motivation to engage on their favorite ideas will ing techniques such as computer vision, natural language not only make teaching more time efficient but also bestow processing, computer graphics, human computer interaction, a long-term meaning to the course project which will open robotics and data analysis to solve open challenges in gam- ing. Student projects cover use-cases such as training AI-bots doors for them. By having a semester-long project, students in gaming benchmark environments and competitions, under- can dive deep into different algorithms. They also receive standing human decision patterns in gaming, and creating in- hands-on experience incorporating various machine learn- telligent non-playable characters or environments to foster ing algorithms for their use case. engaging gameplay. Projects demos can help students open Writing, presenting and teamwork proficiency is a critical doors for an industry career, aim for publications, or lay the component of a higher education, and this courses involve foundations of a future product. Our students gained hands- writing assignments, extensive team collaboration and oral on experience in applying state of the art machine learning presentation to a public audience. Student performance on techniques to solve real-life problems in gaming. formal writing assignments, project actualization and pub- lic presentation provides benchmarks for examining student Introduction progress, both within and across semesters. Applied machine learning in games is now a vividly expand- This experience report describes a three semester-long ef- ing research field that provides a platform for novel vision, fort in an applied machine learning course with advanced language, robotics, and online social interaction algorithms. research orientations in gaming. This course has withstood Exposure to state-of-the-art research literature is an integral the test through in-person, hybrid learning, and completely part of the course plan, in part because research community online modalities separately. We contribute a new course de- is moving forward at an ever-increasing speed and under- sign inline with the most recent advancements in the gam- standing several backbone papers will clarify the research ing research community. This course attracts and caters to question and enhance an understanding of the iterations and mutual interests across engineering graduate programs. Of improvements made. Moreover, an emphasis on the state-of- the 292 students enrolled in this course over 3 semesters; the-art research methods fosters an appreciation of research 1.3% major in Environmental Engineering, Physics , Chem- design and methodology, and more generally, of the impor- istry or Computer Networks, 1.3% are Software Engineer- tance of critical evaluation. Therefore, new ideas can be gen- ing or High-Performance Computing, 2% are Game Devel- erated based on critical thinking. opment, 3.2% are Electrical Engineering, 4% are Intelligent As this course does not require prerequisites on machine Robotics, 7% are Computer Engineering, 9% are Applied learning, we encourage learning by doing. A self-proposed Data Science, 9.2% are Data Science and the majority of project will enable the students to tailor themselves into students, 63%, are majored in General Computer Science. Students are expected to gain both creative and fun hands- Copyright © 2021, Association for the Advancement of Artificial on experience through a semester-long applied deep learn- Intelligence (www.aaai.org). All rights reserved. ing and reinforcement learning project. This course demon- 15695 strates the feasibility of teaching and conducting state-of- Research (Tian et al. 2017), which provides three environ- the-art applied machine learning research within mixed fo- ments, i.e., MiniRTS, Capture the Flag, and Tower Defense. cused engineering graduate students. This course also shows PySC2 is DeepMind’s Python component of the StarCraft the capability to help students open doors for an industry ca- II Learning Environment (SC2LE) (Vinyals et al. 2017). reer, aim for publications, or lay the foundations of a future STARDATA (Lin et al. 2017), a StarCraft: Brood War re- product. play dataset, is published with the StarCraft II API. Mi- crosoft announced Project Malmo (Johnson et al. 2016), Background which provides an open-source platform built on top of Minecraft. MineRL Environments built on Malmo are re- Aiming for approaching Artificial General Intelligence leased for NeurIPS competitions and MineRL imitation (AGI), video games such as Atari, Doom, Minecraft, Dota learning datasets (Johnson et al. 2016) with over 60 million 1 2 , StarCraft, and driving games have been used exten- frames of recorded human player data are published to facil- sively to test the deep learning and reinforcement learn- itate research. The Unity Machine Learning Agents Toolkit ing methods’ performance and generalizability. Following (ML-Agents) (Juliani et al. 2018) is an open-source project Google’s Alpha Go (Silver et al. 2016), researchers have that enables games and simulations created by individuals made steady progress in improving AI’s game playing capa- to serve as environments for training intelligent agents. As bilities. Besides creating intelligent Non-player characters an active research field, new environments and tasks emerge (NPC), game testing and level generation have also seen daily. We leave the constant learning to students as they advancement with deep learning for the gaming industry. progress through their projects. Moreover, Machine learning can unleash the power of data generated from millions of players worldwide. Gaming pro- Computer Vision & Natural Language Processing vides numerous behavioral data for online user profiling, ad- Learning to play from pixels have become a widely accepted vertisement recommendation, modeling social interactions, approach for traning AI agents after DeepMinds paper of and understanding decision-making strategies. Apart from playing Atari with Deep Reinforcement Learning (Mnih in-game trajectories, Esports and streaming open new re- et al. 2013) using raw pixels as input. Vision-based user in- search opportunities for multi-modal machine learning that puts augmented automatic face, and gesture recognition has combines textual, audio natural language processing, com- enabled the fitness game genre to boost. With the pandemic puter vision with social media. Gaming simulated interactive in 2020, virtual reality devices and fitness gaming has of- environments can extend beyond gaming and adopt practical fered a safe and entertaining indoor option. With the boom- values for robotics, health, and broader social good. ing of streaming platforms, elaborate walk-through, strate- We cover all the following topics in our course. The cited gies, and sentiments shared via videos provided a wealth of work also serve as supplementary reading materials. And data for applied computer vision tasks such as motion analy- these topics will be exemplified in the Student Projects sec- sis and activity recognition. Leveraging the information pro- tion. vided in the YouTube videos, researchers can guide deep reinforcement learning explorations for games with sparse Benchmark Environments and Competitions rewards (Aytar et al. 2018). For academic and individual researchers, the IEEE Con- Understanding players’ textual interactions, both in-game ference on Games(COG), AAAI Conference on Artificial and on social networks, is crucial for gaming companies Intelligence and Interactive Digital Entertainment(AIIDE), to prevent toxicity and increase inclusion. In gaming, lan- Conference on the Foundations of Digital Games (FDG), guage generation techniques are leveraged to generate narra- and Conference on Neural Information Processing Systems tives for interactive and creative storytelling. Text adventure (NeurIPS) host a series of annual competitions featuring cre- games