FACULDADE DE ENGENHARIA DA UNIVERSIDADE DO PORTO Framework for Developing Interactive 360-Degree Video Adventure Games Francisco Pinho Mestrado Integrado em Engenharia Informática e Computação Supervisor: Rui Rodrigues Co-Supervisor: Rui Nóbrega July 20, 2019 Framework for Developing Interactive 360-Degree Video Adventure Games Francisco Pinho Mestrado Integrado em Engenharia Informática e Computação July 20, 2019 Abstract Making game environments, be it in 2D or 3D is an extremely laborious and skill intensive task. 360o content is a visual medium known for its increased spatial immersion and potential enhance- ment of the users’ emotional response to content [EEW18]. The possibility of using interactive 360o video or images as the game environments will allow talent to invest their creativity into other crucial aspects of game design such as narrative, sound and game mechanics. Adventure games belong in a diverse genre comprising many different types of experiences and sub-genres, from text-based adventures to puzzle-oriented point-and-click games, a cinematic experience with player choices or even a mixture of crime investigation with courtroom drama. In essence, games with simpler means of user input and a bigger focus on interactive narratives and storytelling - a very relevant field at the cutting edge of virtual reality research [RTGG17]. Further- more these games feature entirely non reflex based mechanics making the genre of "Adventure" the most fitting for integration with 360-degree visual media. This dissertation presents a framework that will allow the streamlining of the creative process of these experiences by giving creators the tools to make a fully-fledged virtual reality adventure game with 360 visual media as an interactive setting. The framework was implemented and usability tests were conducted to validate the viability of 360o video based adventure games as an entertainment and storytelling medium. Results displayed the potential of this format of game as a fully fledged experience that was appreciated for the adapted mechanics from the family of genres derived from adventure games, ease of control and storytelling possibilities. i ii Resumo Criar ambientes de jogo seja em 2D ou em 3D é bastante laborioso e requer um nível alto de aptidão. Entretenha-se a possibilidade de usar video ou imagens 360o como uma experiência interactiva dado que são um meio visual conhecido pela imersão espacial oferecida e um aumento da resposta emocional do utilizador ao conteúdo [EEW18]. Utilizar videos ou imagens 360o possibilitará então que o talento dos criadores de jogos possa ser investido em outros aspetos cruciais do desenho do jogo como a narrativa, som e mecânicas. Jogos de aventura fazem parte de um género heterogéneo que contém diferentes tipos de ex- periências e sub-géneros, desde aventuras em texto a jogos orientados a puzzle do estilo "point- and-click", uma experiência cinemática com escolhas de jogador ou até mesmo uma mistura de investigação de crime com drama de tribunal. Ou seja, jogos com interação mais simples e mais focada em narrativa. Sendo esta um campo de investigação bastante relevante no contexto de re- alidade virtual [RTGG17]. Para além disso as mecânicas de jogo não são baseadas em reflexos o que torna o género de "Aventura" um dos mais apropriados para integração com media visual em 360o. Esta dissertação apresenta uma framework que possibilitará a simplificação do processo cria- tivo deste tipo de experiências, proporcionando aos criadores as ferramentas necessárias para de- senvolver jogos de aventura completos, utilizando videos 360o como o ambiente de jogo e jogáveis em realidade virtual. A framework foi devidamente implementada e testes de usabilidade foram conduzidos para validar a viabilidade de jogos de aventura baseados em videos 360o tanto num foco de jogabilidade como de narrativa. Os resultados mostraram o potencial deste formato de jogo como uma experiência de jogo completa que foi apreciada pelos utilizadores tanto pelas mecânicas adaptadas da família de géneros derivados de jogos de aventura como pela usabilidade dos controlos do jogo e as possibilidades de narrativa. iii iv Acknowledgements I would like to thank my supervisors, Rui Rodrigues and Rui Nóbrega, as well as the people of GIG for their continued advice and for giving me the opportunity to see this project through till the end by providing all the support and resources I needed. I want to express the utmost gratitude to my significant other, Maria de Abreu, my family and friends for their continued support and for the simple fact that they exist. Last but not least a big thank you to António Baía Reis for essentially volunteering as the actor in the proof of concept game on such short notice without having time to practice and for still doing a great job regardless. Francisco Pinho v vi Contents 1 Introduction1 1.1 Motivation . .2 1.2 Research Questions . .2 1.3 Research Goals . .3 1.4 Contributions . .3 1.5 Document Structure . .3 2 State of the Art5 2.1 Adventure Games . .5 2.1.1 Design Challenges and Variations . 11 2.1.2 Spin-off Genres . 15 2.1.3 Core Game Mechanics . 16 2.1.4 Narrative and Storytelling . 20 2.2 Virtual Reality and 360-Degree Video . 22 2.2.1 Presence and Immersion . 22 2.2.2 Focusing Viewer Attention . 24 2.2.3 Interactive 360o Video . 24 2.3 Summary . 26 3 A Framework for Interactive 360 Video Adventure Games 29 3.1 Conceptualization . 29 3.1.1 System Requirements . 29 3.1.2 Target Creation Workflow and Output . 30 3.1.3 Virtual Objects and Interactions . 31 3.2 Development . 36 3.2.1 Development Tools and Technologies . 36 3.2.2 System Overview and Architecture . 40 3.2.3 Scene Elements . 43 3.2.4 Framework Components . 47 3.2.5 Descriptive Language - Real Adventure JSON . 56 3.2.6 Development Methodology . 63 3.3 Summary . 64 4 Evaluation 67 4.1 Proof of Concept Creation - Animus Liber . 67 4.2 Proof of Concept Description . 68 4.3 Evaluation Protocol and Session Structure . 69 4.3.1 Usability Questionnaires . 69 vii CONTENTS 4.3.2 Observations and Additional Data . 70 4.4 User Experiments . 70 4.4.1 User Sample Overview . 70 4.4.2 Results . 71 4.5 Analysis and Discussion . 77 4.5.1 General Control Scheme . 77 4.5.2 Inventory System . 78 4.5.3 Distinguishing Objects . 79 4.5.4 Code Puzzle Interface . 80 4.5.5 Lost During the Experience . 80 4.5.6 Engagement and Replayability . 81 4.5.7 Final Remarks . 82 5 Conclusions 83 5.1 Future Work . 85 References 87 A Cited Games 91 B Animus Liber Storyline 93 C Usability Questionnaires 101 viii List of Figures 2.1 "Adventure" . .5 2.2 "Mystery House" . .6 2.3 "The Secret of Monkey Island" - Menu commands . .7 2.4 "Myst" - First Person Perspective . .8 2.5 "The Walking Dead" - Player Choice . .9 2.6 Branching Narrative - "Detroit: Become Human" and "9 Hours 9 Persons 9 Doors"9 2.7 "Danganronpa V3" - Point-and-click Screen . 10 2.8 "Ace Attorney" - Trial . 11 2.9 Common Mechanics in Adventure Games . 17 2.10 Reality-Virtuality Continuum . 22 2.11 Gamified 360o video . 25 2.12 Google Arts and Culture streetview of Russian Museum . 26 3.1 Mockup . 30 3.2 Game creation workflow . 31 3.3 Common VR controllers . 32 3.4 Real Adventure - Code Puzzle Interaction . 35 3.5 Real Adventure - Combine Objects Interaction . 35 3.6 Real Adventure - Use Object Interaction . 36 3.7 Virtual Object JSON Representation Example . 38 3.8 Real Adventure - Pickable Component Data Schema . 38 3.9 Real Adventure Project Anatomy . 40 3.10 Real Adventure - Game Part Mock . 41 3.11 Real Adventure - General Architecture . 42 3.12 Real Adventure - Scene Graph . 44 3.13 Real Adventure - Examinable Object Hover Icon . 46 3.14 Real Adventure - Dialogue Tree . 46 3.15 Real Adventure - Text Box Example . 53 3.16 Real Adventure - Inventory . 53 3.17 Real Adventure - JSON scene template . 56 3.18 Real Adventure - Cutscene definition with flag-based transition . 57 3.19 Real Adventure - Recorded video element made virtual object . 58 3.20 Real Adventure - Combinations in JSON . 59 3.21 Real Adventure - Flag-based transition example . 60 3.22 Real Adventure - Simple dialogue example . 62 3.23 Real Adventure - Dialogue tree options example . 63 3.24 Real Adventure - Complex dialogue example with all the features . 64 ix LIST OF FIGURES 4.1 Proof of concept - Animus Liber Story Structure . 68 4.2 Results - Feeling lost during the experience . 75 4.3 Results - Would you play/not play it again? . 76 4.4 Results - Would you buy a 360 video based adventure game? . 77 x List of Tables 3.1 Virtual Object Types . 32 3.3 Interactions . 33 3.4 System Components . 47 3.5 Interaction Components . 48 3.6 Media Components . ..