Toolkit for Social Experiments in VR by Unnar Kristjánsson Thesis of 60 ECTS credits submitted to the School of Computer Science at Reykjavík University in partial fulfillment of the requirements for the degree of Master of Science (M.Sc.) in Computer Science May 2019 Examining Committee: Dr.Hannes Högni Vilhjálmsson, Supervisor Associate Professor, Reykjavík University, Iceland Dr.David Thue, Examiner Assistant Professor, Reykjavík University, Iceland Claudio Pedica, Examiner Affiliated Researcher, Reykjavík University, Iceland i Copyright Unnar Kristjánsson May 2019 ii Toolkit for Social Experiments in VR Unnar Kristjánsson May 2019 Abstract Researchers tend to develop their own applications when it comes to running social be- haviour studies in Virtual Reality. These applications are often single use and become obsolete between different projects of studies, despite projects often sharing similar goals or needs. This thesis argues that given a set of principles influenced by modular design patterns found in embedded systems architecture, reusable toolkit applications could be constructed that serve as a shared foundation for social experiments in VR. What follows is a partial implementation of those principles, and an example study en- abled by said implementation. Conclusions suggest that despite need for refinements, particularly to the implementation presented which only partially satisfies those prin- ciples, the principles indicate some merit as to their utility for building a reusable software toolkit for social experiments in VR. iii Hugbúnaðartól Fyrir Félagsrannsóknir í Sýndarveruleika Unnar Kristjánsson maí 2019 Útdráttur Rannsakendur á félagssviði eiga það til að þróa sinn eigin hugbúnað til að keyra fé- lagsrannsóknir í sýnderveruleika. Slíkur hugbúnaður er venjulega sérhæfður og einungi nothæfur í einu ákveðnu samhengi, og er því ekki endurnýttur á milli verkefna á sama sviði, þrátt fyrir að verkefni deili oftast markmiðum og þörfum. Þessi grein færir rök fyrir því að gefið íhald í ákveðnar reglur miðaðar að endurnýtsleika, þá sé hægt að smíða hugbúnaðartól með eiginleika sem geta þjónað sem undirstaða fyrir tilraunir í sýndarveruleika. Það sem fylgir er af hluta til útfærsla af þessum reglum, og dæmi um notendarannsókn sem byggir á þeirri útfærslu. Niðurstöður benda til þess að þrátt fyrir þarfir á breytingum, sérstaklega á útfærslunni sem er gefin - sem aðeins af hluta til fylgir reglunum út, þá lofa reglurnar góðu hvað notendagildi fyrir félagstilraunir í sýndarveruleika varðar og útfærslu á hugbúnaði sem þjónar þeim þörfum. iv Toolkit for Social Experiments in VR Unnar Kristjánsson Thesis of 60 ECTS credits submitted to the School of Computer Science at Reykjavík University in partial fulfillment of the requirements for the degree of Master of Science (M.Sc.) in Computer Science May 2019 Student: Unnar Kristjánsson Examining Committee: Dr.Hannes Högni Vilhjálmsson Dr.David Thue Claudio Pedica v The undersigned hereby grants permission to the Reykjavík University Library to reproduce single copies of this Thesis entitled Toolkit for Social Experiments in VR and to lend or sell such copies for private, scholarly or scientific research purposes only. The author reserves all other publication and other rights in association with the copyright in the Thesis, and except as herein before provided, neither the Thesis nor any substantial portion thereof may be printed or otherwise reproduced in any material form whatsoever without the author’s prior written permission. date Unnar Kristjánsson Master of Science vi Contents Contents vii List of Figures x List of Tables xi List of Abbreviations xii 1 Introduction 1 1.1 Motivation . 1 1.2 Social VR Toolkit . 2 1.3 Problem Statement . 2 1.4 Contributions . 3 1.5 Structure . 3 2 Related Work 4 2.1 Game Engines . 4 2.2 User Friendly Abstractions . 7 2.2.1 Visual Programming . 7 2.2.2 Natural Language Understanding . 7 2.2.3 Native Format Conversion . 8 2.3 Social Agent Research Tools . 8 2.3.1 SAIBA . 8 2.3.2 SAIBA Compliance . 9 3 Approach 10 3.1 Design Principles . 10 3.2 Architecture . 11 3.3 Supporting Custom Content . 12 3.4 Agent Behaviour . 13 3.5 VR Considerations . 13 3.5.1 Design Guidelines . 13 3.5.2 User Interaction . 14 3.5.3 Interface . 14 3.6 Use Case . 14 3.6.1 The Scenario . 14 3.6.2 The Tools . 15 3.6.3 The Procedure . 15 4 Implementation 16 vii 4.1 Implementation Principles . 16 4.2 Architecture . 16 4.2.1 Network Layer . 17 4.2.2 The Command Manager . 17 4.2.3 Networked Commands . 18 4.2.4 Command Handlers . 18 4.2.5 Command Terminals . 19 4.2.6 Native Command Terminal . 20 4.2.7 External Web Interface . 20 4.2.8 Command Macros . 21 4.3 Adding Content . 22 4.4 Creating Agents . 22 4.4.1 Humanoids in Unity . 22 4.4.2 Unity’s Animator Component . 22 4.4.3 Skinned Meshes in Unity . 23 4.4.4 Human Animator . 23 4.4.5 Unity’s Timeline and Sequence Editing . 24 4.4.6 Extending Unity’s Timeline . 25 4.4.6.1 Pose Track . 25 4.4.6.2 Bone Track . 26 4.4.6.3 Gaze Track . 26 4.4.6.4 Expression Track . 27 4.4.7 Behaviour Annotator . 28 4.5 VR Features . 29 4.5.1 VR Interface . 29 4.6 Use Case . 30 4.6.1 The Scenario . 30 4.6.2 Content Creation . 30 4.6.3 Visual Aesthetics . 31 4.6.4 Characters . 31 4.6.5 Characters Aesthetics . 32 4.6.6 Character Dialogue . 33 4.6.7 Behaviour Sequences . 33 4.6.8 User Interaction and Interface . 34 4.6.9 Configuration . 34 5 Results 35 5.1 Principles and Architecture Flexibility . 35 5.2 Content Support . 35 5.3 Usability of Agent Tools . 36 5.4 Use Case Reviewed . 36 5.4.1 Preparation . 36 5.4.2 Running the Study . 37 5.4.3 Scenario Specific Commentary . 39 6 Future Work 40 6.1 Content Pipeline . 40 6.2 Dedicated Scripting Language . 40 6.3 Scripting Abstractions . 41 viii 6.4 Speech Generation . 41 6.5 Reactive Animation System . 41 7 Conclusion 43 7.1 Contributions . 43 7.1.1 Guidelines . 43 7.1.2 Command Line Interface . 43 7.1.3 Animation Tools . 43 7.1.4 Content Pipeline . 44 Bibliography 45 ix List of Figures 2.1 Modded Doom . 5 2.2 Game Engine Developer Consoles . 5 2.3 Addons in WoW . 6 2.4 Visual Programming . 7 3.1 Toolkit Usage . 11 3.2 Application Content Pipeline . 12 3.3 Example Study Composition . 15 4.1 Command Based Architecture . 17 4.2 Command Handler Inspector . 19 4.3 Command Terminal Execution . 19 4.4 Native Terminal . 20 4.5 Networked Command Terminal Execution . 20 4.6 Web App - Terminal . 21 4.7 Web App - UI Widgets . 21 4.8 Timeline Animator Control . 23 4.9 Unity Timeline Editor . 24 4.10 Custom Timeline Tracks . 25 4.11 Pose Track . 25 4.12 Bone Track . 26 4.13 Gaze Track . ..