Scaling Multi-user Virtual and Augmented Reality by Sebastian Herscher A DISSERTATION SUBMITTED IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF DOCTOR OF PHILOSOPHY DEPARTMENT OF COMPUTER SCIENCE NEW YORK UNIVERSITY JANUARY, 2020 Ken Perlin © SEBASTIAN HERSCHER ALL RIGHTS RESERVED, 2020 Dedication TO MY PUZZLE PIECE. iii Acknowledgments I WOULD LIKE TO THANK MY ADVISOR,PROF.KEN PERLIN, FOR HIS PHENOME- NAL AND UNWAVERING SUPPORT THROUGHOUT THE PHD PROCESS. I WOULD LIKE TO THANK THE MEMBERS AND ALUMNI OF THE NYU FUTURE REALITY LAB, WHO ARE TRUE PIONEERS IN THIS FANTASTIC DIGITAL AGE. iv Abstract The Virtual and Augmented Reality (XR) ecosystems have been gaining substantial momentum and traction within the gaming, entertainment, enterprise, and training markets in the past half-decade, but have been hampered by limitations in concur- rent user count, throughput, and accessibility to mass audiences. Although a litany of XR devices have been made available for public purchase, most XR experiences have been developed for either a single user or a small set of users at a time. Few systems or experiments in co-located XR environments have expanded past a small set of users, leaving the paradigm of being part of a larger virtual audience relatively untested. This thesis presents a set of components, systems, and experiments that as- sist in the creation and scaling of multi-user virtual and augmented reality experiences, and outlines the strengths of techniques found in traditional co-located media for the design space of scaled co-located XR. v Contents DEDICATION iii ACKNOWLEDGMENTS iv ABSTRACT v LISTING OF FIGURES viii LIST OF TABLES xi 1 INTRODUCTION 1 1.1 Collective Audience Perception Spaces.................. 2 1.2 Research Questions............................ 3 1.3 Research Contributions.......................... 4 2 CAVRN: SCALING UP VIRTUAL REALITY 6 2.1 Related Work ............................... 8 vi 2.2 Design Constraints ............................ 10 2.3 CAVRN.................................. 11 2.4 CAVE ................................... 18 2.5 Evaluation of CAVRN and CAVE ..................... 20 2.6 Results................................... 21 2.7 Discussion................................. 31 3 MARY:EXPERIMENTING ACROSS PLATFORMS 35 3.1 Related Work ............................... 37 3.2 Design Constraints ............................ 39 3.3 Expanding CAVRN............................ 40 3.4 Mary and the Monster ........................... 44 3.5 View-Enablement Tools.......................... 49 3.6 Initial Evaluation of MARY ........................ 54 3.7 Results................................... 56 3.8 Discussion................................. 62 CONCLUSION 65 REFERENCES 68 vii Listing of figures 2.1 The software architecture relationship between the server and all clients. (1) All client applications report their position and orientation to the server. (2) The server sends experience and client state data to all client applica- tions. (3) The server sends an event signal (e.g. seat assignment) to the client. 11 2.2 An example layout of the server. Note that the status of each client is vis- ible, and that the server can assign a seat to a client, and control the play- back of the experience. .......................... 13 2.3 The representation of the audience in CAVE. Each audience avatar was driven by the position and orientation of the corresponding headset. Note that all audience members can see the position and orientation of the heads of all other audience members, which are calculated using the data received from all other users’ VR headsets. ....................... 16 viii 2.4 An example of how a piece of content in CAVE, the mammoth, is rendered in comparison to the virtual audience. Note that the front row of the au- dience avatars is rendered in between the virtual content and the back row of the audience avatars. This is vital in evoking a sense of audience pres- ence in the virtual experience........................ 19 2.5 A rendering of the physical layout used for CAVE. The audience was split into five rows: front left, back left, front right, middle right, and back right. 21 2.6 An audience watching CAVE at the technical conference. 24 3.1 An example of mapping an Oculus Touch controller to an audience avatar. The images show that the hand mapping occurs using button states, instead of physical hand poses........................... 41 3.2 An overhead view of the VR audience of MARY.............. 44 3.3 An example avatar of MARY........................ 46 3.4 A view of the VR version of MARY through the Oculus Quest. 47 3.5 A view of the AR version of MARY through the Magic Leap One. 48 3.6 The telescopic interface tool implemented in MARY. The rendered style of the opera glasses was made to match that of the avatars. The third im- age shows a vignette rendered while the opera glasses are in use, in order to alleviate motion sickness. ....................... 50 3.7 The magnifier tool implemented in the AR version of MARY. The handle of the magnifier was removed, as the lens was positioned directly above the hand controller. ............................ 52 3.8 The viewing window tool implemented in the VR version of MARY. 53 ix 3.9 An audience watching MARY at SIGGRAPH 2019 in Los Angeles. 54 x List of Tables 2.1 Summary of interview participants..................... 22 2.2 Aggregate survey responses for reported professions. .......... 23 2.3 Aggregate survey responses for prior experience with VR devices. 23 2.4 First part of the aggregate survey responses for what users believe can be replicated in traditional forms of media, and what elements of CAVE stood out. Note that the Scale category refers to the scale of the content in rela- tion to the scale of the audience in the experience............. 27 2.5 Second part of aggregate survey responses................. 27 3.1 Summary of interview participants..................... 55 xi 1 Introduction Virtual and augmented reality (VR and AR, often combined as XR) have been around since the beginnings of 3D rendering, with Sutherland et al. developing the fabled Sword of Damocles VR/AR headset1 a full four years before Catmull et al. released their seminal work A Computer Animated Hand 2. A digital world either sitting on top of or replacing our physical world has been a core trope of science fiction for 1 more than half of a century, with a plethora of research being conducted to make said fiction a reality. Historically, these mediums have been locked behind both monetary and expertise barriers, with limited functionality due to a lack of processing power for realtime rendering. However, this most recent iteration of virtual and augmented reality hardwares have piggybacked on vast increases in compute power, and have been successfully delivered to the world at both consumer and enterprise price points. The wide availability of affordable XR hardware has occurred in tandem with an explosion of interest, investment, and development of awesome immersive experi- ences and games. These have taken the form of Academy award winning immersive experiences30, phenomenally successful at home VR games52, and a new industry of "hyper reality" out of home location based experiences for small groups25. Compa- nies have begun to empower their employees no longer develop 3D content using 2D interfaces, instead building and creating with the full fidelity of 3D input50. It is clear that our future includes both the use XR for both out of home and in home entertain- ment, as well as for everyday communication and collaboration in both professional and personal use. 1.1C OLLECTIVE AUDIENCE PERCEPTION SPACES However, for this future to come to fruition, two fundamental issues with the current wave of XR must be addressed. First, development of real-time immersive 3D content must be unified and standardized, such that when an immersive experience is created, it can be ubiquitously distributed across both the litany of current XR devices as well as those that will be released in the future. Second, the design space of XR must be 2 taken past its current limit of 4-6 concurrent users at a time. The social pattern of congregating together in large, collective groups is consis- tent across all layers of human social practice. In both professional and personal en- vironments, we come together to experience, learn, and perceive en masse. In the immersive space, the ability to join into a large virtual congregation from a remote access point has been successfully built56,35, but their colocated counterparts have only brought together a smattering of people in the same physical space. We must endeavour to build and explore Collective Audience Perception Spaces, or CAPS - physically shared design spaces that include dozens if not hundreds of users that can see, hear, and share an immersive environment as a collective, through both virtual and augmented reality. Each audience member must have their own viewpoint and representation, while being synchronized with the aggregate. These environments mirror their non-immersive counterparts of lecture halls, cinema, and theater, with the added power and capability of spatial computing and immersive design. Whether through using virtual reality to create a shared virtual space, or augmented reality to add to the physical world, CAPS must enable an audience to congregate in a syn- chronous immersive experience, without need for participation outside of contributing to the collective knowledge and focus of the whole. 1.2R ESEARCH QUESTIONS Many research questions arise out of the design space of Collective Audience Percep- tion Spaces. First and foremost, the presence of a representation of an audience in an immersive environment changes both the design and implementation of an experience, 3 as well as the reception and enjoyment of it by each viewer. Furthermore, there is a tension between the amount of agency that can be provided to each individual audi- ence member, and the synchronicity of the experience to the collective. Limitations on both the design of content and systems to support managing the large scale shared system must also be investigated.