Creating Augmented and Virtual Reality Applications: Current Practices, Challenges, and Opportunities Narges Ashtari1, Andrea Bunt2, Joanna McGrenere3, Michael Nebeling4, Parmit K. Chilana1 1Computing Science 2Computer Science 3Computer Science 4School of Information Simon Fraser University University of Manitoba University of British Columbia University of Michigan Burnaby, BC, Canada Winnipeg, MB, Canada Vancouver, BC, Canada Ann Arbor, MI, USA {nashtari, pchilana}@sfu.ca [email protected] [email protected] [email protected] ABSTRACT While research on novel AR/VR tools is growing within the Augmented Reality (AR) and Virtual Reality (VR) devices human-computer interaction (HCI) community, we lack are becoming easier to access and use, but the barrier to entry insights into how AR/VR creators use today’s state-of-the- for creating AR/VR applications remains high. Although the art authoring tools and the types of challenges that they face. recent spike in HCI research on novel AR/VR tools is Findings from preliminary surveys, interviews, and promising, we lack insights into how AR/VR creators use workshops with AR/VR creators mostly shed light on today’s state-of-the-art authoring tools as well as the types of isolated aspects of the proposed AR/VR authoring tools challenges that they face. We interviewed 21 AR/VR [1,15]. We especially lack an understanding of motivations, creators, which we grouped into hobbyists, domain experts, needs, and barriers of the growing population of AR/VR and professional designers. Despite having a variety of creators who have little to no technical training in the motivations and skillsets, they described similar challenges relevant technologies and programming frameworks. in designing and building AR/VR applications. We synthesize 8 key barriers that AR/VR creators face In this paper, we describe a semi-structured interview study nowadays, starting from prototyping the initial experiences with 21 AR/VR creators that investigates how they approach to dealing with “the many unknowns” during design and implementation in AR/VR. To better understand implementation, to facing difficulties in testing the diversity in AR/VR creation practices, we recruited applications. Based on our analysis, we discuss the participants with a wide range of motivations, backgrounds, importance of considering end-user developers as a growing skill levels, and experiences. For example, we included population of AR/VR creators, how we can build learning hobbyists creating their own games, researchers trying to opportunities into AR/VR tools, and the need for building apply AR/VR for domain-specific problems, and AR/VR toolchains that integrate debugging and testing. professional designers working on enterprise-level AR/VR products. These creators worked on a variety of applications, Author Keywords such as in-home design, sports and rehabilitation, medicine, AR/VR authoring; augmented reality; virtual reality; end- cultural studies, and games, among many others. user development; AR/VR development; AR/VR design; Our overall findings indicate that, although AR/VR creators CCS Concepts vastly differ in their motivations and skillsets, they experience • Human-centered computing~Human computer similar challenges in designing and building AR/VR interaction (HCI); applications. We synthesized 8 key barriers described by INTRODUCTION AR/VR creators: from understanding the initial landscape of The increased availability of AR/VR-equipped devices is authoring tools, to designing and prototyping AR/VR opening the door to exploring a wide range of consumer- experiences, to implementation, debugging, and testing. oriented applications and opportunities beyond gaming and While almost everyone mentioned the importance of creating entertainment [53,54]. Although interest in creating AR/VR a good user experience, most of our participants who were applications is rapidly growing, creators are often dealing not in professional design teams were not aware of any with a number of technical hurdles with AR/VR authoring formal UX design approaches and proceeded to development environments [55] and struggle in designing compelling user environments. In contrast, most participants who were experiences [1,53]. designers tried using UX principles to prototype and test Permission to make digital or hard copies of all or part of this work for personal or AR/VR user experiences, but constantly struggled with classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full available design methods and tools, and felt that most citation on the first page. Copyrights for components of this work owned by others required “too much coding.” Compared to web and mobile than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior application development, we found that the barriers that specific permission and/or a fee. Request permissions from [email protected]. AR/VR creators faced during implementation, debugging, CHI '20, April 25–30, 2020, Honolulu, HI, USA © 2020 Copyright is held by the owner/author(s). Publication rights licensed to and testing were more acute as creators had to deal with “too ACM. ACM 978-1-4503-6708-0/20/04…$15.00 https://doi.org/10.1145/3313831.3376722 many unknowns” and keep up with the rapidly evolving studies or evaluations with AR/VR creators. However, as is AR/VR hardware and software. common with system evaluations [25], most have a limited Our paper makes the following contributions: scope to demonstrate the benefits of the tool. 1) Producing empirical insights that portray the unique The closest work that aligns with our project is “DART, ten motivations and backgrounds of AR/VR creators years later” [15], a study on AR prototyping that investigated using today’s authoring environments. the work of 8 AR creators with design backgrounds using the 2) Providing a synthesis of 8 key barriers that were DART timeline-based visual AR authoring tool that reduces common across all AR/VR creators: from the need for programming [31]. Similar to results reported in understanding the landscape of AR/VR that study, we found that even creators with technical development, to exploring designs and prototypes, backgrounds had difficulty in debugging and wanted to to implementation and testing challenges. minimize the extra effort required to integrate more tools into 3) Identifying opportunities for future HCI research to their existing workflows. support end-user developers as a growing population Another relevant study focused on extending AR of AR/VR creators, to build learning opportunities applications for cross-device collaboration [46]. The authors into AR/VR tools, and to design AR/VR authoring reported findings from a survey with 30 designers, toolchains that integrate debugging and testing. developers, and end-users. This study asked participants to RELATED WORK consider two scenarios of an available AR application and to To situate our study findings in the broader HCI literature, discuss challenges they would expect to face when we draw upon insights from existing research on developing implementing them. Findings of this study provided insights AR/VR authoring tools, observations of AR/VR creators in on technical challenges of AR development, such as cross- different contexts, and how we can look at emerging AR/VR device communication, calibration, environmental mapping, creators as end-user developers. obtrusiveness of authoring platforms, gesture definition, and tracking in collaborative settings. AR/VR Authoring Tools in Research and Practice Previous research on AR/VR authoring has addressed the Our findings complement these prior works by adding specific development needs of creators across a wide insights from a more diverse pool of both AR and VR spectrum. For example, early tools such as ARToolKit [21] creators: hobbyists, domain experts, and professional and Studierstube [45] provide a low-level framework designers. We also provide new empirical insights into the requiring the creator to have a strong programming AR/VR authoring process by considering the full spectrum background. Commercial AR/VR creation tools, such as of user research, prototyping, development, and testing, as Unity, Unreal, Visor, and A-Frame, are starting to provide part of AR/VR creation. In addition, we illustrate differences new capabilities that make high-fidelity prototyping easier between AR vs. VR during different stages of authoring. but still require some coding. Overall, the landscape of AR/VR Creators as End-User Developers AR/VR authoring tools appears to be fragmented [39] and Since our study includes participants with a wide range of creators are required to learn multiple different tools. technical expertise and motivations, we look toward the Some research tools have explored how early stages of literature on end-user development to situate our findings. AR/VR prototyping can be facilitated without the need for End-user development has been a core topic in HCI for programming (e.g., DART [31], ProtoAR [38], 360proto decades [5,24]. End-user programmers are non-professional [37], ARtalet [20], iaTAR [26,27], and Amire [51]). Although developers with a variety of backgrounds (often other than these approaches have been instrumental in opening a new computer science) who learn to write code as complementary space of