The Trouble with Augmented Reality/Virtual Reality Authoring Tools
Total Page:16
File Type:pdf, Size:1020Kb
The Trouble with Augmented Reality/Virtual Reality Authoring Tools Michael Nebeling* Maximilian Speicher† University of Michigan School of Information https://mi2lab.com ABSTRACT There are many technical and design challenges in creating new, usable and useful AR/VR applications. In particular, non-technical Class 5 designers and end-users are facing a lack of tools to quickly and eas- Class 4 Unity3D, Unreal Engine, ily prototype and test new AR/VR user experiences. We review and Teddy, A-Frame, ... classify existing AR/VR authoring tools and characterize three pri- SketchUp, mary issues with these tools based on our review and a case study. To 3ds Max, ... address the issues, we discuss two new tools we designed with sup- port for rapid prototyping of new AR/VR content and gesture-based Class 3 interactions geared towards designers without technical knowledge Class 2 ARToolKit, in gesture recognition, 3D modeling, and programming. Class 1 DART, Proto.io, HoloBuilder, InVision, HoloBuilder, ... AR Studio, ... Keywords: augmented reality, virtual reality, authoring, design, Sketch, Adobe ? rapid prototyping, 3D modeling, gestures, Wizard of Oz. XD, ... Skill & resources required Index Terms: Human-centered computing—Interaction paradigms—Mixed / augmented reality Level of fidelity in AR/VR 1 INTRODUCTION Figure 1: AR/VR authoring tools classified along (x) the possible level Today’s AR/VR applications are mostly built by tech-savvy devel- of fidelity in AR/VR, and (y) skill & resources (e.g., 3D models, 360 opers. Significant technical skill and programming experience is photos, hardware) required. An “optimal” class of tools would be on required to create AR/VR experiences. While tools like Unity and the far right and remain low. Tools that combine the principles of our A-Frame have in many ways become the “standard for AR/VR,” ProtoAR [20] and GestureWiz [30] could be examples for this class. they still provide a high threshold for non-technical designers and of future iterations. A significant part of the process is prototyping al- are inaccessible to less experienced end-users. There is a new class ternative designs, typically with several iterations on paper to shape of tools for creating basic AR/VR experiences, allowing users to the concept [1,22,26], then moving to digital tools to increase the choose from pre-made 3D models and existing scripts for anima- level of fidelity and generate new design insights [16,17]. For AR, tion and interactivity. However, these tools usually cover only a however, paper prototyping was recently found too limiting [7,20], very limited spectrum of the AR/VR design space and still require motivating the development of new techniques that support physical programming for more advanced application logic. prototyping but integrate well with digital tools. In this position paper, we classify existing authoring tools relevant There is an extensive number of digital tools available for creating to AR/VR, identify five classes of tools (Fig. 1), and characterize the interactive prototypes, but across all tools, support for AR/VR con- main issues we see with how the tool landscape has been evolving. tent and interactive behavior is limited and scattered. The landscape Both authors have a track record of research on interactive technolo- of existing digital tools can be grouped into five classes of tools. gies with a more recent focus on AR/VR [20,28–30]. For example, they created ProtoAR [20], a tool designed with the vision of making AR/VR prototyping as easy and versatile as paper prototyping, and 2.1 First Class: Basic Mobile Screens & Interactions GestureWiz [30], a Wizard of Oz gesture prototyping environment. The first class consists of tools for mobile and web designers such The second author also contributed to the design and development of as InVision, Sketch, and Adobe XD. Many tools in this class have HoloBuilder [31] from 2015 to 2017. When he joined the company, a concept of multiple screens, support defining active regions in the original idea was to create a “PowerPoint for AR,” enabling these screens that listen to mouse and touch input to trigger menus users without specific design and development skills to create AR or transition to other screens, and enable mobile app previews on experiences. For the future, we envision tools as simple yet powerful smartphones and tablets. While this is sufficient for traditional as PowerPoint or Keynote leveling the playing field for AR/VR. mobile interfaces, tools in this class lack support for AR/VR content and interaction as they do not interface with AR/VR technologies. 2 FIVE CLASSES OF AR/VR AUTHORING TOOLS Interaction design [3, 19, 23] has evolved into a comprehensive, iter- 2.2 Second Class: Basic AR/VR Scenes & Interactions ative process with four main activities: establish user needs, develop alternative designs, build interactive prototypes, and evaluate proto- The second class consists of AR/VR counterparts of tools in the types. Any remaining difficult or new critical areas will be subjects first class supporting basic forms of AR/VR content and interactive behavior. An early example in research is DART [18], extending *e-mail: [email protected] Macromedia Director with scripts for animating 3D avatars useful for †e-mail: [email protected] prototyping storytelling AR experiences. Recent examples, Proto.io, Halo, and HoloBuilder, aim to be InVision-like tools for AR/VR. Users can upload 360-degree photos to create immersive scenes and define basic interactive behaviors via image maps with anchors that trigger menus or load other scenes. While VR previews are supported with Google Cardboard, AR spatial trackers required for 3.2 Design Processes are Unique Patchworks composite views of virtual content and real-world objects are not. Second, every new AR application that is being built essentially requires building a unique tool chain as well, where selection of 2.3 Third Class: AR/VR Focused Interactions tools highly depends on technical requirements of the next design The third class consists of tools focused on AR camera-based in- stage. Developers and engineers can work with tools at the high teractions. Early examples from research include ARToolKit [11], end of the tool chain (choosing tools like Unity) because they have Tiles [21], Studierstube [24] and ComposAR [25]. HoloBuilder as the training and experience. Yet, these are usually out of reach for well falls into this category since it features marker-based AR func- non-technical designers. Even worse is that users with less training tionality. Recent solutions like Facebook’s AR Studio or Snapchat’s often end up working with an even larger patchwork of tools, to try Lens Studio allow users to select from libraries of trackers (face, to work around their challenges with more professional tools [12]. hand, plane), 2D/3D objects, video masks, and animations to create interactive, shareable camera effects responding to people and ob- 3.3 Significant Gaps Within & Between Tools jects in their surroundings. There is support for visual programming to add basic animations, logic, and interactivity, but many interface Third, to move an AR/VR application from early design concepts concepts common to tools from the other classes, such as screens, to interactive prototypes, developing alternative designs and design menus, and transitions, are absent. iterations typically require creating different parts and versions of the application with different classes of tools. However, the tool 2.4 Fourth Class: 3D Content chain is generally optimized in the upwards direction (e.g., tools like Halo export to Unity but not vice-versa). This makes design The fourth class consists of 3D content creation tools like Teddy [8] iterations especially tricky and expensive if they require going back and Lift-Off [10] created by researchers, Google’s SketchUp and to a previous tool from another class. Blocks, as well as Autodesk’s 3ds Max and Maya. Tools in this class support the creation of new 3D objects via digital 3D modeling, 4 CASE STUDY animation, and simulation techniques unique to this class of tools. 3D objects can be exported in common 3D file formats supported by As mentioned earlier, the second author contributed to HoloBuilder, other classes of tools except the first. Further, 3D content sharing a PowerPoint-inspired tool situated in the second and third classes platforms like Google’s 3D Warehouse and Poly have significantly above, for two years and gained first-hand experience with designing grown in recent years because of the increased need for 3D content and promoting a platform intended to enable anyone to create AR due to emerging new technologies for 3D printing as well as AR/VR. applications [31]. AR projects in HoloBuilder are based on custom images or point clouds as markers while different states of an app are 2.5 Fifth Class: 3D Games & Applications represented by different “slides”, each of which can feature zero or one markers (Fig. 2). Moreover, users are provided with a selection The fifth and last class consists of tools like Unity, Unreal, and A- of predefined 3D models, such as arrows, cubes, and spheres. The Frame, which are comprehensive game and application development 3D models added to a “slide” can be enhanced with basic animations platforms. Tools in this class come with a visual editor in which and click events that either trigger a transition to a different “slide” scenes with 3D objects can be visually specified and assets such or an animation. During his time at HoloBuilder, the second author as 3D models, textures, and sounds can be imported and managed. made three key observations analogous to the problems above. Although most AR/VR device platforms integrate with Unity and Unreal, there is relatively little support specific to AR/VR in them. A-Frame is a relatively new and increasingly popular AR/VR appli- cation development layer on top of HTML, CSS, and three.js.