DOM2AFRAME: PUTTING THE WEB BACK IN WEBVR Robin Marx1, Sander Vanhove1, Wouter Vanmontfort2, Peter Quax1, Wim Lamotte1 1UHasselt-tUL-imec, EDM, Hasselt, [email protected] 2Androme, Diepenbeek, Belgium. [email protected] ABSTRACT VR browser embeds web pages as part of a larger 3D world), or as separate VR rendering engine or view in an existing web As the Virtual Reality (VR) market continues to grow, so does the browser (e.g., Google Chrome and Samsung Internet). Sadly, these need for high-quality experiences. In order to unlock the wide pool browsers typically only allow the singular use case UC0 and do not of existing web-based content, common and more specialized web provide web developers with direct control over how their content browsers allow users to visit existing web pages in VR. Additionally, is rendered in VR nor how the user can interact with it. This is the latest version of the WebVR [24] standard facilitates the integra- problematic, as the browser’s typical approach (simply displaying tion of custom 3D/VR content in a web page. Sadly, both options the site on a flat 3D surface (e.g., plane, cylinder)) is unlikely to consciously exclude the use of standard 2D web technology (such produce a satisfactory user experience [16]. as HTML and CSS) in other common use cases, such as creating a To support UC1 and UC2, we might look towards the new highly interactive 2D UI for a 3D/VR game. Consequently, web WebVR standard [24]. WebVR provides an abstraction on top of developers wanting to use WebVR are required to learn an entirely the various VR hardware setups and enables developers to use the new skill set to create VR experiences. existing WebGL [10] rendering capabilities of the HTML5 <can- This work surveys and explores workaround options for ren- vas> element [21] to easily create stereographic 3D experiences. dering 2D HTML/CSS/JavaScript-based content in WebVR. We Various VR input methods are made available through the JS-based find that existing methods are often too slow to allow for a highly Gamepad API [24]. WebVR is currently supported in both Google interactive experience. We introduce DOM2AFrame, a new frame- Chrome and Mozilla Firefox, with Apple’s Safari recently joining work that couples 2D page elements directly to their equivalent 3D the WebVR standardization effort [5]. counterparts (using the A-Frame library [2]) to allow for smooth updating, animation and user interaction at frame rates close to 60 In theory, web developers could thus use WebVR to render 2D FPS on modern hardware. Two case studies validate our approach web content in VR and support all three use cases. In practice and show its potential for rendering 2D web content in VR. however, modern browsers do not grant developers direct access to their rendering pipeline and also do not provide an API to Index Terms— Virtual Reality, WebVR, rasterization, DOM render/rasterize typical web content onto a <canvas> element1. The main argument for not providing such a coupling or API has 1. INTRODUCTION always been security and privacy related, as it could lead to abusers taking and saving “screenshots” of sensitive user data (e.g., an Virtual Reality (VR) is on the rise [11]! The need for premium VR- attacker could load an online banking page in an <iframe>, enabled content has exploded in tandem with the availability to the render it to a texture and send it to their server; while this is certainly general public of VR-capable hardware (e.g., headsets, controllers a solvable problem, the possible solutions are not easy to implement and personal computers). While this content is available in sufficient in the rendering pipeline [20]). measure in the form of stereographic video and interactive game- For better or worse, web developers who want to create VR con- related experiences, the rich ecosystem of (2D) web-based content tent are currently limited to the functionality provided by the 2D and has not yet been fully explored in the VR medium. This is a pity, as 3D rendering contexts of <canvas> and WebGL, which means there are various interesting use cases (UCs) for using standard 2D that most of the more powerful and established web technologies web-based technologies such as HTML, CSS and JavaScript (JS) in such as HTML, CSS and large parts of JS (e.g., input event listeners) VR. For example: are not directly usable in WebVR. This implies that web developers cannot use their established skill set to start creating VR content, • UC0 : Rendering and interacting with a full web page as-is instead requiring them to develop new skills (e.g., knowledge of 3D • UC1 : Rendering a 2D User Interface (UI) or other 2D elements primitives, shaders), which are typically taught to game developers. in a 3D experience (e.g., complex menu in a game, text reader) • UC2 : Rendering (parts of) a web page in a different way when viewed in VR (e.g., integrating 3D objects as part of an e-shop, In this work, we first look at existing options to bypass these auto-selecting stereographic video, horizontal instead of vertical limitations and how to render full-featured 2D web content as part placement) of interactive 3D and VR experiences (Section 2). We discover that Some recent developments try to unlock this large body of existing 1Note that Mozilla Firefox does provide a similar API [1], but that it is only web content either in a special purpose VR browser (e.g., the Janus available from within browser extensions and plugins, not to normal web content. developers have been using workarounds to render web content to validate with an emulated rendering-engine3. 2D textures, then mapping them onto 3D primitives for use in VR. Ultimately however, all these approaches require either a custom We find that these methods are far too slow to support interactive browser implementation/plugin and/or propose an extension to 3D WebVR experiences and are inflexible in their handling of existing web standards, making them difficult to deploy in practice user input. We contribute our own DOM2AFrame framework and in combination with modern APIs such as WebVR/WebGL. (Section 3), which does not first render to a 2D texture but instead Additionally, most of the discussed methods focus primarily on couples each 2D web element to an equivalent 3D representation supporting UC0 and sometimes UC2 (highlighting that enabling directly. Our approach provides fully interactive frame rates and UC2 is traditionally difficult without built-in browser support), but more flexible input handling (with support for HTML forms), at the rarely UC1. expense of the resulting render not being a pixel-perfect replica of the browser engine’s result. We validate our approach in Section 2.2. Rendering to a 2D texture 4 with two case studies that show our support for UC1 and UC2 and, in part, UC0. Note that we look primarily at the rendering and As discussed in the previous sections, rendering web content in input capturing aspects and consider techniques for interacting with VR without changing browser source code is difficult because a (Web)VR content out of scope for this work. developer cannot directly access the browser’s internal rendering pipeline through JS or any other means4. However, there are some workarounds that obtain a very similar result by using the available 2. RELATED WORK 2D/3D rendering contexts of the <canvas> element. These methods first render (large) groups of DOM elements to a 2D 2.1. Adjusting the native rendering pipeline image/texture, either by using the internal pipeline in a roundabout way via SVG images (Section 2.2.1) or by re-implementing the Rendering a web page is an iterative process (see Figure 1a). The browser’s Styling logic (Section 2.2.2). This texture is then mapped browser engine starts from the raw HTML/CSS/JS strings and onto a WebGL 3D (quad) primitive for use in VR. builds an internal representation: the Document Object Model 2 Two figures aptly summarize the next two Sections: Figure 1 (DOM ). This internal representation is then used in multiple steps, shows the various pipeline details, while Figure 2 clarifies how each mainly Layouting and Styling. Layouting deals with how elements method maps web elements onto multiple textures. are positioned and sized (e.g., with CSS properties such as width, height, margin) while Styling changes the visual appearance of the elements (e.g., borders, color, font-weight). Once the Layout and 2.2.1. rasterizeHTML and html2three Style are calculated, the browser can use this information to draw Using the special <foreignObject> tag of the XML-based the rasterized representation of the web page via the GPU. When Scalable Vector Graphics (SVG) image standard, HTML and CSS something changes on the web page, the browser only updates content can be embedded in an SVG image element. The <can- the relevant elements of its internal representation to prevent a full vas> 2D drawing API then allows to draw/rasterize the SVG page re-draw for every update. It is thus straightforward for the image, causing the browser to internally render the SVG in largely browser to provide a VR-specific rendering viewport on top of this the same way as it would with normal HTML and CSS content, internal representation, since it mainly needs to change how the using the default browser rendering pipeline (see Figure 1b). content is dispatched to the GPU and can reuse most of the existing However, once again due to security and privacy considerations, implementation. various web features are not directly supported.