Gltf and Openxr: Open Standards for AR in the Cloud

glTF and OpenXR Open standards for AR in the cloud Neil Trevett Khronos President NVIDIA VP Developer Ecosystems [email protected] | @neilt3d

June 2019

© Khronos® Group Inc. 2019 - Page 1 Khronos Mission Khronos members are industry leaders from around the world that join to safely cooperate - to advance their own businesses and the industry as a whole

Software

Silicon

Khronos is an open, member-driven industry consortium developing royalty-free standards, and vibrant ecosystems, to harness the power of silicon acceleration for demanding graphics rendering and computationally intensive applications

Compact to Transmit Simple and Fast to Load Describes Full Scenes Runtime Neutral Open and Extensible glTF 2.0 – June 2017 Efficient, reliable transmission glTF 1.0 – December 2015 Native AND Web APIs Bring 3D assets into 1000s of Primarily for WebGL Physically Based Rendering apps and engines Uses GLSL for materials Metallic-Roughness and Specular-Glossiness

Modo Dedicated 3D Authoring Tools Game Engines

Discover Repositories Authoring Tools that Export 3D Web Engines

Oculus

3D Builder VR / AR Authoring Tools Create Experience Prep for 3D printing Tools Apps / Engines Sony 3D Creator Ecosystem 3D Apps and Engines 3D Live Object 3D Scanning Tools Drive Demand Users Mixed Reality Viewer Collada2gltf FBX2glTF OBJ2GLTF Windows Mixed Convertors and Optimizers Reality Home

glTF Reference Viewer VR / AR Apps and Engines gltf-vscode glTF-asset-generator glTF-validator glTF-Toolkit Validation and Reference Tools Productivity and Social Apps

© Khronos® Group Inc. 2019 - Page 6 Next Generation glTF PBR Materials • glTF 2.0 has high-quality Physically-Based Rendering (PBR) materials - Mandatory ‘Metallic-Roughness’ and optional ‘Specular-Glossiness’ • Working on next generation PBR - Absorption/attenuation, clear coat, subsurface scattering, anisotropy • Extending Metallic-Roughness parameters - Consistency and fallbacks for performance for any device • Inspiration from Dassault Systèmes Enterprise PBR Shading Model (DSPBR) - https://github.com/DassaultSystemes-Technology/EnterprisePBRShadingModel/tree/master/gltf_ext • Wide industry collaboration for compatibility Images from https://dassaultsystemes- - Dassault Systèmes technology.github.io/EnterprisePBRShadingModel/ - Google Filament - Microsoft BabylonJS - OTOY Octane - NVIDIA MDL Join the GitHub Discussion! https://github.com/KhronosGroup/glTF/issues/1442

© Khronos® Group Inc. 2019 - Page 7 CTTF Universal Textures for glTF • glTF 2.0 extension today for Google Draco-based mesh compression - Draco-based compressed point clouds soon! • Binomial and Google recently open sourced ‘Basis Universal’ compressor and transcoder - C++ and WebAssembly for code and Web stacks - https://github.com/binomialLLC/basis_universal • Basis Universal format is being contributed to glTF to enable CTTF Universal Textures • Compressed Texture Transmission Format - ‘Compressed Texture Transmission Format’ - CTTF extension for glTF will use KTX2 as a container format - github.com/KhronosGroup/CTTF-Specification

Encoding decoupled from Transcode on-the-fly to a target device. natively supported GPU format GPU-accelerated One encode pass per Desktop: BC1-5, BC7 Texture Original texture asset Mobile: ETC1/2, PVRTC1, ASTC TextureOriginal CTTF Original AssetsTexture Encode and JPEG-sized Transcode GPU-accelerated AssetsTexture Supercompress Universal to GPU formats Texture Assets (.png) Texture Assets GPU-accelerated Texture

VR AR VR AR VR AR VR AR App App App App App App App App 1 2 3 4 1 2 3 4

Proprietary Proprietary WebXR Engine WebXR Engine

Application Interface

Device Integration Layer

VR VR VR VR VR VR VR VR VR VR Device Device Device Device Device Device Device Device Device Device 1 2 3 4 5 1 2 3 4 5 Before OpenXR After OpenXR XR Market Fragmentation Wide interoperability of XR apps and devices

Application or Engine

Display, High-performance, low-latency composition and Cross-platform access to XR 3D rendering and composition* optical correction HMDs and sensors Multiview parameters XR application lifecycle Context priority Frame timing and display composition Front buffer rendering Sensor tracking and pose calculation Tiled rendering (beam racing) Input device discovery and events Variable rate rendering Haptics Control

* OpenXR can be used with other 3D APIs such as Direct3D, OpenGL and OpenGL ES

OpenXR is a collaborative design Integrating many lessons from proprietary ‘first-generation’ XR API designs

Native XR Apps Web XR Apps Future versions of OpenXR will include cross-platform extended AR functionality

WebXR 3D Engines 3D Engines System-exposed AR Capabilities

Close ongoing collaboration between WebXR and OpenXR

Khronos providing the foundation for 3D and XR in the Web and native stacks

© Khronos® Group Inc. 2019 - Page 12 MEC (Multi-access Edge Computing) Server XR and 5G 1. Processes sensor data, including machine learning for environmental Leveraging High Bandwidth and Low Latency lighting, occlusion, scene semantics, object reconstruction and UI 2. Generates imagery from 3D models, including stereo, foveal rendering, ray-tracing, optics pre-distortion, varifocal processing

Rich Sensor Data Location–aware Content Requests

Sensor handling

Image: Nreal

Wireless mobile device with display and sensors MEC Server Display composition Apps 3D Assets Generated OpenXR APIs can hide Augmentations & Any needed assets the 5G round trip loaded from the from applications Scenes cloud to edge server © Khronos® Group Inc. 2019 - Page 13 Any Questions? • Standards will be key to pervasive XR - In the cloud and on client devices - Compute, XR and content interoperability • Khronos is creating cutting-edge royalty-free open standards - For 3D, VR/AR, Compute, Vision and Machine Learning - Any company welcome to join and help steer the direction of the industry • More Information - www.khronos.org - [email protected] - @neilt3d