sign in sign up
<<
Home , Metaverse, OpenXR, Vulkan (API)

Building the One Open Standard at a Time Khronos and 3D Asset Formats for XR Neil Trevett Khronos President VP Developer Ecosystems [email protected]| @neilt3d

April 2020

© Khronos® Group 2020 This work is licensed under a Creative Commons Attribution 4.0 International License © The Khronos® Group Inc. 2020 - Page 1 Khronos Connects to Silicon Open interoperability standards to enable software to effectively harness the power of multiprocessors and accelerator silicon

3D graphics, XR, parallel programming, vision acceleration and machine learning

Non-profit, member-driven standards-defining industry consortium

Open to any interested company

All Khronos standards are royalty-free

Well-defined IP Framework protects participant’s intellectual property >150 Members ~ 40% US, 30% Europe, 30% Asia

This work is licensed under a Creative Commons Attribution 4.0 International License © The Khronos® Group Inc. 2020 - Page 2 Khronos Active Initiatives

3D Graphics 3D Assets Portable XR Parallel Computation Desktop, Mobile, Web Authoring Augmented and Vision, Inferencing, Machine Embedded and Safety Critical and Delivery Learning

Guidelines for creating APIs to streamline system safety certification

This work is licensed under a Creative Commons Attribution 4.0 International License © The Khronos® Group Inc. 2020 - Page 3 Pervasive Vulkan Desktop and Mobile GPUs

http://vulkan.gpuinfo.org/ Platforms

Apple Desktop Android (via porting Media Players Consoles (Android 7.0+) layers) (Vulkan 1.1 required on Android Q)

Virtual Reality Cloud Services Game Streaming Embedded Engines

Croteam Serious Engine

Note: The version of Vulkan available will depend on platform and vendor This work is licensed under a Creative Commons Attribution 4.0 International License © The Khronos® Group Inc. 2020 - Page 4 Vulkan Roadmap

Vulkan 1.1 Extensions Maintenance updates plus additional functionality Roadmap Discussions Timeline semaphores DX/HLSL compatibility Variable Rate Shading Bindless resources Reduced precision arithmetic Accelerated Video Encode/Decode Formal memory model Machine Learning Primitives Buffer references Mesh SPIR-V 1.5

January 2020 Variable Rate Shading Enables concentration of rendering power at the foveal point in eye- tracked XR systems

This work is licensed under a Creative Commons Attribution 4.0 International License © The Khronos® Group Inc. 2020 - Page 5 Ray Tracing is a Flexible Technique

Ambient Occlusion Reflections

Programmers need programmable flexibility to trace rays through scenes for a wide variety of visual effects – some examples…

Shadows Global Illumination

This work is licensed under a Creative Commons Attribution 4.0 International License © The Khronos® Group Inc. 2020 - Page 6 Vulkan Ray Tracing

Two ways to launch rays into scene to generate high realistic visuals

Using a BVH data structure to Ray Tracing Pipelines enable efficient ray tracing A new type of graphics pipeline through a 3D scene Implicit management of ray intersections

Application compiles a set of shaders into the pipeline to 1. Launch 2D/3D grid of rays provide desired ray and material processing Ray into scene contained in an Generation Acceleration Structure

Ray Queries 3. Invoke ‘Any Hit’ if intersection is found. Multiple Any type of shader can launch a ray at any time intersections possible - arbitrary Any Hit Shader can process intersection data however it wishes Acceleration order Shader controls how traversal proceeds Structure Traversal Intersection 2. ‘Intersection’ Shader computes ray intersections Ray-triangle intersections are built-in Hit?

No Yes 4. Invoke ‘Closest Hit’ shader on the closest intersection of the ray OR Miss Closest Hit Invoke ‘Miss’ Shader if no hit is found Can trace more rays Ray Tracing Pipeline

This work is licensed under a Creative Commons Attribution 4.0 International License © The Khronos® Group Inc. 2020 - Page 7 OpenXR – Cross-Platform Portable AR/VR

Working Group Participants

Games Engines

Desktop/Console VR

AR

Mobile VR

GPUs

OpenXR is a collaborative design UI Devices Integrating many lessons from proprietary ‘first-generation’ XR API designs

* OpenXR 1.0 is focused on enabling cross-platform applications. Optional device plugin interface will be supported post V1.0 This work is licensed under a Creative Commons Attribution 4.0 International License © The Khronos® Group Inc. 2020 - Page 8 OpenXR 1.0 Availability

Significant Community Feedback Finalize Conformance Test Suite Enable Officially Conformant Implementations Improved Input subsystem, game engine editor support, loader … Provisional OpenXR for Windows headsets Specification and HoloLens 2 GDC, March 2019 PLUS extensions to support HoloLens 2 hand tracking, eye tracking, spatial mapping and spatial anchors Many more coming! Ratify and Release OpenXR support for Rift and OpenXR 1.0 Oculus PC (for Rift) and Android SDK (for Quest) SIGGRAPH, July 2019 include OpenXR for native C/C++ development

‘Monado’ OpenXR open implementation Support OpenHMD hardware and Nova Northstar AR HMD

OpenXR 1.0 plugin for v4.2.4

This work is licensed under a Creative Commons Attribution 4.0 International License © The Khronos® Group Inc. 2020 - Page 9 OpenXR Win-Win-Win XR End-Users Can run the apps they want on their system – reducing market confusion and increasing consumer confidence

XR Vendors Can bring more applications onto their platform by leveraging the OpenXR content ecosystem

XR ISVs Can easily ship on more platforms for increased market reach

This work is licensed under a Creative Commons Attribution 4.0 International License © The Khronos® Group Inc. 2020 - Page 10 OpenXR is used with a 3D API

Application or Engine

Display, High-performance, low-latency composition and Cross-platform access to XR 3D rendering and composition* optical correction HMDs and sensors Multiview parameters XR application lifecycle Context priority Input device discovery and events Front buffer rendering Sensor tracking and pose calculation Tiled rendering (beam racing) Frame timing and display composition Variable rate rendering Haptics Control

* OpenXR can be used with other 3D APIs such as , OpenGL and OpenGL ES

This work is licensed under a Creative Commons Attribution 4.0 International License © The Khronos® Group Inc. 2020 - Page 11 Bringing XR to the Web

The Web will Evolve Native XR Apps into the Metaverse Web XR Apps

Lifting OpenXR functionality into the Web stack WebXR Native 3D Web 3D Engines Close cooperation Engines between WebXR and OpenXR

Khronos provides the foundation for native and Web-based 3D/XR

This work is licensed under a Creative Commons Attribution 4.0 International License © The Khronos® Group Inc. 2020 - Page 12 OpenXR and Edge Server Applications with 5G

Low latency Sensor Data

Location–aware Content Requests OpenXR APIs hide the 5G round trip from applications MEC (Multi-access Edge Sensor handling Computing) Server 1. Processes sensor data, including machine learning for environmental lighting, occlusion, Needed assets Display composition scene semantics, object loaded to reconstruction and UI edge server 2. Generates imagery from 3D Wireless mobile device with models, including stereo, foveal rendering, ray-tracing, optics pre- display and sensors distortion, varifocal processing Apps and 3D Assets NVIDIA EGX

Generated Augmentations & Scenes

This work is licensed under a Creative Commons Attribution 4.0 International License © The Khronos® Group Inc. 2020 - Page 13 glTF 2.0 Scene Description Structure

. (JSON) Node hierarchy, PBR material textures, cameras

.bin .png Geometry: vertices and indices .jpg Animation: key-frames .ktx2 Skins: inverse-bind matrices Textures

Mandatory Metallic-Roughness Materials

Optional Specular-Glossiness Materials

Geometry Texture based PBR materials

This work is licensed under a Creative Commons Attribution 4.0 International License © The Khronos® Group Inc. 2020 - Page 14 This work is licensed under a Creative Commons Attribution 4.0 International License © The Khronos® Group Inc. 2020 - Page 15 glTF Roadmap Second Generation Physically-Based Rendering (PBR) JPEG must be fully Set of coherent extensions decompressed into 14,000,000Clear coat (imminent) GPU memory Absorption/attenuation 12,582,912 File Size GPU Size Subsurface scattering Anisotropy

Inspiration from Dassault 10,500,000 Systèmes Enterprise PBR Shading Model (DSPBR) and MDL

Wide industry cooperation

7,000,000

Universal textures can beSeeking Requirements directly transcoded to Subdivision3,500,000 surfaces FlightHelmet_baseColor 2,778,518 compressed GPU textures 2048 x 2048, RGB Advanced Animation 2,097,152 LOD and Streaming Compressed Point Clouds Cross-asset linking 315,619 232,104 0Enhanced Metadata Composability PNG JPEG

glTF Universal Textures ETC1S Instancing (imminent) Working Group is constantly balancing CAD/BIM model support Basis Universal encoding/transcoding feature requests against the ‘glTF Encryption and security Uncompressed Basis Universal KTX2 Container Prime Directive’ - remain a universal 3D Printing and easy to process delivery format

This work is licensed under a Creative Commons Attribution 4.0 International License © The Khronos® Group Inc. 2020 - Page 16 3D Commerce - The Opportunity

3D Commerce = E Commerce enhanced with the use of 3D Models on any platform – including VR and AR

Early Experience Shows Increased customer engagement! Strengthened brand loyalty! Deeper product understanding! =$$$! More online sales! Fewer returns!

IKEA catalog uses to give a virtual So why is 3D Commerce preview of furniture in a room – August 2013 taking so long to become widespread?

This work is licensed under a Creative Commons Attribution 4.0 International License © The Khronos® Group Inc. 2020 - Page 17 3D Commerce – Today’s Reality

I wish I had high quality, Everyone defines their product I need the materials in realistic 3D models for virtual data for sizes and colors my 3D models to look promotional photoshoots! differently – nothing is consistent! completely realistic!

Products don’t The green couch looks come with 3D data – blue on some devices – and I can’t lots of product returns physically scan them are expensive! all fast enough! Many 3D products on CAD tools don’t let my ecommerce web- me easily generate site first appear upside the data I need for E down! I have to hand- Commerce! tune 1000s of models!

Complex retail pipeline with hundreds of companies and millions of products Many friction points: tooling, technical and commercial 3D Commerce can’t reach industrial scale so… Interoperability Standards to the rescue!

This work is licensed under a Creative Commons Attribution 4.0 International License © The Khronos® Group Inc. 2020 - Page 18 Khronos 3D Working Group

Announced SIGGRAPH 2019

Creating specifications and guidelines to align the 3D asset workflow from product design through manufacturing and each stage of retail to end-user delivery platforms

Any company is welcome to join! https://www.khronos.org/3dcommerce [email protected]

Broad Industry Participation from tooling, retail, technology and platform companies

This work is licensed under a Creative Commons Attribution 4.0 International License © The Khronos® Group Inc. 2020 - Page 19 3D Commerce - Four Initial Areas of Focus

Asset Creation Guidelines Product Configuration For tools and product designers to Universal product configurability create assets with consistent data to be data and guidelines on how to drive used through the 3D Commerce pipeline consistent product display

First Goals Helping evolve glTF to meet the needs of 3D Commerce - Next generation PBR - Advanced metadata

Industry cooperation to urgently develop guidelines and tools to address priority problem areas

Metadata Viewer Validation and Certification Structured metadata definitions and Test models, reference viewer, display analysis examples to consistently carry product tools and capability specifications to guarantee information through the retail pipeline a consistent and accurate end user experience

This work is licensed under a Creative Commons Attribution 4.0 International License © The Khronos® Group Inc. 2020 - Page 20 How To Get Involved! • Any company or organization is welcome to join Khronos! - For a voice and a vote in any of these standards – membership starts at $3,500 • OR request an invite to Vulkan, OpenXR and 3D Commerce Advisory Panels/Forums - No fee, under Khronos NDA - and IP Framework for draft specification feedback • We welcome your feedback - Khronos Forums: https://forums.khronos.org/ - Khronos Slack Channels: https://khronosdevs.slack.com/messages - Khronos open source GitHub repositories: https://github.khronos.org/ • Contact Neil Trevett - [email protected] | @neilt3d | www.khronos.org

This work is licensed under a Creative Commons Attribution 4.0 International License © The Khronos® Group Inc. 2020 - Page 21