Building the Metaverse One Open Standard at a Time
Total Page:16
File Type:pdf, Size:1020Kb
Building the Metaverse One Open Standard at a Time Khronos APIs and 3D Asset Formats for XR Neil Trevett Khronos President NVIDIA VP Developer Ecosystems [email protected]| @neilt3d April 2020 © Khronos® Group 2020 This work is licensed under a Creative Commons Attribution 4.0 International License © The Khronos® Group Inc. 2020 - Page 1 Khronos Connects Software to Silicon Open interoperability standards to enable software to effectively harness the power of multiprocessors and accelerator silicon 3D graphics, XR, parallel programming, vision acceleration and machine learning Non-profit, member-driven standards-defining industry consortium Open to any interested company All Khronos standards are royalty-free Well-defined IP Framework protects participant’s intellectual property >150 Members ~ 40% US, 30% Europe, 30% Asia This work is licensed under a Creative Commons Attribution 4.0 International License © The Khronos® Group Inc. 2020 - Page 2 Khronos Active Initiatives 3D Graphics 3D Assets Portable XR Parallel Computation Desktop, Mobile, Web Authoring Augmented and Vision, Inferencing, Machine Embedded and Safety Critical and Delivery Virtual Reality Learning Guidelines for creating APIs to streamline system safety certification This work is licensed under a Creative Commons Attribution 4.0 International License © The Khronos® Group Inc. 2020 - Page 3 Pervasive Vulkan Desktop and Mobile GPUs http://vulkan.gpuinfo.org/ Platforms Apple Desktop Android (via porting Media Players Consoles (Android 7.0+) layers) (Vulkan 1.1 required on Android Q) Virtual Reality Cloud Services Game Streaming Embedded Engines Croteam Serious Engine Note: The version of Vulkan available will depend on platform and vendor This work is licensed under a Creative Commons Attribution 4.0 International License © The Khronos® Group Inc. 2020 - Page 4 Vulkan Roadmap Vulkan 1.1 Extensions Maintenance updates plus additional functionality Roadmap Discussions Timeline semaphores Ray Tracing DX/HLSL compatibility Variable Rate Shading Bindless resources Reduced precision arithmetic Accelerated Video Encode/Decode Formal memory model Machine Learning Primitives Buffer references Mesh Shaders SPIR-V 1.5 January 2020 Variable Rate Shading Enables concentration of rendering power at the foveal point in eye- tracked XR systems This work is licensed under a Creative Commons Attribution 4.0 International License © The Khronos® Group Inc. 2020 - Page 5 Ray Tracing is a Flexible Technique Ambient Occlusion Reflections Programmers need programmable flexibility to trace rays through scenes for a wide variety of visual effects – some examples… Shadows Global Illumination This work is licensed under a Creative Commons Attribution 4.0 International License © The Khronos® Group Inc. 2020 - Page 6 Vulkan Ray Tracing Two ways to launch rays into scene to generate high realistic visuals Using a BVH data structure to Ray Tracing Pipelines enable efficient ray tracing A new type of graphics pipeline through a 3D scene Implicit management of ray intersections Application compiles a set of shaders into the pipeline to 1. Launch 2D/3D grid of rays provide desired ray and material processing Ray into scene contained in an Generation Acceleration Structure Ray Queries 3. Invoke ‘Any Hit’ Shader if intersection is found. Multiple Any type of shader can launch a ray at any time intersections possible - arbitrary Any Hit Shader can process intersection data however it wishes Acceleration order Shader controls how traversal proceeds Structure Traversal Intersection 2. ‘Intersection’ Shader computes ray intersections Ray-triangle intersections are built-in Hit? No Yes 4. Invoke ‘Closest Hit’ shader on the closest intersection of the ray OR Miss Closest Hit Invoke ‘Miss’ Shader if no hit is found Can trace more rays Ray Tracing Pipeline This work is licensed under a Creative Commons Attribution 4.0 International License © The Khronos® Group Inc. 2020 - Page 7 OpenXR – Cross-Platform Portable AR/VR Working Group Participants Games Engines Desktop/Console VR AR Mobile VR GPUs OpenXR is a collaborative design UI Devices Integrating many lessons from proprietary ‘first-generation’ XR API designs * OpenXR 1.0 is focused on enabling cross-platform applications. Optional device plugin interface will be supported post V1.0 This work is licensed under a Creative Commons Attribution 4.0 International License © The Khronos® Group Inc. 2020 - Page 8 OpenXR 1.0 Availability Significant Community Feedback Finalize Conformance Test Suite Enable Officially Conformant Implementations Improved Input subsystem, game engine editor support, loader … Provisional OpenXR for Windows Mixed Reality headsets Specification and HoloLens 2 GDC, March 2019 PLUS extensions to support HoloLens 2 hand tracking, eye tracking, spatial mapping and spatial anchors Many more coming! Ratify and Release OpenXR support for Oculus Rift and Oculus Quest OpenXR 1.0 Oculus PC (for Rift) and Android SDK (for Quest) SIGGRAPH, July 2019 include OpenXR for native C/C++ development ‘Monado’ OpenXR open source implementation Support OpenHMD hardware and Nova Northstar AR HMD OpenXR 1.0 plugin for Unreal Engine v4.2.4 This work is licensed under a Creative Commons Attribution 4.0 International License © The Khronos® Group Inc. 2020 - Page 9 OpenXR Win-Win-Win XR End-Users Can run the apps they want on their system – reducing market confusion and increasing consumer confidence XR Vendors Can bring more applications onto their platform by leveraging the OpenXR content ecosystem XR ISVs Can easily ship on more platforms for increased market reach This work is licensed under a Creative Commons Attribution 4.0 International License © The Khronos® Group Inc. 2020 - Page 10 OpenXR is used with a 3D API Application or Engine Display, High-performance, low-latency composition and Cross-platform access to XR 3D rendering and composition* optical correction HMDs and sensors Multiview parameters XR application lifecycle Context priority Input device discovery and events Front buffer rendering Sensor tracking and pose calculation Tiled rendering (beam racing) Frame timing and display composition Variable rate rendering Haptics Control * OpenXR can be used with other 3D APIs such as Direct3D, OpenGL and OpenGL ES This work is licensed under a Creative Commons Attribution 4.0 International License © The Khronos® Group Inc. 2020 - Page 11 Bringing XR to the Web The Web will Evolve Native XR Apps into the Metaverse Web XR Apps Lifting OpenXR functionality into the Web stack WebXR Native 3D Web 3D Engines Close cooperation Engines between WebXR and OpenXR Khronos provides the foundation for native and Web-based 3D/XR This work is licensed under a Creative Commons Attribution 4.0 International License © The Khronos® Group Inc. 2020 - Page 12 OpenXR and Edge Server Applications with 5G Low latency Sensor Data Location–aware Content Requests OpenXR APIs hide the 5G round trip from applications MEC (Multi-access Edge Sensor handling Computing) Server 1. Processes sensor data, including machine learning for environmental lighting, occlusion, Needed assets Display composition scene semantics, object loaded to reconstruction and UI edge server 2. Generates imagery from 3D Wireless mobile device with models, including stereo, foveal rendering, ray-tracing, optics pre- display and sensors distortion, varifocal processing Apps and 3D Assets NVIDIA EGX Generated Augmentations & Scenes This work is licensed under a Creative Commons Attribution 4.0 International License © The Khronos® Group Inc. 2020 - Page 13 glTF 2.0 Scene Description Structure .gltf (JSON) Node hierarchy, PBR material textures, cameras .bin .png Geometry: vertices and indices .jpg Animation: key-frames .ktx2 Skins: inverse-bind matrices Textures Mandatory Metallic-Roughness Materials Optional Specular-Glossiness Materials Geometry Texture based PBR materials This work is licensed under a Creative Commons Attribution 4.0 International License © The Khronos® Group Inc. 2020 - Page 14 This work is licensed under a Creative Commons Attribution 4.0 International License © The Khronos® Group Inc. 2020 - Page 15 glTF Roadmap Second Generation Physically-Based Rendering (PBR) JPEG must be fully Set of coherent extensions decompressed into 14,000,000Clear coat (imminent) GPU memory Absorption/attenuation 12,582,912 File Size GPU Size Subsurface scattering Anisotropy Inspiration from Dassault 10,500,000 Systèmes Enterprise PBR Shading Model (DSPBR) and MDL Wide industry cooperation 7,000,000 Universal textures can beSeeking Requirements directly transcoded to Subdivision3,500,000 surfaces FlightHelmet_baseColor 2,778,518 compressed GPU textures 2048 x 2048, RGB Advanced Animation 2,097,152 LOD and Streaming Compressed Point Clouds Cross-asset linking 315,619 232,104 0Enhanced Metadata Composability PNG JPEG glTF Universal Textures ETC1S Instancing (imminent) Working Group is constantly balancing CAD/BIM model support Basis Universal encoding/transcoding feature requests against the ‘glTF Encryption and security Uncompressed Basis Universal KTX2 Container Prime Directive’ - remain a universal 3D Printing and easy to process delivery format This work is licensed under a Creative Commons Attribution 4.0 International License © The Khronos® Group Inc. 2020 - Page 16 3D Commerce - The Opportunity 3D Commerce = E Commerce enhanced with the use of 3D Models on any platform – including VR and AR Early Experience Shows Increased customer engagement! Strengthened brand loyalty! Deeper product understanding! =$$$!