Overview and AR/VR Roadmap Neil Trevett Khronos President NVIDIA VP Developer Ecosystems [email protected] | @neilt3d © Khronos® Group Inc. 2018 - Page 1 Khronos Connects Software to Silicon Software Silicon Khronos is an International Industry Consortium creating royalty-free, open standards to enable software to access hardware acceleration for 3D Graphics, Virtual and Augmented Reality, Parallel Computing, Neural Networks and Vision Processing © Khronos® Group Inc. 2018 - Page 2 What is An Open Standard? An INTEROPERABILITY STANDARD enables two entities to COMMUNICATE. E.g. Software <-> Hardware Bad Standards Good Standards - Overprescribes implementation details - Prescribes ONLY interoperability - Forces lowest common denominator - Enables implementation diversity - Stifles innovation – Encourages innovation -> Commoditization -> Differentiation A Truly OPEN standard Is not controlled by a single company – but by the whole industry Is freely available to use by any company without royalty payments Has a well defined IP Framework to protect standard AND member’s IP Standards Grow Markets… … Reduce Costs … … Accelerate Time to Market By reducing consumer confusion and By sharing development between With well-proven functionality, increasing capabilities and usability many companies and driving volume testing and interoperability © Khronos® Group Inc. 2018 - Page 3 The Need for AR/VR Standards Vertically Integrated One-off Solutions in 1990s High-volume Consumer Platforms Today! Interoperability standards so Consumer platforms have insatiable need for content content can run on all VR devices! to drive demand - can’t afford silo’d content! Compete on adding user value – Consumer confidence – will my device run the content not restricting user choice! I want? Avoid Betamax vs. VHS Syndrome! VR/AR will build on a HMD Connectivity Wireless and 5G Verticals and Security constellation of standards VR Video Formats Acceleration APIs XR in the Browser © Khronos® Group Inc. 2018 - Page 4 Standards for AR and VR Download 3D object and scene data Vision and sensor processing - including neural network inferencing for machine learning High-performance, low-latency 3D Graphics Portable interaction with VR/AR sensor, haptic and display devices © Khronos® Group Inc. 2018 - Page 5 Vulkan and New Generation GPU APIs Modern architecture | Low overhead| Multi-thread friendly EXPLICIT GPU access for EFFICIENT, LOW-LATENCY, PREDICTABLE performance Vulkan Porting Tools Non-proprietary, royalty-free open standard ‘By the industry for the industry’ Portable across multiple platforms - desktop and mobile © Khronos® Group Inc. 2018 - Page 6 Vulkan – Now Available Everywhere VR-Related Features NOW Multi-GPU support Multiview Rendering Context priority Front buffer rendering IN DISCUSSION Variable Rate Rendering Tiled rendering (beam racing) © Khronos® Group Inc. 2018 - Page 7 The Metaverse will be the Web! https://xkcd.com/1367/ © Khronos® Group Inc. 2018 - Page 8 WebGL Stack Content downloaded Content from the Web JavaScript, HTML, CSS, ... Low-level WebGL API provides a Middleware provides accessibility JavaScript Middleware powerful foundation for a rich for non-expert programmers JavaScript middleware E.g. three.js library ecosystem CSS Reliable WebGL Browser provides WebGL relies on work by 3D engine alongside other HTML5 both GPU and technologies - no plug-in required JavaScript HTML5 Browser Vendors -> Khronos has the OS Provided Drivers right membership WebGL uses native OpenGL to enable that or OpenGL ES or cooperation Angle = OpenGL ES over DX © Khronos® Group Inc. 2018 - Page 9 WebGL Deployment – WebGL 2.0 is Here! Pervasive, portable access to WebGL 1.0 93.26% Globally OpenGL ES 2.0-class mobile graphics OpenGL ES 3.0-based desktop- class graphics comes to the Web! http://caniuse.com/#feat=webgl WebGL 2.0 65.77% Globally WebGL 2.0 will reach WebGL 1.0 levels of availability when Safari and Edge ship enabling a new class of Web-based AR/VR 3D Apps! © Khronos® Group Inc. 2018 - Page 10 Khronos Standards for AR and VR Download 3D object and scene data Vision and sensor processing - including neural network inferencing for machine learning High-performance, low-latency 3D Graphics Portable interaction with VR/AR sensor, haptic and display devices © Khronos® Group Inc. 2018 - Page 11 glTF – The JPEG of 3D! glTF spec development on open GitHub – get involved! https://github.com/KhronosGroup/glTF Compact to Transmit Simple and Fast to Load Describes Full Scenes Runtime Neutral Open and Extensible glTF 2.0 – June 2017 Efficient, reliable transmission glTF 1.0 – December 2015 Native AND Web APIs Bring 3D assets into 1000s of Primarily for WebGL Physically Based Rendering apps and engines Uses GLSL for materials Metallic-Roughness and Specular-Glossiness © Khronos® Group Inc. 2018 - Page 12 glTF Ecosystem Repositories Creation Tools Discover Windows Mixed Sony 3D Creator Create Experience Reality Home Oculus Modo Paint 3D Drive Demand Mixed Reality 3D Builder Viewer Collada2gltf gltf-vscode FBX2glTF Prep for 3D printing glTF-validator glTF-asset-generator Apps and Engines Users © Khronos® Group Inc. 2018 - Page 13 glTF Roadmap • glTF manages its roadmap very carefully – complexity is the enemy - Mission #1: ensure widespread, consistent, reliable usage • Optional Draco Mesh Compression Extension Shipping Now - Adoption enabled with open source encoders and decoders • Extensions and projects in discussion… - Draco compressed Point clouds, Universal compressed textures, LODs and streaming - Nexgen PBR materials, Advanced animations (e.g. Avatars and Face emoji), Metadata • Developing reference viewer for visual consistency across web and native engines Domain-specific extensions stay as extensions Stable Mesh Compression Core Spec New widely needed Ratios functionality ships first as extensions Integrate extensions into new core spec only when: 1) Widespread need is confirmed by the industry 2) Widespread reliable implementation is enabled (e.g. open source) © Khronos® Group Inc. 2018 - Page 14 Khronos Standards for AR and VR Download 3D object and scene data Vision and sensor processing - including neural network inferencing for machine learning High-performance, low-latency 3D Graphics Portable interaction with VR/AR sensor, haptic and display devices © Khronos® Group Inc. 2018 - Page 15 Machine Learning Acceleration Training on Desktop and Cloud Deployment on Embedded Devices Training Data Sets Vision and Inferencing Applications Live NeuralNeural Net Net Training Training OR Data NeuralTrainingFrameworks Net Training Trained Frameworks Optimization Vision and Inferencing FrameworksFrameworks Networks Runtimes Desktop and Cloud Diverse Inferencing GPU/TPU Acceleration Acceleration Hardware © Khronos® Group Inc. 2018 - Page 16 OpenVX – Vision and Inferencing Acceleration Wide range of vision and inferencing hardware architectures so OpenVX provides a high-level Graph-based abstraction for portability -> Enables Graph-level optimizations! Can be implemented on almost any hardware or processor! -> Portable, Efficient Vision and Inference Processing! Shipping Implementations Vision Node Native Vision Vision Downstream Camera Application Control Node CNN Nodes Node Processing An OpenVX graph mixing CNN nodes with traditional vision nodes NNEF Translator converts NNEF representation into OpenVX Node Graphs © Khronos® Group Inc. 2018 - Page 17 OpenCL – Unique Heterogeneous Runtime OpenCL is the only industry standard for low-level heterogeneous compute Portable control over memory and parallel task execution “The closest you can be to your embedded accelerator and still be portable” Application or Application or Inferencing Run-time Inferencing Run-time Growing number of optimized OpenCL vision and inferencing libraries Vision: OpenCV, Halide, Visioncpp Machine Learning: Xiaomi MACE, Arm Compute Library GPU APIs Linear Algebra: clDNN, clBlast, ViennaCL CPU GPU CPUCPU GPUGPU FPGA DSP Custom Hardware © Khronos® Group Inc. 2018 - Page 18 Khronos Standards for AR and VR Download 3D object and scene data Vision and sensor processing - including neural network inferencing for machine learning High-performance, low-latency 3D Graphics Portable interaction with VR/AR sensor, haptic and display devices © Khronos® Group Inc. 2018 - Page 19 XR = AR + VR V1.0 - focused on VR V A After 1.0 – equal focus on AR © Khronos® Group Inc. 2018 - Page 20 OpenXR – Solving VR/AR Fragmentation VR AR VR AR VR AR VR AR App App App App App App App App 1 2 3 4 1 2 3 4 Proprietary Proprietary WebXR Engine WebXR Engine Application Interface Device Integration Layer VR VR VR VR VR VR VR VR VR VR Device Device Device Device Device Device Device Device Device Device 1 2 3 4 5 1 2 3 4 5 Before OpenXR After OpenXR XR Market Fragmentation Wide interoperability of XR apps and devices © Khronos® Group Inc. 2018 - Page 21 OpenXR Architecture Portable AR/VR Input Device Discovery Multiple Sensor Tracking Device Events Pose Normalization Haptics Control Optical Corrections OpenXR doesn’t replace AR/VR runtimes! It enables those runtimes to use PORTABLE APIs to expose their functionality © Khronos® Group Inc. 2018 - Page 22 Companies Publicly Supporting OpenXR is a collaborative design Integrating many lessons from proprietary ‘first-generation’ API designs © Khronos® Group Inc. 2018 - Page 23 Epic ‘Showdown’ VR Demo at SIGGRAPH Demo runs portably across StarVR and Microsoft Windows Mixed Reality headsets through the OpenXR APIs via an Unreal Engine 4 plugin https://www.youtube.com/watch?v=FCAM-3aAzXg&t=17250s