Khronos Template 2015

Overview and AR/VR Roadmap

Neil Trevett Khronos President NVIDIA VP Developer Ecosystems [email protected] | @neilt3d

Software

Silicon

Khronos is an International Industry Consortium creating royalty-free, open standards to enable software to access hardware acceleration for 3D Graphics, Virtual and Augmented Reality, Parallel Computing, Neural Networks and Vision Processing

An INTEROPERABILITY STANDARD enables two entities to COMMUNICATE. E.g. Software <-> Hardware

Bad Standards Good Standards - Overprescribes implementation details - Prescribes ONLY interoperability - Forces lowest common denominator - Enables implementation diversity - Stifles innovation – Encourages innovation -> Commoditization -> Differentiation

A Truly OPEN standard Is not controlled by a single company – but by the whole industry Is freely available to use by any company without royalty payments Has a well defined IP Framework to protect standard AND member’s IP

Standards Grow Markets… … Reduce Costs … … Accelerate Time to Market By reducing consumer confusion and By sharing development between With well-proven functionality, increasing capabilities and usability many companies and driving volume testing and interoperability

Vertically Integrated One-off Solutions in 1990s High-volume Consumer Platforms Today!

Interoperability standards so Consumer platforms have insatiable need for content content can run on all VR devices! to drive demand - can’t afford silo’d content! Compete on adding user value – Consumer confidence – will my device run the content not restricting user choice! I want? Avoid Betamax vs. VHS Syndrome!

VR/AR will build on a HMD Connectivity Wireless and 5G Verticals and Security constellation of standards

VR Video Formats Acceleration APIs XR in the Browser

Download 3D object and scene data Vision and sensor processing - including neural network inferencing for machine learning

High-performance, low-latency 3D Graphics

Portable interaction with VR/AR sensor, haptic and display devices

Modern architecture | Low overhead| Multi-thread friendly EXPLICIT GPU access for EFFICIENT, LOW-LATENCY, PREDICTABLE performance

Vulkan Porting Tools

Non-proprietary, royalty-free open standard ‘By the industry for the industry’ Portable across multiple platforms - desktop and mobile

VR-Related Features NOW Multi-GPU support Multiview Rendering Context priority Front buffer rendering

IN DISCUSSION Variable Rate Rendering Tiled rendering (beam racing)

https://xkcd.com/1367/

Content downloaded Content from the Web JavaScript, HTML, CSS, ...

Low-level WebGL API provides a Middleware provides accessibility JavaScript Middleware powerful foundation for a rich for non-expert programmers JavaScript middleware E.g. three.js library ecosystem

CSS Reliable WebGL Browser provides WebGL relies on work by 3D engine alongside other HTML5 both GPU and technologies - no plug-in required JavaScript HTML5 Browser Vendors -> Khronos has the OS Provided Drivers right membership WebGL uses native OpenGL to enable that or OpenGL ES or cooperation Angle = OpenGL ES over DX

Pervasive, portable access to WebGL 1.0 93.26% Globally OpenGL ES 2.0-class mobile graphics

OpenGL ES 3.0-based desktop- class graphics comes to the Web! http://caniuse.com/#feat=webgl WebGL 2.0 65.77% Globally

WebGL 2.0 will reach WebGL 1.0 levels of availability when Safari and Edge ship enabling a new class of Web-based AR/VR 3D Apps!

Download 3D object and scene data Vision and sensor processing - including neural network inferencing for machine learning

High-performance, low-latency 3D Graphics

Portable interaction with VR/AR sensor, haptic and display devices

Compact to Transmit Simple and Fast to Load Describes Full Scenes Runtime Neutral Open and Extensible glTF 2.0 – June 2017 Efficient, reliable transmission glTF 1.0 – December 2015 Native AND Web APIs Bring 3D assets into 1000s of Primarily for WebGL Physically Based Rendering apps and engines Uses GLSL for materials Metallic-Roughness and Specular-Glossiness

Discover

Windows Mixed Sony 3D Creator Create Experience Reality Home Oculus

Modo Paint 3D Drive Demand

Mixed Reality 3D Builder Viewer Collada2gltf gltf-vscode FBX2glTF Prep for 3D printing

glTF-validator glTF-asset-generator Apps and Engines Users © Khronos® Group Inc. 2018 - Page 13 glTF Roadmap • glTF manages its roadmap very carefully – complexity is the enemy - Mission #1: ensure widespread, consistent, reliable usage • Optional Draco Mesh Compression Extension Shipping Now - Adoption enabled with open source encoders and decoders • Extensions and projects in discussion… - Draco compressed Point clouds, Universal compressed textures, LODs and streaming - Nexgen PBR materials, Advanced animations (e.g. Avatars and Face emoji), Metadata • Developing reference viewer for visual consistency across web and native engines

Domain-specific extensions stay as extensions Stable Mesh Compression Core Spec New widely needed Ratios functionality ships first as extensions

Integrate extensions into new core spec only when: 1) Widespread need is confirmed by the industry 2) Widespread reliable implementation is enabled (e.g. open source)

Download 3D object and scene data Vision and sensor processing - including neural network inferencing for machine learning

High-performance, low-latency 3D Graphics

Portable interaction with VR/AR sensor, haptic and display devices

Training on Desktop and Cloud Deployment on Embedded Devices

Training Data Sets

Vision and Inferencing Applications Live NeuralNeural Net Net Training Training OR Data NeuralTrainingFrameworks Net Training Trained Frameworks Optimization Vision and Inferencing FrameworksFrameworks Networks Runtimes

Desktop and Cloud Diverse Inferencing GPU/TPU Acceleration Acceleration Hardware

Wide range of vision and inferencing hardware architectures so OpenVX provides a high-level Graph-based abstraction for portability -> Enables Graph-level optimizations! Can be implemented on almost any hardware or processor! -> Portable, Efficient Vision and Inference Processing!

Shipping Implementations

Vision Node Native Vision Vision Downstream Camera Application Control Node CNN Nodes Node Processing

An OpenVX graph mixing CNN nodes with traditional vision nodes NNEF Translator converts NNEF representation into OpenVX Node Graphs

OpenCL is the only industry standard for low-level heterogeneous compute Portable control over memory and parallel task execution “The closest you can be to your embedded accelerator and still be portable”

Application or Application or Inferencing Run-time Inferencing Run-time

Growing number of optimized OpenCL vision and inferencing libraries Vision: OpenCV, Halide, Visioncpp Machine Learning: Xiaomi MACE, Arm Compute Library GPU APIs Linear Algebra: clDNN, clBlast, ViennaCL

CPU GPU CPUCPU GPUGPU

FPGA DSP

Custom Hardware

Download 3D object and scene data Vision and sensor processing - including neural network inferencing for machine learning

High-performance, low-latency 3D Graphics

Portable interaction with VR/AR sensor, haptic and display devices

V1.0 - focused on VR V A After 1.0 – equal focus on AR

VR AR VR AR VR AR VR AR App App App App App App App App 1 2 3 4 1 2 3 4

Proprietary Proprietary WebXR Engine WebXR Engine

Application Interface

Device Integration Layer

VR VR VR VR VR VR VR VR VR VR Device Device Device Device Device Device Device Device Device Device 1 2 3 4 5 1 2 3 4 5 Before OpenXR After OpenXR XR Market Fragmentation Wide interoperability of XR apps and devices

Portable AR/VR Input Device Discovery Multiple Sensor Tracking Device Events Pose Normalization Haptics Control Optical Corrections

OpenXR doesn’t replace AR/VR runtimes! It enables those runtimes to use PORTABLE APIs to expose their functionality

OpenXR is a collaborative design Integrating many lessons from proprietary ‘first-generation’ API designs

Demo runs portably across StarVR and Microsoft Windows Mixed Reality headsets through the OpenXR APIs via an Unreal Engine 4 plugin https://www.youtube.com/watch?v=FCAM-3aAzXg&t=17250s

© Khronos® Group Inc. 2018 - Page 24 Mobile Augmented Reality Libraries Encapsulated Vision-based Functionality Also leveraging motion sensors ARKit Pose Tracking Yes Yes Plane Detection and Tracking (Hori. & Vert.) Yes Yes Image Recognition and Tracking Yes Yes 3D Object Scanning, Recognition and Tracking Yes No Ambient light level and temperature Yes Yes 3rd party AR Libraries Environment Light Probes Yes No typically use Environment Reflective Texturing Yes No ARKit/ARCore if available or Link to Neural Net-based Object Detection Yes Yes implement own Access to Point Cloud Yes Yes tracking and Camera Intrinsics Yes Yes functionality if not Multi-user and persistent cloud-based anchors Yes Yes Multi-user Viewing Yes No Face tracking (iPhone X) Yes No OS Availability iOS Android and iOS

Native XR Apps Web XR Apps Future versions of OpenXR will include cross-platform extended AR functionality WebXR 3D Engines 3D Engines System-exposed AR Capabilities

Close ongoing collaboration between WebXR and OpenXR

Khronos providing the foundation for 3D and XR in the Web and native stacks

© Khronos® Group Inc. 2018 - Page 26 Summary • Khronos is creating cutting-edge royalty-free open standards - For 3D, VR/AR, Compute, Vision and machine learning • Join Khronos – any company is welcome! - Influence the direction of industry standards - Get early access to draft specifications • More Information - www.khronos.org - [email protected] - @neilt3d