SID Khronos Open Standards for AR May17

Open Standards for AR Neil Trevett | Khronos President NVIDIA VP Developer Ecosystem [email protected] | @neilt3d LA, May 2017

Software

Silicon

Khronos is an International Industry Consortium of over 100 companies creating royalty-free, open standard APIs to enable software to access hardware acceleration for 3D graphics, Virtual and Augmented Reality, Parallel Computing, Neural Networks and Vision Processing

3D for the Web Real-time 2D/3D - Real-time apps and games in-browser - Cross-platform gaming and UI - Efficiently delivering runtime 3D assets - VR and AR Displays - CAD and Product Design - Safety-critical displays

VR, Vision, Neural Networks Parallel Computation - VR/AR system portability - Tracking and odometry - Machine Learning acceleration - Embedded vision processing - Scene analysis/understanding - High Performance Computing (HPC) - Neural Network inferencing

Without API Standards With API Standards

Platform Application Fragmentation Portability

Everything Silicon runs on CPU Acceleration

Standard Acceleration APIs provide PERFORMANCE, POWER AND PORTABILITY

Download 3D augmentation object and scene data Tracking and Positioning Generate Low Latency Vision Geometric scene 3D Augmentations for sensor(s) reconstruction display by optical system Semantic scene understanding (Neural Networks) AR Application

Interact with sensor, haptic and display devices

7 Cross Platform

All Major GPU Companies shipping Vulkan Drivers – for Desktop and Mobile Platforms

Mobile, Embedded and Console Platforms Supporting Vulkan

Android 7.0 Nintendo Switch Android TV Embedded Linux

Vulkan support Initial Vulkan mobile DOTA 2 on Vulkan DOOM shipping on Vulkan support Vulkan support in V5.4 in Unity 5.6 renderer in 4.12 port of Source 2 Vulkan port of id Tech 6 since V1.8 Late May 2017 © Copyright Khronos Group 2017 - Page 6 Vulkan Explicit GPU Control Vulkan = high performance and low latency 3D – Ideal for VR/AR applications

Resource management offloaded to app: low-overhead, low-latency Complex drivers Application driver cause overhead Single thread per context and inconsistent Application Consistent behavior: behavior across Memory allocation Multiple Front-end no ‘fighting with driver vendors Thread management heuristics’ Synchronization Compilers High-level Driver GLSL, HLSL etc. Always active Multi-threaded generation Validation and debug layers of command buffers error handling Abstraction loaded only when needed Layered GPU Control Full GLSL Context management SPIR-V SPIR-V intermediate preprocessor and Memory allocation pre-compiled shaders language: shading language compiler in Full GLSL compiler flexibility Error detection driver Thin Driver Loadable debug and Explicit GPU Control validation layers Multi-threaded command OpenGL vs. creation. Multiple graphics, OpenGL ES command and DMA queues GPU GPU Unified API across all platforms with feature set flexibility Vulkan 1.0 provides access to OpenGL ES 3.1 / OpenGL 4.X-class GPU functionality but with increased performance and flexibility

Khronos 3D APIs already But What About powering VR rendering INPUT Hardware for AR/VR.. Device discovery Multiple sensor tracking Device Events The standard for desktop VR today Haptics Parameters for optics corrections etc. etc…

On millions of mobile VR devices

Powering WebVR in browsers OpenXR! Cross-Platform, portable AR/VR

The future of explicit, low-latency VR

VR VR VR VR VR VR VR VR App App App App App App App App 1 2 3 4 1 2 3 4

Proprietary Proprietary Engine Engine

Application Interface

Device Layer

VR VR VR VR VR VR VR VR VR VR Device Device Device Device Device Device Device Device Device Device 1 2 3 4 5 1 2 3 4 5 Before OpenXR After OpenXR VR Market Wide interoperabilityof Fragmentation VR apps and devices

Design work started in December 2016 Typically 12-18 months to develop a V1.0 specification

Access to any OpenXR Application

Any successful standard encourages and Proprietary APIs enables healthy industry competition Extensions Application Interface OpenXR will not replace VR Run-time VR run-times – or outlaw existing interfaces

Proprietary OpenXR will simply provide cross-vendor Driver Interfaces APIs that can be exposed by a runtime to Extensions Device Layer access more apps and devices

Access to any OpenXR Device OpenXR for portable AR/VR apps and devices – with initial focus on VR

© Copyright Khronos Group 2017 - Page 11 OpenVX Vision Engines Wide range of vision hardware architectures Middleware OpenVX provides a high-level Graph-based abstraction -> Applications Graph-level optimizations Can be implemented on almost any hardware or processor -> Portable, Efficient Vision Processing! Software GPU Portability DSP Hardware X100 Dedicated Hardware Vision DSPs X10 Vision GPU Node Compute Vision Vision Node Node Multi-core Vision CPU Node Power Efficiency X1 Computation Flexibility Vision Processing Graph

© Copyright Khronos Group 2017 - Page 12 OpenVX - Graph-Level Abstraction • OpenVX developers express a graph of image operations (‘Nodes’) - Using a C API • Nodes can be executed on any hardware or processor coded in any language - Implementers can optimize under the high-level graph abstraction • Graphs are the key to run-time power and performance optimizations - E.g. Node fusion, tiled graph processing for cache efficiency etc.

Pyr Camera OpenVX Nodes t Rendering Input Output

RGB YUV Gray Array of Frame Color Frame Channel Frame Image Optical Keypoints Harris Conversion Extract Pyramid Flow Track

Array of Features OpenVX Graph Ftrt-1

Feature Extraction Example Graph

E.g. AMD Open source OpenVX Extensions Neural Network Acceleration To o ls Graph Save and Restore - Highly optimized for x86 16-bit image operation CPU and OpenCL for GPU - “Graph Optimizer” looks New Functionality at entire processing pipeline and removes, Under Discussion replaces, merges functions to improve NNEF Import performance and bandwidth Programmable user - Scripting for rapid kernels with prototyping, without re- accelerator offload compiling, at production performance levels http://gpuopen.com/compute- Evolving product/amd-openvx/ Functionality Multiple Shipping Tool s Implementations

© Copyright Khronos Group 2017 - Page 14 OpenVX 1.2 and Neural Net Extension • Convolution Neural Network topologies can be represented as OpenVX graphs - Layers are represented as OpenVX nodes - Layers connected by multi-dimensional tensors objects - Layer types include convolution, activation, pooling, fully-connected, soft-max - CNN nodes can be mixed with traditional vision nodes • Import/Export Extension - Efficient handling of network Weights/Biases or complete networks • OpenVX will be able to import NNEF files into OpenVX Neural Nets

Vision Node Native Vision Vision Downstream Camera Application Node Node Control CNN Nodes Processing

An OpenVX graph mixing CNN nodes with traditional vision nodes

New Generation APIs for safety Experience and Guidelines certifiable vision, graphics and compute OpenGL SC 1.0 - 2005 OpenGL SC 2.0 - April 2016 e.g. ISO 26262 and DO-178B/C Fixed function graphics subset Shader programmable pipeline subset

OpenGL ES 1.0 - 2003 OpenGL ES 2.0 - 2007 Fixed function graphics Shader programmable pipeline

Khronos SCAP OpenVX SC 1.1 Released 1st May 2017 ‘Safety Critical Advisory Panel’ Guidelines for designing APIs that Vulkan SC being discussed Restricted “deployment” implementation Small driver size executes on the target hardware by reading ease system certification. Open to Khronos member AND Advanced functionality the binary format and executing the pre- industry experts. If interested to Graphics and compute compiled graphs join contact [email protected]

glTF enables portable, efficient, sophisticated 3D assets for all 3D applications

Now glTF 2.0 has PBR! Cool, portable materials Rendering API independence glTF – OpenGL Transmission Format Coming to Microsoft Office! Efficient transmission of 3D assets

Laugh Engine running on Vulkan https://github.com/jian-ru/laugh_engine

© Copyright Khronos Group 2017 - Page 17 Please Get Involved! • Information on all these standards at Khronos - www.khronos.org • Khronos is driving to new levels of community engagement - Specifications and resources increasingly open for industry input and feedback • Any company or organization is welcome to join Khronos - For a voice and a vote in any of these standards • If joining is not possible – ask about a Khronos Advisory Panel - Free of charge – enables design reviews, requirements and contributions • Still no cross-vendor camera APIs? - Is the time right for this to be a target for standardization? • Neil Trevett - [email protected] - @neilt3d