AR Devices: Grand Challenges and How Open Standards Can Help

AR Devices: Grand Challenges and how open standards can help

Neil Trevett Khronos President, NVIDIA VP Developer Ecosystems [email protected] | @neilt3d

Spatial Computing Workshop

Real Augmented Virtual Reality Reality Reality

Mixed Reality is the same thing as Augmented Reality! Virtual Reality has already ‘emerged’ steady growth in Gaming LBE Training Medical Etc. See- Mobile Phone/Tablet through - based AR optics AR

Disruptive?

AR on today’s mobile phones good for demos and short experiences See-through AR may eventually replace today’s mobile phones

www.http://thearea.org/

Guided Task Data Visualization

Design Insights Remote Expert

Tablets and HoloLens already widely deployed in industrial and enterprise use cases Less sensitive to cost and social factors that may hold back consumer AR

Optics Resolution, brightness, field of view, focus accommodation, eyebox

Power Consumption Interoperability Acceleration API standards All day continuous Connectivity standards operation in small, socially Content standards acceptable form factor Content discoverability

Sensors Position tracking, occlusion, lighting, eye-tracking, scene semantics, UI

Wide Field Of View Varifocal Near-Eye Display Using See-Through Deformable Membrane Mirrors, Proc. of IEEE VR 2017 http://telepresence.web.unc.edu/files/2017/01/Dunn_2017_TVCG_MembraneAR.pdf

Real World Stereoscopic Display Focus accommodation needed: 1) Avoid fatigue/nausea 2) Realistic mixing of real and virtual imagery

HoloLens 2 – MWC 2019 Self-contained 1080P per eye 52 degree FOV 2-3 hour battery life Hand and voice UI $3,500

Nreal Light - CES 2019 Socially acceptable form factor - 85g Single USB-C wire to processor pack 1080P per eye 52 degree FOV How can we deliver Three hour battery life varifocal displays in a Google Glass 2 $999 $999 mobile device

Vuzix Blade Magic Leap One ? $999 Daqri Smart glasses $2,295 HUD Single Focal Plane AR Varifocal AR Monocular, no tracking Stereo, 6DOF positional tracking Focus Accommodation

Throwing massive compute at overcome physical constraints Create light field using dynamic holograph Spatial Light Modulator Reflect into eye using (static) see-through, holographic element With enough compute can adjust focal depth of every pixel Can also correct for optical path aberrations & vision correction

Uses LOTS of GPU compute cycles to enable compact size and wide-FOV Acceleratable with standard GPU APIs

Maimone, et al., “Holographic Near-Eye Displays for Virtual and Augmented Reality”, SIGGRAPH 2017 https://www.microsoft.com/en-us/research/wp-content/uploads/2017/05/holo_author.pdf

HoloLens 2 Most HMD Pixels are Peripheral Column1Magic Leap One

4% 20°

73% Foveal Peripheral Peripheral 96%

Eye Tracking iPhone 7 27" Desktop 2016 VR Reduces rendering load - foveal rendering Plus Monitor HMD Enables realistic focus cues Gaze based UI

Optics Resolution, brightness, field of view, focus accommodation, eyebox

Power Consumption Interoperability Acceleration API standards All day continuous Connectivity standards operation in small, socially Content standards acceptable form factor Content discoverability

Sensors Position tracking, occlusion, lighting, eye-tracking, scene semantics, UI

Solved Position tracking Machine Learning Eye-tracking needed as problems become ‘fuzzier’ 3D Reconstruction Environmental lighting Occlusion More compute cycles needed Real-time GPU-accelerated Lighting Calculations Scene semantics for inferencing High-Quality Reflections, Refractions, and Caustics in Augmented Reality and their Contribution to Visual Coherence P. Kán, H. Kaufmann, Institute of Software Technology and Interactive Systems, Natural UI Vienna University of Technology, Vienna, Austria Unsolved https://www.youtube.com/watch?v=i2MEwVZzDaA

https://dl.acm.org/citation.cfm?id=3275041 Environmental Occlusion © Khronos® Group Inc. 2019 - Page 10 See-Through AR Grand Challenges

Optics Resolution, brightness, field of view, focus accommodation, eyebox

Power Consumption Interoperability Acceleration API standards All day continuous Connectivity standards operation in small, socially Content standards acceptable form factor Content discoverability

Sensors Position tracking, occlusion, lighting, eye-tracking, scene semantics, UI

Offload Processors – such as GPUs - are FASTER AND more power EFFICIENT than CPUs

Everything runs on CPU

Silicon Acceleration

Pokémon Go AR processing runs Developers need graphics and compute APIs to on CPU efficiently access acceleration processors

© Khronos® Group Inc. 2019 - Page 12 Khronos Mission Khronos members are industry leaders from around the world that join to safely cooperate - to advance their own businesses and the industry as a whole

Software

Silicon

Khronos is an open, member-driven industry consortium developing royalty-free standards, and vibrant ecosystems, to harness the power of silicon acceleration for demanding graphics rendering and computationally intensive applications

XR-Related Features NOW Multi-GPU support Multiview Rendering Context priority Front buffer rendering

IN DISCUSSION Variable Rate Rendering Tiled rendering (beam racing) XR Devices

https://xkcd.com/1367/

Content downloaded Content from the Web JavaScript, HTML, CSS, ...

Low-level WebGL API provides a Middleware provides accessibility JavaScript Middleware powerful foundation for a rich for non-expert programmers JavaScript middleware E.g. three.js library ecosystem

CSS Reliable WebGL Browser provides WebGL relies on work by 3D engine alongside other HTML5 both GPU and technologies - no plug-in required JavaScript HTML5 Browser Vendors -> Khronos has the OS Provided Drivers right membership WebGL uses native OpenGL to enable that or OpenGL ES or cooperation Angle = OpenGL ES over DX

Many mobile SOCs and Embedded Systems becoming increasingly heterogeneous Autonomous vehicles, Vision and inferencing Augmented and Virtual Reality

Application

OpenCL Hardware Implementations

Huawei OpenCL provides a programming and Intel runtime framework for programming clDNN heterogeneous compute resources Intel Synopsis MetaWare EV Android NNAPI OpenCL C kernels are compiled and distributed across available processors SYCL-DNN under programmer control Arm Compute Library TI Deep Learning Library (TIDL) VeriSilicon OpenCL-accelerated Machine Learning Libraries CPU CPUCPU GPUGPU Heterogeneous FPGA DSP Compute OpenCL-accelerated Machine Learning Inferencing Compilers Resources Custom Hardware OpenCL-accelerated Vision and Imaging Libraries

© Khronos® Group Inc. 2019 - Page 18 Khronos OpenVX and NNEF for Inferencing OpenVX X100 High-level graph-based abstraction for Portable, Efficient Vision Processing Dedicated Hardware Optimized OpenVX drivers created and shipped by processor vendors Vision Can be implemented on almost any hardware or processor DSPs X10 Graph can contain vision processing and NN nodes – for global optimization GPU Compute Multi-core Power Efficiency Power X1 CPU Vision Computation Flexibility Node Downstream Native Vision Camera Vision Application Control Node CNN Nodes Node Processing

NNEF Translator converts NNEF representation into OpenVX Node Graphs

Performance comparable to hand-optimized, non-portable code Real, complex applications on real, complex hardware Much lower development effort than hand-optimized code Hardware Implementations

Optics Resolution, brightness, field of view, focus accommodation, eyebox

Power Consumption Interoperability Acceleration API standards All day continuous Connectivity standards operation in small, socially Content standards acceptable form factor Content discoverability

Sensors Position tracking, occlusion, lighting, eye-tracking, scene semantics, UI

Compact to Transmit Simple and Fast to Load Describes Full Scenes Runtime Neutral Open and Extensible glTF 2.0 – June 2017 Efficient, reliable transmission glTF 1.0 – December 2015 Native AND Web APIs Bring 3D assets into 1000s of Primarily for WebGL Physically Based Rendering apps and engines Uses GLSL for materials Metallic-Roughness and Specular-Glossiness

Modo Dedicated 3D Authoring Tools Game Engines

Discover Repositories Authoring Tools that Export 3D Web Engines

Oculus

3D Builder Prep for 3D printing VR / AR Authoring Tools Create Experience Tools Apps / Engines Sony 3D Creator Ecosystem 3D Apps and Engines 3D Live Object 3D Scanning Tools Drive Demand Users Mixed Reality Viewer Collada2gltf FBX2glTF OBJ2GLTF Windows Mixed Convertors and Optimizers Reality Home

glTF Reference Viewer VR / AR Apps and Engines gltf-vscode glTF-asset-generator glTF-validator glTF-Toolkit Validation and Reference Tools Productivity and Social Apps

Khronos 3D Commerce Exploratory Group Retail AND Technology companies making virtual 3D product representations possible on an industrial scale

Open to any company under NDA during exploratory phase https://www.khronos.org/exploratory/3d-commerce/

VR AR VR AR VR AR VR AR App App App App App App App App 1 2 3 4 1 2 3 4

Proprietary Proprietary WebXR Engine WebXR Engine

Application Interface

Device Integration Layer

VR VR VR VR VR VR VR VR VR VR Device Device Device Device Device Device Device Device Device Device 1 2 3 4 5 1 2 3 4 5 Before OpenXR After OpenXR XR Market Fragmentation Wide interoperability of XR apps and devices

OpenXR is a collaborative design Integrating many lessons from proprietary ‘first-generation’ XR API designs

Application or Engine

Display, High-performance, low-latency composition and Cross-platform access to XR 3D rendering and composition* optical correction HMDs and sensors Multiview parameters XR application lifecycle Context priority Frame timing and display composition Front buffer rendering Sensor tracking and pose calculation Tiled rendering (beam racing) Input device discovery and events Variable rate rendering Haptics Control

* OpenXR can be used with other 3D APIs such as Direct3D, OpenGL and OpenGL ES

XR Vendors Can bring more applications onto their platform by leveraging the OpenXR content ecosystem

XR ISVs Can easily ship on more platforms for increased market reach

Native XR Apps Web XR Apps Future versions of OpenXR will include cross-platform extended AR functionality

WebXR 3D Engines 3D Engines System-exposed AR Capabilities

Close ongoing collaboration between WebXR and OpenXR

Khronos providing the foundation for 3D and XR in the Web and native stacks

© Khronos® Group Inc. 2019 - Page 30 MEC (Multi-access Edge Computing) Server XR and 5G 1. Processes sensor data, including machine learning for environmental Leveraging High Bandwidth and Low Latency lighting, occlusion, scene semantics, object reconstruction and UI 2. Generates imagery from 3D models, including stereo, foveal rendering, ray-tracing, optics pre-distortion, varifocal processing

Rich Sensor Data Content Requests Location

Image: Nreal

Wireless mobile device with display and sensors MEC Server Apps 3D Assets Generated OpenXR APIs can hide Augmentations & the 5G round trip from applications Scenes © Khronos® Group Inc. 2019 - Page 31 Any Questions? • Opportunities for VR and AR are here today! - Mainly in enterprise today, disruptive consumer AR coming • Standards will be key to pervasive XR - Compute, XR and content interoperability • 5G can help accelerate XR deployment - Offload significant processing from mobile XR devices - Lightweight devices with high-bandwidth, low-latency link to Edge Computing • Khronos is creating cutting-edge royalty-free open standards - For 3D, VR/AR, Compute, Vision and machine learning - Any company welcome to join and help steer the direction of the industry • More Information - www.khronos.org - [email protected] - @neilt3d