Open Standards for Cross-Platform Gaming, Virtual & Augmented Reality

Open Standards for Cross-Platform Gaming, Virtual & Augmented Reality Casual Connect 2018 Neil Trevett | Khronos President NVIDIA | VP Developer Ecosystem [email protected] | @neilt3d www.khronos.org

Software

Silicon

Khronos is an International Industry Consortium of over 100 companies creating royalty-free, open standard APIs to enable software to access hardware acceleration for 3D graphics, Virtual and Augmented Reality, Parallel Computing, Neural Networks and Vision Processing Copyright ©2018 The Khronos™ Group Inc. - Page 2 Khronos Open APIs for Gaming and VR/AR

Download 3D augmentation object and scene data

Vision and sensor Generate Low Latency 3D processing - Content and Simulations including neural networks

Khronos 3D APIs are on almost Interact with sensor, haptic every Workstation, Desktop, and VR/AR display devices Phone, Devices and Web Browser in the world = Billions of seats

Vulkan Provides Clean, modern architecture | Low overhead, explicit GPU access Portable across desktop and mobile | Multi-thread / multi-core friendly Primary 3D API on Android Platform Efficient, low-latency, predictable performance Access to multiple platforms

Rich Desktop VR Functionality OpenGL 4.6 released August 2017 High-performance, Low-latency VR/AR RENDERING Vulkan 1.0 released February 2016 Available on diverse platforms + 3D on millions of mobile VR devices OpenGL ES 3.2 released August 2015

Cross-Platform, Portable VR/AR DEVICE MANAGEMENT Device discovery Movement tracking Powering WebVR in browsers Input and haptics Flexible graphics configuration WebGL 2.0 Released February 2017

Cross-Platform, Portable VR/AR DEVICE MANAGEMENT Device discovery | Movement tracking | Input and haptics Flexible graphics configuration

VR VR VR VR VR VR VR VR App App App App App App App App 1 2 3 4 Device Discovery 1 2 3 4 Device Events

Proprietary Sensor tracking Proprietary Engine Haptics Engine HMD Parameters

Application Interface

Device Layer

Installable VR VR VR VR VR Drivers VR VR VR VR VR Device Device Device Device Device Device Device Device Device Device 1 2 3 4 5 1 2 3 4 5 Before OpenXR After OpenXR VR Market Wide Interoperability of Fragmentation VR Apps and Devices

Design work started in December 2016 Typically 12-18 months to develop a V1.0 specification

Access to any OpenXR Application

Any successful standard encourages and Proprietary APIs enables healthy industry competition Extensions Application Interface OpenXR will not replace VR run-times – or VR Run-times ‘outlaw’ existing interfaces OpenXR will simply provide cross-vendor

Proprietary APIs that can be exposed by a runtime to Driver Interfaces access more apps and devices Extensions Device Layer

Access to any OpenXR for portable OpenXR Device AR AND VR apps and devices with initial focus on VR

Resource management offloaded to app: low-overhead, low-latency Complex drivers Application driver cause overhead Single thread per context and inconsistent Application Consistent behavior: no ‘fighting with driver behavior across Memory allocation Multiple Front-end vendors Thread management heuristics’ Synchronization Compilers High-level Driver Always active Multi-threaded generation GLSL, HLSL etc. Validation and debug layers of command buffers error handling Abstraction loaded only when needed Layered GPU Control Full GLSL Context management SPIR-V SPIR-V intermediate preprocessor and Memory allocation pre-compiled shaders language: shading language compiler in Full GLSL compiler flexibility Error detection driver Thin Driver Loadable debug and Explicit GPU Control validation layers Multi-threaded command OpenGL vs. creation. Multiple graphics, OpenGL ES command and DMA queues GPU GPU Unified API across all platforms with feature set flexibility Vulkan 1.0 provides access to OpenGL ES 3.1 / OpenGL 4.X-class GPU functionality but with increased performance and flexibility

All Major GPU Companies shipping Vulkan Drivers for Desktop and Mobile Platforms

http://vulkan.gpuinfo.org/ Desktop, Mobile, Embedded and Console Platforms Supporting Vulkan Including phones and tablets from Google, Huawei, Samsung, Sony, Xiaomi - both premium and mid-range devices

Desktop Android 7.0+ Nintendo Switch Android TV Embedded Linux

VR Platforms

SteamVR GearVR Oculus Rift Google Daydream Game Engines

http://www.game-debate.com/news/22525/star-citizen-directx-11-and-12-support-axed-in-favour-of-vulkan-api http://www.game-debate.com/news/23312/up-for-debate-vulkan-vs-directx-12-which-is-the-better-graphics-api

Vulkan is Powering Mobile Gaming…

Publicly announced games as of November 2017 24th August 2017 #Vulkan Games = 35 #DX12 Games = 25 https://en.wikipedia.org/wiki/List_of_games_with_Vulkan_support

http://www.intrinsic-engine.com/ Many more mobile titles coming… http://www.pcgamer.com/ashes-of-the-singularity-escalation-v24-update-adds-vulkan-support/ Copyright ©2018 The Khronos™ Group Inc. - Page 12 Vulkan Evolution

Strengthened Ecosystem and SDK Enhanced developer and debugging tools Regression testing for SDK stability Enhanced Conformance Testing (API now has 198K test cases - up from 107K last year) Compiler robustness - including HLSL support Vulkan Roadmap Vulkan Extensions Maintenance updates plus additional functionality Regular Vulkan Core Releases Explicit Building Blocks for VR Integrates proven KHR extensions Explicit Building Blocks for Homogeneous Multi-GPU Enhanced Windows System Integration Increased Shader Language Flexibility – inc. HLSL Roadmap Discussions Enhanced Cross-Process and Cross-API Sharing Pushes forward the envelope of GPU Acceleration: Feb16 Enhanced Compute and Language Flexibility Vulkan 1.0 Optimized Vision and Inferencing Acceleration Ray Tracing Explicit Access to GPU Acceleration Widened Platform Support Through Vulkan Universal Portability Including Vulkan on macOS and iOS

Games Native 3D Browser Engines Apps Engines Games Native 3D Browser JavaScript and WebAssembly Native bindings Engines Apps Engines for ‘nexgen WebGL’

Vulkan Universally Portable Subset

Tools Layers Community Outreach at GDC 2017 MAP Vulkan to Metal and DX12 API Libraries Create a hybrid Portability API? Shader Translators

Feedback - AVOID CREATING A FOURTH API!!! Would need new specification, CTS, Documentation. Additional developer learning curve. A whole new specification to name, brand, promote. Would INCREASE industry fragmentation

Content downloaded Content from the Web JavaScript, HTML, CSS, ... Low-level WebGL API provide a Middleware provides accessibility powerful foundation for a rich for non-expert programmers JavaScript middleware JavaScript Middleware E.g. three.js library ecosystem

CSS Reliable WebGL Browser provides WebGL relies on work by 3D engine alongside other HTML5 both GPU and technologies - no plug-in required JavaScript HTML5 Browser Vendors -> Khronos has the right membership OS Provided Drivers to enable that WebGL uses OpenGL ES 2.0 or cooperation Angle for OpenGL ES 2.0 over DX9

32-bit integers and floats General Purpose Compute Programmable NPOT, 3D/depth textures Multi-threaded Rendering Vertex and Texture arrays Low-latency renderring fragment shaders Multiple Render Targets

2007 2012 2014 2016 ?? OpenGL ES 2.0 OpenGL ES 3.0 OpenGL ES 3.1 Apple Compute Shaders GPUWeb W3C Apple not shipping ES 3.1 Vulkan Community Group? Compute shader extension? 2011 Portability WebGL 1.0 ‘WebGL Next’? Pervasive OpenGL ES 2.0 OpenGL and OpenGL ES ship on March 2017 every desktop and mobile OS WebGL 2.0 Conformance Testing is vital for Cross-Platform Reliability WebGL 2.0 conformance tests are very thorough 10x more tests than WebGL 1.0 tests

http://caniuse.com/#feat=webgl

WebGL 2.0 brings Desktop-class graphics to the Web The time to create a new class of Web-based 3D Apps is now!

Application Native AR Apps 3D Web Apps VR Web Apps

Middleware

Browser

Native

All glTF spec development on open GitHub: https://github.com/KhronosGroup/glTF

OpenGL Transmission Format glTF 1.0 glTF 2.0 has PBR! Efficient transmission of 3D Aimed at loading assets into WebGL apps Cool, portable materials scenes and assets Uses GLSL for materials Rendering API independence Released December 2015 Released June 2017

glTF on View 3D over DX12 https://www.microsoft.com/en-us/store/p/windows-view-3d/9nblggh42ths glTF on WebGL Reference Implementation http://github.khronos.org/glTF-WebGL-PBR/

glTF Model on Laugh Engine over Vulkan glTF Model running on Metal PBR Renderer https://github.com/jian-ru/laugh_engine https://twitter.com/warrenm/status/891558755657175040 Copyright ©2018 The Khronos™ Group Inc. - Page 23 COLLADA and glTF 3D Asset Formats

COLLADA FOR AUTHORING INTERCHANGE glTF FOR RUN-TIME TRANSMISSION Retains extensive data to enable editable Compact file size and assets to be passed between authoring tools efficient processing/import

More tools …

Collada2gltf Translator

Many More…

Apps and Engines Based COLLADA is intended to be used for glTF can be generated from intermediate on any 3D API intermediate interchange, glTF is formats such as COLLADA, or exported directly designed for run-time delivery from authoring tools to fit individual use cases

Please let us know if your company logo is not here and you would like it added!

Publicly Stated Support for glTF

Khronos Board API Working Groups Advisory Panels By invitation, no fee. Strategy, budget and (Industry, Academic and Provide requirements oversight Associate members) $$ and draft spec feedback $ One vote per industry member

Industry and Adopters Conformance Royalty-Free Implementations, Tools Educator Guidelines Community $ Programs Tests Specifications Documentation, Samples Courseware Materials Feedback and contributions on released materials Open and royalty-free. Often open sourced

Adopters Developers Educators / Certifiers Build conformant Develop applications Create Courses implementation and products using the APIs Training and Certification Membership Fees Academic $1,000 Small companies $175/employee (min $3,500) Non-profits $7,500 Larger companies - $18,000

Copyright ©2018 The Khronos™ Group Inc. - Page 27 Invitation to Get Involved! • Khronos is creating cutting-edge, royalty-free, open API acceleration standards - Relevant and enabling for gaming, VR and AR • Join Khronos - ensure these standards meet your company’s needs - And the needs of the gaming industry • Use Khronos standards to bring cutting edge capabilities to your products - While reducing costs and accelerating time to market • Khronos encourages your participation - As a Member, Advisor or Adopter • More Information - [email protected] | @neilt3d | www.khronos.org - These slides will be posted on the Khronos website