Khronos Fast Forward SIGGRAPH Asia Nov19

Fast Forward Neil Trevett Khronos President NVIDIA VP Developer Ecosystems [email protected]| @neilt3d November 2019

© Khronos® Group 2019 This work is licensed under a Creative Commons Attribution 4.0 International License © The Khronos® Group Inc. 2019 - Page 1 What are Open Standards? Interoperability Standards define an agreed communication protocol between two ‘entities’ Common products use 100s of open standards

Device to Downloaded Web content Wireless Networks to the Web Browser

Device to Camera App to Video and JP E G its Charger Photo Playback Applications

Internal Components to Games to other Internal Components 3D GPU Acceleration

Many Standard Defining Organizations (SDOs) Each has a focus area of expertise that gathers an effective quorum Each creates a safe space for cooperation

This work is licensed under a Creative Commons Attribution 4.0 International License © The Khronos® Group Inc. 2019 - Page 2 The Need for Interoperability Standards

Standards Grow Markets Standards Reduce Costs By reducing consumer confusion and By sharing development between increasing capabilities and usability many companies and driving volume

Standards Accelerate Standards Do Not Time to Market Stifle Innovation With well-proven testing Companies can compete on implementation and interoperability quality, performance, power etc. etc.

True OPEN Standards Are not controlled by a single company – but by the industry – typically through an SDO Well defined participation, governance and intellectual property frameworks

This work is licensed under a Creative Commons Attribution 4.0 International License © The Khronos® Group Inc. 2019 - Page 3 >150 Members ~ 40% US, 30% Europe, 30% Asia

Khronos is an open, non-profit, member-driven industry consortium developing royalty-free standards to harness the power of silicon acceleration for demanding graphics rendering and computationally intensive applications such as 3D Graphics, Virtual Reality, Augmented Reality, and Machine Learning

This work is licensed under a Creative Commons Attribution 4.0 International License © The Khronos® Group Inc. 2019 - Page 4 Khronos Asia Pacific Members

Khronos warmly welcomes Australian and Asian company participation!!

This work is licensed under a Creative Commons Attribution 4.0 International License © The Khronos® Group Inc. 2019 - Page 5 Khronos Active Initiatives

3D Graphics 3D Assets Portable XR Parallel Computation Desktop, Mobile, Web Authoring Augmented and Vision, Inferencing, Embedded and Safety Critical and Delivery Virtual Reality Machine Learning

3D Commerce

Guidelines for creating APIs to streamline system safety certification

Exploratory Groups Heterogeneous Communications Rendering for scientific between offload compute devices Making High-Level Languages more visualization and data analytics effective at acceleration offload

This work is licensed under a Creative Commons Attribution 4.0 International License © The Khronos® Group Inc. 2019 - Page 6 Vulkan for Direct GPU Control

Simpler drivers - application has the best knowledge for holistic optimization – no ‘driver magic’ Complex drivers Application cause overhead Single thread per context Explicit creation of API objects and inconsistent Application before usage – efficient, behavior across Memory allocation Multiple Front-end predictable execution vendors Thread management Explicit Synchronization Compilers Easier portability - no fighting High-level Driver GLSL, HLSL etc. Always active Multi-threaded generation with different vendor heuristics of command buffers error handling Abstraction Layered GPU Control Validation and debug layers Full GLSL Context management SPIR-V loaded only when needed preprocessor and Memory allocation pre-compiled shaders compiler in Full GLSL compiler SPIR-V intermediate language: Error detection driver Thin Driver Loadable debug and shading language flexibility Explicit GPU Control validation layers OpenGL vs. Unified API across mobile and OpenGL ES desktop platforms GPU GPU Multiple graphics, command and DMA queues A Graphics API A GPU API

This work is licensed under a Creative Commons Attribution 4.0 International License © The Khronos® Group Inc. 2019 - Page 7 Vulkan and New Generation GPU APIs

Modern architecture | Low overhead| Multi-thread friendly EXPLICIT GPU access for EFFICIENT, LOW-LATENCY, PREDICTABLE performance

Vulkan Porting Tools

Vulkan is a non-proprietary, royalty-free open standard Portable across multiple platforms – desktop, mobile and embedded

This work is licensed under a Creative Commons Attribution 4.0 International License © The Khronos® Group Inc. 2019 - Page 8 Pervasive Vulkan Major GPU Companies supporting Vulkan for Desktop and Mobile Platforms

http://vulkan.gpuinfo.org/ Platforms

Apple Desktop Android (via porting Media Players Consoles (Android 7.0+) layers) (Vulkan 1.1 required on Android Q)

Virtual Reality Cloud Services Game Streaming Embedded

Game Engines

Croteam Serious Engine

This work is licensed under a Creative Commons Attribution 4.0 International License © The Khronos® Group Inc. 2019 - Page 9 Vulkan AAA Content Shipping on Desktop… Vulkan-only AAA Titles on PC

AAA titles on Linux Titles on PC AND macOS

This work is licensed under a Creative Commons Attribution 4.0 International License © The Khronos® Group Inc. 2019 - Page 10 …and Mobile

Plus…. Lineage 2 Revolution Heroes of Incredible Tales Dream League Soccer…

This work is licensed under a Creative Commons Attribution 4.0 International License © The Khronos® Group Inc. 2019 - Page 11 Vulkan 1.1 Ecosystem Evolution Strengthening Tools and Compilers Improved developer tools (SDK, validation/debug layers) Shader toolchain improvements (size, speed, robustness) Shading language flexibility – HLSL and OpenCL C support More rigorous conformance testing Building Vulkan’s Future Listen and prioritize developer needs Drive GPU technology

Released Vulkan 1.1 Extensions Vulkan 1.0 Extensions Reduced precision arithmetic types in shaders Maintenance updates plus additional functionality Bindless resources Multiview HLSL-compatible memory layouts Multi-GPU Formal memory model Enhanced Windows System Integration Buffer references Increased Shader Flexibility: Timeline semaphores 16-bit storage, Variable Pointers OpenGL-class lines and Interop Enhanced Cross-Process and https://www.khronos.org/registry/vulkan/specs/1.1-khr- February 2016 Cross-API Sharing March 2018 extensions/html/vkspec.html#extension-appendices-list Vulkan 1.0 Vulkan 1.1 Roadmap Discussions Integration of 1.0 Extensions Machine Learning acceleration plus new functionality Ray Tracing e.g. Subgroup Operations Video encode / decode Generalized subgroup operations Widening Platform Support Pervasive GPU vendor native driver availability Open source drivers – ANV (Intel), AMDVLK/RADV (AMD) Vulkan Portability to macOS/iOS and DX12

This work is licensed under a Creative Commons Attribution 4.0 International License © The Khronos® Group Inc. 2019 - Page 12 OpenGL Vulkan Interop • Enables OpenGL applications to incrementally leverage Vulkan functionality - Shared explicit memory objects • Dassault Systèmes achieves interactive object space AO in CATIA, an OpenGL application - Using the NVIDIA Vulkan VKRay vendor extension for Ray Tracing - See the Demo at the NVIDIA booth

This work is licensed under a Creative Commons Attribution 4.0 International License © The Khronos® Group Inc. 2019 - Page 13 Key Vulkan Online Open Source Resources

Vulkan Samples Vulkan Guide Collection of samples and resources to aid developing Help for developers to get up and going with the world of optimized Vulkan applications Vulkan with kinks to many other useful resources https://github.com/KhronosGroup/Vulkan-Samples https://github.com/KhronosGroup/Vulkan-Guide

RenderDoc Debugger Single-frame capture and detailed Vulkan SDK with Development/Debug Layers introspection of any application Windows, Linux - Ubuntu packages, Linux- Tarball, macOS https://renderdoc.org/ www.vulkan.lunarg.com

This work is licensed under a Creative Commons Attribution 4.0 International License © The Khronos® Group Inc. 2019 - Page 14 SPIR-V Ecosystem SPIR-V Khronos-defined cross-API IR Native graphics and parallel compute support GLSL HLSL Easily parsed/extended 32-bit stream Data object/control flow retained for effective code generation/translation Third party kernel and shader languages glslang DXC MSL C++ for HLSL SYCL for OpenCL in SPIRV-Cross OpenCL C OpenCL C++ ISO C++ clang GLSL Front-end Front-end Front-end Front-end

SPIR-V (Dis)Assembler clspv

SPIR-V Validator Khronos cooperating LLVM to SPIR-V with clang/LLVM Bi-directional LLVM Community Translators SPIRV-opt | SPIRV-remap Optimization Tools 3rd Party-hosted Open Source Projects

Environment spec for Khronos-hosted each target API usedIHV to Driver Open Source Projects drive compilationRuntimes https://github.com/KhronosGroup/SPIRV-Tools

This work is licensed under a Creative Commons Attribution 4.0 International License © The Khronos® Group Inc. 2019 - Page 15 Open Source Layering Projects Breaking through platform fragmentation

Vulkan added OpenGL- Vulkan adding more compute for fuller Vulkan added extensions style line extension support for OpenCL kernel deployment to ease layering of DX

Layers Vulkan OpenGL OpenCL OpenGL ES DX12 DX9-11 Over Vulkan is an effective porting clspv GLOVE DXVK layer for app Vulkan Zink clvk Angle vkd3d D9VK portability and stack simplification gfx-rs OpenGL Ashes Angle Vulkan Portability enables multi- DX12 gfx-rs vendor layered subsets to be gfx-rs queryable and DX9-11 Angle conformant Ashes

MoltenVK Growing interest to offset Metal gfx-rs Apple deprecation? Angle

Demand for Vulkan Need for consistent everywhere, even if no OpenGL ES everywhere, native drivers on platform primarily for WebGL This work is licensed under a Creative Commons Attribution 4.0 International License © The Khronos® Group Inc. 2019 - Page 16 Vulkan Portability Initiative

Enabling Vulkan applications on platforms without native drivers by layering cleanly queryable subsets of Vulkan over DX12, Metal and other APIs

Multiple Layered Vulkan Implementations Additional open source run-times over additional backends E.g. gfx-rs for Vulkan over Metal and DX12 - useful for Vulkan on UWP platforms such as Windows 10 S, Polaris, Xbox One. Secondary backends include OpenGL/D3D11 https://github.com/gfx-rs/gfx https://github.com/gfx-rs/portability

Portability Extension Layered implementations can portably expose what Vulkan functionality is not supported TODAY Open source tools, SDKs and libraries to Extend Vulkan Conformance Test Suite bring Vulkan 1.0 applications to Apple To handle layered implementations – what is present must work! using Metal Enhanced Vulkan Layers Extend DevSim/Validation Layers to flag or simulate queries for features not present

This work is licensed under a Creative Commons Attribution 4.0 International License © The Khronos® Group Inc. 2019 - Page 17 Vulkan Portability Initiative on Apple

Almost all mandatory Vulkan 1.0 functionality is supported: No Triangle Fans No separate stencil reference masks Open source SDK to build, Selected Optional Features and Applications run, and debug applications Extensions are added as required - on macOS - including driven by industry input and feedback validation layer support Robust buffer access https://vulkan.lunarg.com/ BC texture compressed formats Fragment shader atomics Tesselation Vulkan https://github.com/KhronosGroup/MoltenVK macOS SDK

SPIRV-Cross macOS / iOS Open source beta release for macOS Convert SPIR-V shaders to Run-time Metal Shaders Maps Vulkan to Metal Open source for MacOS and iOS MoltenVK supports Free to use - no fees or royalties macOS 10.11 / iOS 9.0 and up including commercial apps

This work is licensed under a Creative Commons Attribution 4.0 International License © The Khronos® Group Inc. 2019 - Page 18 Vulkan Apps Shipping On Apple

Multiple iOS and macOS Production Dota 2 on Mac RPCS3 PlayStation apps shipping e.g. Ships – up to 50% more 3 Emulator on Mac perf than Apple’s OpenGL Forsaken Remastered Dolphin Google Filament PBR First iOS Apps using Artifact from GameCube and Artifact from Renderer on Mac MoltenVK ship through Steam ships on Wii Emulator Steam ships on app store MoltenVK on working on MacOS MoltenVK on Initial ports of DX macOS - first macOS - second games in progress using Qt Running on Mac Vulkan-only Valve Diligent Engine Vulkan-only Valve Vulkan on Mac through MoltenVK app on Mac runs on MacOS app on Mac June September November January June 2018 2018 2018 2019 2019

This work is licensed under a Creative Commons Attribution 4.0 International License © The Khronos® Group Inc. 2019 - Page 19 Running DX Games on Linux Over Vulkan • DXVK – Direct3D 10/11 emulator running over Vulkan - Open source on GitHub - developed by Philip Rebohle with support from Valve • Vulkan has added multiple extensions to support efficient layering of D3D - Removing impedance mismatches between the two APIs • DXVK, Wine Windows Compatibility Layer and Valve Proton tool https://www.protondb.com - Enable thousands of PC games on Linux

Extensions created in response to DXVK issues VK_EXT_transform_feedback VK_EXT_depth_clip_enable VK_EXT_host_query_reset VK_EXT_texel_buffer_alignment VK_EXT_shader_demote_to_helper_invocation Other extensions used by DXVK VK_EXT_conditional_rendering VK_EXT_memory_budget VK_EXT_memory_priority VK_EXT_shader_viewport_index_layer VK_EXT_vertex_attribute_divisor VK_KHR_draw_indirect_count VK_KHR_shader_draw_parameters

This work is licensed under a Creative Commons Attribution 4.0 International License © The Khronos® Group Inc. 2019 - Page 20 Khronos Active Initiatives

3D Graphics 3D Assets Portable XR Parallel Computation Desktop, Mobile, Web Authoring Augmented and Vision, Inferencing, Embedded and Safety Critical and Delivery Virtual Reality Machine Learning

3D Commerce

Guidelines for creating APIs to streamline system safety certification

This work is licensed under a Creative Commons Attribution 4.0 International License © The Khronos® Group Inc. 2019 - Page 21 Analytic Rendering Exploratory Group

Analytic Rendering is image generation performed primarily to gain and communicate insights into complex data sets primarily for scientific visualization and data analytics Is there a need for a cross-platform open standard API?

Visualization Apps and Engines have Cross-vendor API to provide access to state- to be ported to multiple APIs of-the-art rendering across multiple platforms

This work is licensed under a Creative Commons Attribution 4.0 International License © The Khronos® Group Inc. 2019 - Page 22 Potential Analytic Rendering API Design

Rather than specifying the details of the rendering process, an Analytic Rendering API would enable a visualization application to simply describe the relationship between objects in a scene to be rendered and leave the details of the rendering process to a backend renderer

Khronos Exploratory Groups discuss the need for a new standard with no cost or IP Implications Open to all – even non-members - more details https://www.khronos.org/exploratory/analytic-rendering/ Some Initial Exploratory Group Members

This work is licensed under a Creative Commons Attribution 4.0 International License © The Khronos® Group Inc. 2019 - Page 23 Khronos Active Initiatives

3D Graphics 3D Assets Portable XR Parallel Computation Desktop, Mobile, Web Authoring Augmented and Vision, Inferencing, Embedded and Safety Critical and Delivery Virtual Reality Machine Learning

3D Commerce

Guidelines for creating APIs to streamline system safety certification

This work is licensed under a Creative Commons Attribution 4.0 International License © The Khronos® Group Inc. 2019 - Page 24 XR = AR + VR

OpenXR provides cross-platform, high-performance access to AR and VR platforms and devices Virtual Reality V A Augmented Reality

This work is licensed under a Creative Commons Attribution 4.0 International License © The Khronos® Group Inc. 2019 - Page 25 OpenXR – Solving XR Fragmentation

* OpenXR 1.0 is focused on enabling cross-platform applications. Optional device plugin interface will be supported post V1.0 ** Check OpenXR Landing Page for exact availabiliy of OpenXR in shipping run-times and devices www.khronos.org/openxr This work is licensed under a Creative Commons Attribution 4.0 International License © The Khronos® Group Inc. 2019 - Page 26 OpenXR is used with a 3D API

Application or Engine

Display, High-performance, low-latency composition and Cross-platform access to XR 3D rendering and composition* optical correction HMDs and sensors Multiview parameters XR application lifecycle Context priority Input device discovery and events Front buffer rendering Sensor tracking and pose calculation Tiled rendering (beam racing) Frame timing and display composition Variable rate rendering Haptics Control

* OpenXR can be used with other 3D APIs such as Direct3D, OpenGL and OpenGL ES

This work is licensed under a Creative Commons Attribution 4.0 International License © The Khronos® Group Inc. 2019 - Page 27 Companies Publicly Supporting OpenXR

OpenXR is a collaborative design Integrating many lessons from proprietary ‘first-generation’ XR API designs

This work is licensed under a Creative Commons Attribution 4.0 International License © The Khronos® Group Inc. 2019 - Page 28 OpenXR Win-Win-Win XR End-Users Can run the apps they want on their system – reducing market confusion and increasing consumer confidence

XR Vendors Can bring more applications onto their platform by leveraging the OpenXR content ecosystem

XR ISVs Can easily ship on more platforms for increased market reach

This work is licensed under a Creative Commons Attribution 4.0 International License © The Khronos® Group Inc. 2019 - Page 29 OpenXR 1.0 Released July 2019!

Significant community feedback – thank you! Improved OpenXR input subsystem, game engine editor support, loader …

Ratify and Provisional Finalize Conformance Release OpenXR Specification Test Suite Enable Officially Conformant GDC, March 2019 1.0 Specification SIGGRAPH, July 2019 Implementations

OpenXR runtime for Windows Mixed Reality headsets OpenXR support and HoloLens 2 from Microsoft shipping TODAY for Oculus Rift and Oculus Quest PLUS extensions to support HoloLens 2 hand tracking, eye tracking, spatial Coming soon mapping and spatial anchors by end of year

OpenXR 1.0 plugin for Many more Unreal Engine 4.23 coming ‘Monado’ OpenXR open source implementation Starting with preview 4 from Collabora shipping TODAY

This work is licensed under a Creative Commons Attribution 4.0 International License © The Khronos® Group Inc. 2019 - Page 30 Bringing XR to the Web

Native XR Apps The Web will Evolve into the Metaverse Web XR Apps

Lifting OpenXR functionality into the Web stack WebXR Native 3D Web 3D Engines Engines Close cooperation between WebXR and OpenXR

Khronos provides the foundation for native and Web-based 3D/XR

This work is licensed under a Creative Commons Attribution 4.0 International License © The Khronos® Group Inc. 2019 - Page 31 MEC (Multi-access Edge Computing) Server XR and 5G 1. Processes sensor data, including machine learning for environmental lighting, occlusion, scene semantics, object reconstruction and UI Leveraging High Bandwidth and Low Latency 2. Generates imagery from 3D models, including stereo, foveal rendering, ray-tracing, optics pre-distortion, varifocal processing

Rich Sensor Data Location–aware Content Requests

Sensor handling

OpenXR APIs can hide the 5G round trip from applications

Image: Nreal

Wireless mobile device with display and sensors NVIDIA EGX Display composition Apps 3D Assets Generated Augmentations & Any needed assets loaded from the Scenes cloud to edge server

This work is licensed under a Creative Commons Attribution 4.0 International License © The Khronos® Group Inc. 2019 - Page 32 glTF Real-time 3D Asset Transmission

glTF is an efficient, reliable run-time 3D transmission format with advanced photorealistic functionality

Functionality in Development Draco Mesh Compression Compact to Transmit Universal Compressed Textures Simple and Fast to Load Second generation PBR Describes Full Scenes (absorption/attenuation, clear coat, subsurface scattering, anisotropy) Runtime Neutral Subdivision surfaces Open and Extensible glTF 2.0 – June 2017 Physically Based Rendering

This work is licensed under a Creative Commons Attribution 4.0 International License © The Khronos® Group Inc. 2019 - Page 33 glTF 2.0 Scene Description Structure

.gltf (JSON) Node hierarchy, PBR material textures, cameras

.bin .png Geometry: vertices and indices .jpg Animation: key-frames .ktx2 Skins: inverse-bind matrices Textures

Mandatory Metallic-Roughness Materials

Optional Specular-Glossiness Materials

Geometry Texture based PBR materials

This work is licensed under a Creative Commons Attribution 4.0 International License © The Khronos® Group Inc. 2019 - Page 34 Titania Paint 3D

Modo Dedicated 3D Authoring Tools Discover Repositories Game Engines

Authoring Tools that Export 3D Web Engines

Oculus Create Experience Tools Apps / Engines Ecosystem 3D Builder VR / AR Authoring Tools Prep for 3D printing Sony 3D Creator Drive Demand Apps and Engines Users 3D Live Object 3D Scanning Tools gltfpack FBX2glTF Mixed Reality OBJ2GLTF Viewer

Collada2gltf Windows Mixed Convertors and Optimizers Reality Home

glTF Reference Viewer VR / AR Apps and Engines gltf-vscode glTF-asset-generator glTF-validator glTF-Toolkit Validation and Reference Tools Productivity and Social Apps

This work is licensed under a Creative Commons Attribution 4.0 International License © The Khronos® Group Inc. 2019 - Page 35 glTF Ecosystem Evolution

Striving for native glTF import and Tools! Export from every tool. Catalyzed Blender IO as exemplar

glTF 2.0 import/export with Blender 2.80

Avoid dialects at all costs! Sample viewer and Asset Validator in open source. Consistency! Sample models and asset generator for unit tests Sample Viewer for accurate Ground Truth glTF renderings glTF 2.0 – June 2017

Balancing functionality versus complexity. Functionality! glTF is extensible – only bring widely adopted glTF Mesh compression extension extensions into core provides up to 25x geometry compaction

glTF/Draco-enabled apps and engines This work is licensed under a Creative Commons Attribution 4.0 International License © The Khronos® Group Inc. 2019 - Page 36 Draco glTF Mesh Compression Extension • Library for compressing and decompressing 3D geometric meshes and point clouds - Draco designed and built for compression efficiency and speed - great fit with glTF! - https://github.com/google/draco • Draco glTF extension launched in February 2018 - https://github.com/KhronosGroup/glTF/blob/master/extensions/2.0/Khronos/KHR_draco_mesh_compression/README.md • Google has released Draco encoders and decoders in open source - C++ source code encoder to compress 3D data - C++ and JavaScript decoders for the encoded data - https://github.com/google/draco/tree/gltf_2.0_draco_extension • glTF/Draco compression already in use - Blender, three.js, BABYLON.JS, Adobe Dimension, glTF pipeline, FBX2glTF, AMD Compressonator and glTF sample models

Mesh Compression Ratios

This work is licensed under a Creative Commons Attribution 4.0 International License © The Khronos® Group Inc. 2019 - Page 37 Universal Textures for glTF • Fragmentation of GPU texture formats is significant issue for developers - Binomial’s ‘Basis Universal’ technology enables JPEG-sized texture assets - Transcodable on-the-fly to natively supported compressed GPU formats • glTF Universal Texture extension uses KTX2 subset as a flexible container - Precisely defined for consistent, cross-vendor generation and validation - Wide range of (un)(super)compressed texture formats used in Vulkan/DirectX/Metal - Supports streaming and full random access to MIP levels - Open source tools to create, transcode and upload to WebGL, OpenGL and Vulkan - https://github.com/KhronosGroup/KTX-Software/tree/ktx2

Transcode on-the-fly to a Encoding decoupled from natively supported compressed target device. GPU formats GPU-accelerated One encode pass per Desktop: BC1-5, BC7 Universal Texture Original texture asset Mobile: ETC1/2, PVRTC1, *ASTC TextureOriginal Textures Original Encode and Transcode AssetsTexture KTX2 Container GPU-accelerated AssetsTexture Supercompress to GPU formats Texture Assets (.png) with Basis ‘toktx’ Universal - ‘libktx’ compressed OSS Tool OSS Tool GPU-accelerated payload Texture *ASTC support in development

This work is licensed under a Creative Commons Attribution 4.0 International License © The Khronos® Group Inc. 2019 - Page 38 Universal Textures: Compression Ratios

Bytes 14,000,000 12,582,912 File Size GPU Size

10,500,000

7,000,000

3,500,000 2,778,518 2,097,152

315,619 232,104 0 PNG FlightHelmet_baseColor JPEG ETC1S 2048 x 2048, RGB JPEG must be fully Universal textures can be decompressed directly transcoded to into GPU memory compressed GPU textures Uncompressed Basis Universal

This work is licensed under a Creative Commons Attribution 4.0 International License © The Khronos® Group Inc. 2019 - Page 39 KTX2 and .basis files

Two complementary container formats for Basis Universal assets Binomial and Google open sourced ‘Basis Universal’ compressor and transcoder C++ or WebAssembly code for handling ‘.basis’ format textures in native apps and web sites https://github.com/binomialLLC/basis_universal Fine if you are in full control of your texture assets and rendering ‘Basis Universal’ texture compression technology Enables JPG-sized textures that Binomial’s ‘Basis Universal’ can be transcoded on-the-fly to technology contributed to glTF natively supported compressed Rigorously-defined KTX2 container format supports wide range GPU formats of texture formats used in Vulkan/DirectX/Metal with streaming and full random access to MIP levels glTF extension uses KTX2 subset with supercompressed payload using Basis Universal Technology Great for cross-vendor distribution of textures to multiple applications and engines

This work is licensed under a Creative Commons Attribution 4.0 International License © The Khronos® Group Inc. 2019 - Page 40 Next Generation glTF PBR Materials • Demand for advanced PBR for photorealistic assets - Beyond current ‘Metallic-Roughness’ and ‘Specular-Glossiness’ - E.g. Absorption/attenuation, clear coat, subsurface scattering, anisotropy • Extending Metallic-Roughness parameters - Consistency and fallbacks for performance for any device • Inspiration from Dassault Systèmes Enterprise PBR Shading Model (DSPBR) - https://github.com/DassaultSystemes-Technology/EnterprisePBRShadingModel/tree/master/gltf_ext • Wide industry collaboration for compatibility - Dassault Systèmes Images from https://dassaultsystemes- - Google Filament technology.github.io/EnterprisePBRShadingModel/ - Microsoft BabylonJS - NVIDIA MDL - OTOY Octane

Join the GitHub Discussion! https://github.com/KhronosGroup/glTF/issues/1442

This work is licensed under a Creative Commons Attribution 4.0 International License © The Khronos® Group Inc. 2019 - Page 41 glTF Evolution

Seeking Requirements Extensions shipping Advanced Animation LOD and Streaming and in development Compressed Point Clouds Draco Mesh Compression Cross-asset linking The glTF Roadmap is Driven by Universal Compressed Textures Enhanced Metadata Second generation PBR Composability Developer Feedback (absorption/attenuation, clear coat, Instancing Join the GitHub Discussion! subsurface scattering, anisotropy) CAD/BIM model support https://github.com/KhronosGroup/glTF/i Subdivision surfaces Encryption and security ssues/1442 3D Printing

Baseline of significant industry adoption glTF 2.0 – June 2017 Nexgen glTF Native AND Web Apps Version and timing not yet decided Metallic-Roughness and Incorporate popular extensions Specular-Glossiness PBR (with fallbacks)

glTF 1.0 – December 2015 Primarily for WebGL Uses GLSL for materials

This work is licensed under a Creative Commons Attribution 4.0 International License © The Khronos® Group Inc. 2019 - Page 42 VRM Using glTF 2.0

Hiroki Omae - Unity

This work is licensed under a Creative Commons Attribution 4.0 International License © The Khronos® Group Inc. 2019 - Page 43 3D Commerce - The Opportunity

3D Commerce = E Commerce enhanced with the use of 3D Models on any platform – including VR and AR

Early Experience Shows Increased customer engagement! Strengthened brand loyalty! Deeper product understanding! =$$$! More online sales! Fewer returns!

IKEA catalog uses augmented reality to give a virtual So why is 3D Commerce preview of furniture in a room – August 2013 taking so long to become widespread?

This work is licensed under a Creative Commons Attribution 4.0 International License © The Khronos® Group Inc. 2019 - Page 44 3D Commerce – Today’s Reality

I wish I had high quality, Everyone defines their product I need the materials in realistic 3D models for virtual data for sizes and colors my 3D models to look promotional photoshoots! differently – nothing is consistent! completely realistic!

Products don’t come The green couch looks with 3D data – and I blue on some devices – can’t physically scan lots of product returns them all fast are expensive! enough! Many 3D products on CAD tools don’t let my ecommerce web- me easily generate site first appear upside the data I need for E down! I have to hand- Commerce! tune 1000s of models!

Complex retail pipeline with hundreds of companies and millions of products Many friction points: tooling, technical and commercial 3D Commerce can’t reach industrial scale so… Interoperability standards to the rescue!

This work is licensed under a Creative Commons Attribution 4.0 International License © The Khronos® Group Inc. 2019 - Page 45 Khronos 3D Commerce Initiative

Working Group Announced SIGGRAPH 2019

Creating specifications and guidelines to align the 3D asset workflow from product design through manufacturing and each stage of retail to end-user delivery platforms

Broad Industry Participation from tooling, retail, technology and platform companies

This work is licensed under a Creative Commons Attribution 4.0 International License © The Khronos® Group Inc. 2019 - Page 46 3D Commerce - Four Areas of Focus

Asset Creation Guidelines Product Configuration For tools and product designers to Universal product configurability create assets with consistent data to be data and guidelines on how to drive used through the 3D Commerce pipeline consistent product display

First Goals Industry cooperation to urgently develop solutions to address priority problem areas

Metadata Viewer Validation and Certification Structured metadata definitions and Test models, reference viewer, display analysis examples to consistently carry product tools and capability specifications to guarantee information through the retail pipeline a consistent and accurate end user experience

This work is licensed under a Creative Commons Attribution 4.0 International License © The Khronos® Group Inc. 2019 - Page 47 3D Commerce Khronos Synergy

Portable AR and VR Apps

3D Asset Format

High-performance cross-platform 3D graphics Khronos 3D Commerce Interactive 3D on the Web

Vision processing and inferencing for AR and scanning

This work is licensed under a Creative Commons Attribution 4.0 International License © The Khronos® Group Inc. 2019 - Page 48 Khronos Active Initiatives

3D Graphics 3D Assets Portable XR Parallel Computation Desktop, Mobile, Web Authoring Augmented and Vision, Inferencing, Embedded and Safety Critical and Delivery Virtual Reality Machine Learning

3D Commerce

Guidelines for creating APIs to streamline system safety certification

This work is licensed under a Creative Commons Attribution 4.0 International License © The Khronos® Group Inc. 2019 - Page 49 Khronos Open Standard Compute APIs

High-level APIs Import Single source C++ Programming Vision and Inferencing Trained Neural Network with Compute Acceleration Acceleration Exchange Format

Low-level

APIs GPU Rendering + Compute Acceleration Heterogeneous Compute Acceleration Increasing interest in parallel heterogonous compute acceleration GPU CPUCPUCPU GPUGPU to combat the ‘End of Moore’s Law’ FPGA DSP

Custom Hardware

This work is licensed under a Creative Commons Attribution 4.0 International License © The Khronos® Group Inc. 2019 - Page 50 SYCL Single Source C++ Parallel Programming

SYCL is ideal for accelerating Multiple SYCL libraries for E.g. complex ML frameworks can be larger C++-based engines and vision and inferencing directly compiled and accelerated applications SYCL-BLAS, SYCL-DNN, SYCL- Eigen, SYCL Parallel STL Single application source C++ templates and lambda file using STANDARD C++ functions separate host & device code

C++ Kernel Fusion can give better performance on complex apps and libs than hand-coding

Accelerated code passed into device OpenCL compilers

This work is licensed under a Creative Commons Attribution 4.0 International License © The Khronos® Group Inc. 2019 - Page 51 SYCL Implementations SYCL enables Khronos to influence ISO to enable standard C++ to (eventually) support heterogenous compute

Multiple Backend Support Coming SYCL beginning to be supported on low- level APIs in addition to OpenCL e.g. Vulkan and CUDA http://sycl.tech Intel Adoption Intel’s ‘One API’ Initiative uses SYCL https://newsroom.intel.com/news/intels-one-api-project- delivers-unified-programming-model-across-diverse- architectures/#gs.bydj6z

triSYCL LLVM/clang SYCL Compiler Open-source test-bed to Compiles C++-based SYCL ComputeCpp experiment with the HipSYCL source files into code for both Codeplay Software's v1.2.1 specification of the OpenCL SYCL 1.2.1 implementation CPU and a wide range of conformant implementation SYCL C++ layer and to give that builds upon NVIDIA compute accelerators available to download today feedback to Khronos CUDA/AMD HIP/ROCm

This work is licensed under a Creative Commons Attribution 4.0 International License © The Khronos® Group Inc. 2019 - Page 52 OpenVX Cross-Vendor Inferencing OpenVX A high-level graph-based abstraction for portable, efficient vision processing Optimized OpenVX drivers created and shipped by processor vendors Can be implemented on almost any hardware or processor Graph can contain vision processing and NN nodes – enables global optimizations Run-time graph execution can be almost completely autonomously from the host CPU

An OpenVX graph mixing CNN nodes with traditional vision nodes Vision Node Native Vision Vision Downstream Application Camera Node Control Node CNN Nodes Processing

NNEF Translator converts NNEF representation into OpenVX Node Graphs

Performance comparable to hand-optimized, non-portable code Real, complex applications on real, complex hardware Much lower development effort than hand-optimized code Hardware Implementations

This work is licensed under a Creative Commons Attribution 4.0 International License © The Khronos® Group Inc. 2019 - Page 53 OpenVX 1.3 Released October 2019

OpenVX 1.3 Feature Sets Enables deployment flexibility while avoiding fragmentation Implementations with one or more complete feature sets are conformant - Baseline Graph Infrastructure (enables other Feature Sets) - Default Vision Functions - Enhanced Vision Functions (introduced in OpenVX 1.2) Version 1.3 - Neural Network Inferencing (including tensor objects) - NNEF Kernel import (including tensor objects) - Binary Images - Safety Critical (reduced features for easier safety certification) https://www.khronos.org/registry/OpenVX/specs/1.3/html/OpenVX_Specification_1_3.html

Open Source Prototype OpenVX 1.3 Open source OpenVX 1.3 for Raspberry Pi Conformance Test Suite Raspberry Pi 3 Model B with Raspbian OS Finalization expected before the end of 2019 Automatic optimization of memory access patterns via tiling and chaining https://github.com/KhronosGroup/OpenVX-cts/tree/openvx_1.3 Highly optimized kernels leveraging multimedia instruction set Automatic parallelization for multicore CPUs and GPUs Automatic merging of common kernel sequences Open Source OpenVX Tutorial https://github.com/KhronosGroup/OpenVX-sample-impl/tree/openvx_1.3 and Code Samples https://github.com/rgiduthuri/openvx_tutorial

This work is licensed under a Creative Commons Attribution 4.0 International License © The Khronos® Group Inc. 2019 - Page 54 Extending OpenVX with Custom Nodes

OpenVX/OpenCL Interop Kernel/Graph Import • Provisional Extension • Provisional Extension • Enables custom OpenCL acceleration to be • Defines container for executable or IR code invoked from OpenVX User Kernels • Enables arbitrary code to be inserted as an • Memory objects can be mapped or copied OpenVX Node in a graph

Application OpenVX user-kernels can access command queue and cl_mem objects to asynchronously Fully asynchronous host-device schedule OpenCL kernel execution operations during data exchange

Copy or export OpenVX data objects cl_mem buffers into OpenVX cl_mem buffers data objects

OpenCL Command Queue

Runtime Map or copy OpenVX data objects into cl_mem buffers Runtime

OpenVX/OpenCL Interop

This work is licensed under a Creative Commons Attribution 4.0 International License © The Khronos® Group Inc. 2019 - Page 55 Neural Network Exchange Formats

Before - Training and Inferencing Fragmentation After - NN Training and Inferencing Interoperability

Training Framework 1 Inference Engine 1 Training Framework 1 Neural Inference Engine 1 Network Training Framework 2 Inference Engine 2 Training Framework 2 Inference Engine 2 Exchange Format Training Framework 3 Inference Engine 3 Training Framework 3 Inference Engine 3 Every Inferencing Engine needs a Common Optimization custom importer from every Framework and processing tools

Two Neural Network Exchange Format Initiatives

ONNX and NNEF Embedded Inferencing Import Training Interchange are Complementary ONNX moves quickly to track authoring Defined Specification Open Source Project framework updates NNEF provides a stable bridge from Multi-company Governance at Khronos Initiated by Facebook & Microsoft training into edge inferencing engines

Stability for hardware deployment Software stack flexibility

This work is licensed under a Creative Commons Attribution 4.0 International License © The Khronos® Group Inc. 2019 - Page 56 NNEF and ONNX Industry Support

NNEF V1.0 released in August 2018 ONNX 1.6 Released in September 2019 After positive industry feedback on Provisional Specification. Introduced support for Quantization Maintenance update issued in September 2019 ONNX Runtime being integrated with GPU inferencing Extensions to V1.0 released for expanded functionality engines such as NVIDIA TensorRT

NNEF Working Group Participants ONNX Supporters

This work is licensed under a Creative Commons Attribution 4.0 International License © The Khronos® Group Inc. 2019 - Page 57 NNEF Tools Ecosystem

Compound operations TensorFlow and Live captured by exporting TensorFlow Lite Imminent graph Python script Import/Export NNEF Model Zoo Now available on GitHub. Useful for checking that ingested NNEF produces ONNX Caffe and Import/Export Caffe2 acceptable results on target system Import/Export Files NNEF adopts a rigorous approach to design lifecycle Syntax OpenVX Especially important for safety-critical or Parser and Ingestion and mission-critical applications in automotive, Validator Execution industrial and infrastructure markets

NNEF open source projects hosted on Khronos NNEF GitHub repository under Apache 2.0 https://github.com/KhronosGroup/NNEF-Tools

This work is licensed under a Creative Commons Attribution 4.0 International License © The Khronos® Group Inc. 2019 - Page 58 Primary Machine Learning Compilers

TensorFlow Graph, Caffe, Keras, MXNet, TensorFlow Graph, MXNet, PaddlePaddle, PyTorch, ONNX Import Formats ONNX PyTorch Keras, ONNX

Front-end / IR NNVM / Relay IR nGraph / Stripe IR Glow Core / Glow IR XLA HLO

LLVM, TPU IR, XLA IR OpenCL, LLVM, OpenCL, OpenCL TensorFlow Lite / NNAPI Output CUDA, Metal LLVM, CUDA LLVM (inc. HW accel)

This work is licensed under a Creative Commons Attribution 4.0 International License © The Khronos® Group Inc. 2019 - Page 59 ML Compiler Steps

Embedded NN Compilers CEVA Deep Neural Network (CDNN) Cadence Xtensa Neural Network Compiler (XNNC)

Consistent Steps

1.Import Trained Network Description

2. Apply graph-level optimizations e.g. node fusion, node lowering and memory tiling

3. Decompose to primitive instructions and emit programs for accelerated run-times

Fast progress but still area of intense research If compiler optimizations are effective - hardware accelerator APIs can stay ‘simple’ and won’t need complex metacommands (combined primitive commands) like DirectML

This work is licensed under a Creative Commons Attribution 4.0 International License © The Khronos® Group Inc. 2019 - Page 60 OpenCL – Low-level Parallel Programing • Low-level programming of heterogeneous parallel compute resources - One code tree can be executed on CPUs, GPUs, DSPs and FPGA … • OpenCL C or C++ language to write kernel programs to execute on any compute device - Platform Layer API - to query, select and initialize compute devices - Runtime API - to build and execute kernels programs on multiple devices • The programmer gets to control: - What programs execute on what device - Where data is stored in various speed and size memories in the system - When programs are run, and what operations are dependent on earlier operations

GPU OpenCL CPU KernelOpenCL Kernel code DSP Runtime API CodeKernelOpenCL compiled for CPU loads and executes CPU CodeKernelOpenCL devices FPGA kernels across devices CodeKernel Devices Code Host

This work is licensed under a Creative Commons Attribution 4.0 International License © The Khronos® Group Inc. 2019 - Page 61 OpenCL is Widely Deployed and Used

Parallel Computation Languages

MATHLAB

Math and Physics Libraries

Hardware Implementations Vision and Imaging Libraries

Machine Learning Inferencing Compilers Modo

Huawei Intel clDNN Desktop Creative Apps Intel Synopsis MetaWare EV Android NNAPI SYCL-DNN CLBlast SYCL-BLAS Arm Compute Library

Linear Algebra Libraries TI Deep Learning Library (TIDL) VeriSilicon Machine Learning Libraries This work is licensed under a Creative Commons Attribution 4.0 International License © The Khronos® Group Inc. 2019 - Page 62 OpenCL Evolution

OpenCL Extension Specs Scratch-Pad Memory Management Vulkan / OpenCL Interop Integration of Extensions Extended Subgroups plus New Core functionality SPIR-V 1.4 ingestion for compiler efficiency SPIR-V Extended debug info Focus for OpenCL Next is ‘Deployment Flexibility’ Flexible Profile enables embedded vendors to ship targeted functionality for their customers and be officially conformant Regular Maintenance Updates May 2017 Regular updates for spec clarifications, Target 2020 OpenCL 2.2 formatting and bug fixes ‘OpenCL Next’ https://www.khronos.org/registry/OpenCL/

Repeat Cycle for next Core Specification

This work is licensed under a Creative Commons Attribution 4.0 International License © The Khronos® Group Inc. 2019 - Page 63 Deploying OpenCL C Over Vulkan • Clspv – Google’s experimental compiler for OpenCL C to Vulkan SPIR-V - Open source - tracks top-of-tree LLVM and clang, not a fork • Adobe Premiere Rush has 200K lines of OpenCL C kernel code - Professional-quality, cross-platform video capture and editing system - Now shipping on Android on Vulkan

OpenCL C OpenCL Source Host Code

Clspv Run-time Compiler API Translator Prototype open Prototype open source project Runtime source project https://github.com/google/clspv https://github.com/kpet/clvk

This work is licensed under a Creative Commons Attribution 4.0 International License © The Khronos® Group Inc. 2019 - Page 64 Khronos Active Initiatives

3D Graphics 3D Assets Portable XR Parallel Computation Desktop, Mobile, Web Authoring Augmented and Vision, Inferencing, Embedded and Safety Critical and Delivery Virtual Reality Machine Learning

3D Commerce

Guidelines for creating APIs to streamline system safety certification

This work is licensed under a Creative Commons Attribution 4.0 International License © The Khronos® Group Inc. 2019 - Page 65 Safety Critical GPU API Evolution

New Generation Safety Critical APIs for Graphics, Compute and Display OpenGL SC 1.0 - 2005 OpenGL SC 2.0 - April 2016 Fixed function graphics safety Programmable Shaders critical subset safety critical subset

OpenGL ES 1.0 - 2003 OpenGL ES 2.0 - 2007 Vulkan 1.0 - 2016 Fixed function graphics Programmable Shaders Explicit Graphics and Compute

Rendering Compute Display Clearly Definable Design Goals Vulkan is Compelling Starting to Adapt Vulkan for SC Reduce driver size and complexity Point for SC GPU API Design Industry Need -> Offline pipeline creation, no dynamic • Widely adopted, royalty-free display resolutions for GPU Acceleration APIs open standard Deterministic Behavior designed to ease system safety • Low-level explicit API – smaller -> No ignored parameters, static memory certification is increasing surface area than OpenGL management, eliminate undefined ISO 26262 / ASIL-D • Not burdened by debug behaviors functionality Robust Error Handling • Very little internal state -> Error callbacks so app can respond, • Well-defined thread behavior Fatal error callbacks for fast recovery initiation Potential OpenCL SC work will leverage C API - MISRA C Compliance the deployment flexibility of ‘OpenCL Next’ to minimize API surface area Khronos Vulkan SC Working Group started work in February 2019

This work is licensed under a Creative Commons Attribution 4.0 International License © The Khronos® Group Inc. 2019 - Page 66 Khronos Standards Immersive Computing

Download 3D object and scene data Vision and sensor processing - including neural network inferencing for machine learning

High-performance, low-latency 3D Graphics

Portable interaction with VR/AR sensor, haptic and display devices

This work is licensed under a Creative Commons Attribution 4.0 International License © The Khronos® Group Inc. 2019 - Page 67 Khronos Proven Process and Organization

Open membership. Open Source Any company is welcome to join. Conformance Tests and One company one vote Software Adopters Programs

Open specifications. Non-profit organization - ROYALTY-FREE through a strong, Membership and Adopters modern IP Framework fees cover expenses

Any member, or non- Invest where strong industry member, can propose Silicon momentum and relevance – let new standards Darwinism rule! initiatives

This work is licensed under a Creative Commons Attribution 4.0 International License © The Khronos® Group Inc. 2019 - Page 68 Khronos Ecosystem Engagement

$0 Open to all! https://community.khronos.org/ www.khr.io/slack

$0 Spec fixes and suggestions made under the Khronos Forums Khronos IP Framework. Open source contributions under repo’s CLA – typically Apache 2.0 and Slack Channels https://github.com/KhronosGroup

Contribute to open source specs, $0 Advisors under the Khronos NDA and IP Framework can comment and contribute CTS, tools and ecosystem to requirements and draft specifications https://www.khronos.org/advisors/ Advisory Panels $ Khronos members under Khronos NDA and IP Framework participate and vote in working group meetings Working https://www.khronos.org/members/ Groups

This work is licensed under a Creative Commons Attribution 4.0 International License © The Khronos® Group Inc. 2019 - Page 69 Benefits of Khronos membership

Gain early insights Influence the design and Accelerate your time-to- into industry trends direction of key open standards market with early access and directions that will drive your business to specification drafts

Gather industry Draft Specifications Publicly Release requirements for Confidential to Specifications and future open standards Khronos members Conformance Tests

Network with domain State-of-the-art IP Enhance your company experts from diverse Framework protects your reputation as an industry leader companies in your industry Intellectual Property through Khronos participation

This work is licensed under a Creative Commons Attribution 4.0 International License © The Khronos® Group Inc. 2019 - Page 70 Thank You and Resources • Khronos is creating cutting-edge royalty-free open standards - For 3D, compute, inferencing gaming • These slides and information on Khronos Standards - www.khronos.org • Any company is welcome to join Khronos - https://www.khronos.org/members/ - We warmly welcome members from Australia and Asia • Dedicated developer resources - Khronos Developer Forum: https://community.khronos.org/ - Khronos Developer Slack Channel: www.khr.io/slack WeChat: neiltrevett • We are happy to help answer any questions! - Neil Trevett, Khronos President: [email protected], @neilt3d - Khronos Developer Relations, Kris Rose: [email protected], @kristoferrose

This work is licensed under a Creative Commons Attribution 4.0 International License © The Khronos® Group Inc. 2019 - Page 71