Open Standards for Today's Gaming Industry

Open Standards for Today’s Gaming Industry

Erik Noreke VP of Business Development, Khronos Group

© Copyright Khronos Group 2013 - Page 1 The Topics for this Afternoon • Why do we need open standards for gaming and animation? - What makes successful standards? • Graphics Acceleration Standards - Khronos acceleration API standards for low power and high performance gaming and animation • Bringing accelerated graphics into the Browser - WebGL – revolutionary access to the GPU in standard browsers • 3D Content Standards - How to make content re-usable without stifling innovation

© Copyright Khronos Group 2013 - Page 2 Why Do We Need Standards? • Standards are interoperability interfaces so that compelling user experiences can be created inexpensively to build mass markets - Don’t slow growth with functionality fragmentation that adds no value • E.g. Wireless and IO standards - GSM/EDGE, UMTS/HSPA, LTE, IEEE 802.11, Bluetooth, USB …

Standards drive mobile market growth by expanding device capabilities

Right time to Standardize?

Vendor differences adding no value - Darwinian industry is still fragmentation is slowing growth – clear experimenting with what works goals emerge for a standard and what doesn’t

REFINE BY COMMITTEE DESIGN BY COMMITTEE Industry agrees on what to standardize Experimentation and design by – cooperative refinement from multiple committee can be viewpoints creates a robust solution slow and unfocused

A good standard enables A bad standard stifles innovation implementation innovation and causes commoditization

Every successful open standard has a de facto proprietary competitor and is open to competitive evolution Ecosystems seem to work best when both are healthy and evolving

© Copyright Khronos Group 2013 - Page 4 Busting Some Standardization Myths • “Standards are slow to develop” - Time to productive ecosystems is the key … … rather than minimizing time to a proprietary specification - OpenCL 1.0 took just 6 months – intensive cooperation • “If I particpate in standards I ‘lose’ my IP” - Good IP Framework fully protects Members IP and the specification - Members agree not to assert IP claims against other Members or Adopters - License grant is VERY narrow – but protects the specification in practice • “Standards are boring” - A good standard is the industry coming together to solve real issues

ROYALTY-FREE, OPEN STANDARD APIs for advanced hardware acceleration

Low level silicon to software interfaces needed on every platform

Graphics, video, audio, compute, vision, sensor and camera processing

Defines the forward looking roadmap for the silicon community

Shipping on billions of devices across multiple operating systems

Rigorous conformance tests for cross-vendor consistency

Khronos is OPEN for any company to join and participate

Acceleration APIs BY the Industry FOR the Industry

• Native framework for programming diverse parallel computing resources OpenCL KernelOpenCL CodeKernel OpenCL - CPU, GPU, DSP – as well as hardware blocks(!) CodeKernel OpenCL CodeKernel • Powerful, low-level flexibility Code - Foundational access to compute resources for GPU higher-level engines, frameworks and languages DSP CPU CPU • Embedded profile HW - No need for a separate “ES” spec - Reduces precision requirements One code tree can be executed on CPUs, GPUs, DSPs and hardware. Dynamically interrogate system load A cross-platform, cross-vendor standard for and load balance work across harnessing all the compute resources in an SOC available processors

© Copyright Khronos Group 2013 - Page 8 OpenCL Overview • C Platform Layer API - Query, select and initialize compute devices • Kernel Language Specification - Subset of ISO C99 with language extensions - Well-defined numerical accuracy - IEEE 754 rounding with specified max error - Rich set of built-in functions: cross, dot, sin, cos, pow, log … • C Runtime API - Runtime or build-time compilation of kernels - Execute compute kernels across multiple devices • Memory management is explicit - Application must move data from host  global  local and back - Implementations can optimize data movement in unified memory systems

OpenCL HLM (High Level Model) High-level programming model, unifying host and device execution environments through language syntax for increased usability and broader optimization opportunities

OpenCL 2.0 Provisional released! OpenCL 2.0 Significant enhancements to memory and execution models to expose emerging hardware capabilities and provide increased flexibility, functionality and performance to developers

OpenCL SPIR 1.2 Provisional released!

OpenCL SPIR (Standard Parallel Intermediate Representation) LLVM-based, low-level Intermediate Representation for IP Protection and as target back-end for alternative high-level languages

© Copyright Khronos Group 2013 - Page 10 Mobile OpenCL Shipping • Android ICD extension released in latest extension specification - OpenCL implementations can be discovered and loaded as a shared object • Multiple implementations shipping in Android NDK - ARM, Imagination, Vivante, Qualcomm, Samsung …

© Copyright Khronos Group 2013 - Page 11 OpenGL 3D API Family Tree ES3 is backward compatible Fixed function Programmable vertex so new features can be 3D Pipeline and fragment shaders added incrementally

OpenGL ES 1.1 OpenGL ES 2.0 OpenGL ES 3.0 Content Content Content Mobile 3D WebGL 1.0 WebGL-Next

OpenGL ES 1.1 OpenGL ES 2.0 OpenGL ES 3.0 ES-Next OpenGL ES 1.0

OpenGL 1.3 OpenGL 1.5 OpenGL 2.0 OpenGL 2.1 OpenGL 3.1 OpenGL 3.3 OpenGL 4.2 OpenGL 4.3 OpenGL 4.4 GL-Next

OpenGL 3.0 OpenGL 3.2 OpenGL 4.0

OpenGL 4.1

OpenGL 4.4 is a Desktop 3D superset of DX11

2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013

© Copyright Khronos Group 2013 - Page 12 OpenGL ES 3.0 Highlights • Better looking, faster performing games and apps – at lower power - Incorporates proven features from OpenGL 3.3 / 4.x - 32-bit integers and floats in shader programs - NPOT, 3D textures, depth textures, texture arrays - Multiple Render Targets for deferred rendering, Occlusion Queries - Instanced Rendering, Transform Feedback … • Make life better for the programmer - Tighter requirements for supported features to reduce implementation variability • Backward compatible with OpenGL ES 2.0 - OpenGL ES 2.0 apps continue to run unmodified • Standardized Texture Compression - #1 developer request!

Bringing state-of-the- art functionality to cross-platform OpenGL 4.4 graphics OpenGL 4.3 OpenGL 4.2 OpenGL 4.1 OpenGL 3.3/4.0 OpenGL 3.2 OpenGL 3.1 OpenGL 2.0 OpenGL 2.1 OpenGL 3.0

2004 2005 2006 2007 2008 2009 2010 2011 2012 2013

DirectX 9.0c DirectX 10.0 DirectX 10.1 DirectX 11 DirectX 11.1

© Copyright Khronos Group 2013 - Page 14 OpenGL 4.3 Compute Shaders • Execute algorithmically general-purpose GLSL shaders - Can operate on uniforms, images and textures • Process graphics data in the context of the graphics pipeline - Easier than interoperating with a compute API IF processing ‘close to the pixel’ • Standard part of all OpenGL 4.3 implementations - Matches DX11 DirectCompute functionality

Physics AI Simulation Ray Tracing Imaging Global Illumination

 Conformance submissions are required for GL 4.4 implementations

 encouraged for earlier driver versions

 Shared codebase with OpenGL ES 3.0 CTS

 additional desktop-specific tests

 Core profile functionality

 Enhancements underway to add more coverage

© Copyright Khronos Group 2013 - Page 16 OpenCL and OpenGL Compute Shaders • OpenGL compute shaders and OpenCL support distinctly different use cases - OpenCL provides a significantly more powerful and complete compute solution

1. Fine grain compute operations 1. Full ANSI C programming of inside OpenGL heterogeneous CPUs and GPUs 2. GLSL Shading Language 2. Utilize multiple processors 3. Execute on single GPU only 3. Precisely defined IEEE accuracy

Enhanced 3D Imaging Pure compute Graphics apps Video apps touching “Shaders++” Physics no pixels AI

Compute Shaders

Courtesy of PC Gaming Alliance www.pcgamingalliance.org © Copyright Khronos Group 2013 - Page 18 Visual Sensor Revolution • Single sensor RGB cameras are just the start of the mobile visual revolution - IR sensors – LEAP Motion, eye-trackers • Multi-sensors: Stereo pairs -> Plenoptic array -> Depth cameras - Stereo pair can enable object scaling and enhanced depth extraction - Plenoptic Field processing needs FFTs and ray-casting • Hybrid visual sensing solutions - Different sensors mixed for different distances and lighting conditions • GPUs today – more dedicated ISPs tomorrow?

Dual Camera Plenoptic Array Capri Structured Light 3D Camera LG Electronics Pelican imaging PrimeSense © Copyright Khronos Group 2013 - Page 19 OpenVX • Vision Hardware Acceleration Layer - Enables hardware vendors to implement accelerated imaging and vision algorithms - For use by high-level libraries or apps • Focus on enabling real-time vision Application - On mobile and embedded systems • Diversity of efficient implementations OpenCV open Other higher-level - From programmable processors, through source library CV libraries GPUs to dedicated hardware pipelines

Dedicated hardware can help make vision processing performant and low-power enough for pervasive ‘always-on’ use

Open source sample Hardware vendor implementation implementations

© Copyright Khronos Group 2013 - Page 20 OpenVX - Power Efficient Vision Acceleration • Create vision processing graph for power and performance efficiency - Each Node can be implemented in software or accelerated hardware - Nodes may be fused by the implementation to eliminate memory transfers • EGLStreams can provide data and event interop with other APIs - BUT use of other Khronos APIs are not mandated • VXU Utility Library provides efficient access to single nodes - Open source implementation – easy way to start using OpenVX

OpenVX Node Native OpenVX OpenVX Camera Node Node Control OpenVX Node Heterogeneous Processing

Open Source Formal specification and full Governance Community Driven conformance tests No formal specification Implemented by hardware vendors Very wide Tight focus on hardware accelerated Scope 1000s of functions of imaging and vision functions for mobile vision Multiple camera APIs/interfaces Use external camera API No Conformance testing Full conformance test suite / process Conformance Every vendor implements different subset Reliable acceleration platform Use Case Rapid prototyping Production deployment Memory-based architecture Graph-based execution Efficiency Each operation reads and writes memory Optimized nodes and data transfer Sub-optimal power / performance Highly efficient

© Copyright Khronos Group 2013 - Page 22 Typical Imaging Pipeline • Pre- and Post-processing can be done on CPU, GPU, DSP… • ISP controls camera via 3A algorithms Auto Exposure (AE), Auto White Balance (AWB), Auto Focus (AF) • ISP may be a separate chip or within Application Processor

Lens, sensor, aperture control 3A Bayer RGB/YUV Image Signal Processor Post- Pre-processing App (ISP) processing

CMOS sensor Need for advanced Color Filter Array camera control API! Lens

© Copyright Khronos Group 2013 - Page 23 Camera Control API Goals • Provide functional portability for advanced camera applications - Reduce extreme fragmentation for ISVs wanting more than point and shoot • Generate image bursts with parameterized camera control and ISP control - For downstream processing by flexible combination of CPU, GPU and DSP • Control multiple sensors with multi-sensor synch and alignment - Stereo pairs, Plenoptic arrays, Depth Cameras • Enable system-wide sensor time-stamping - Synchronize MEMS and image sensor samples

• This functionality is not available on any current platform APIs - Make this API align with future platform direction for easy adoption

© Copyright Khronos Group 2013 - Page 24 Camera API Design Philosophy • C-language API starting from proven designs - e.g. FCAM, Android camera platform • Design alignment with widely used hardware standards - e.g. MIPI CSI • Focus on mobile, power-limited devices - But do not preclude other use cases such as automotive, surveillance, DSLR… • Minimize overlap and maximize interoperability with other Khronos APIs - But other Khronos APIs are not required • Provide support for vendor-specific extensions

Group charter First draft Specification approved specification ratification Apr13 4Q13 2Q14 Jul13 1Q14 3Q14 Provisional Sample specification implementation and tests © Copyright Khronos Group 2013 - Page 25 Low Power Environment Scanning • Many sensor use cases would consume too much power to be running 24/7 - Environment aware use cases have to be very low power • ‘Scanners’ - very low power, always on, detect things in the environment - Trigger the next level of processing capability

ARM 7 DSP GPU 1 MIP and accelerometers can Low power activation of camera GPU acceleration for precision detect someone in the vicinity to detect someone in field of view gesture processing

Applications

Middleware engines need platform- Platforms can provide portable access to native, low-level increased access to OS Sensor OS APIs Middleware sensor data stream improved sensor data stream (E.g. Android SensorManager or (E.g. Augmented Reality engines, – driving faster, deeper iOS CoreMotion) gaming engines) sensor usage by applications

Mobile or embedded platforms without sensor fusion APIs can provide direct application access Low-level native API defines access to to StreamInput StreamInput implementations fused sensor data stream and context-awareness compete on sensor stream quality, reduced power consumption, environment triggering and context detection – enabling sensor subsystem vendors to increased Sensor Hardware transport ADDED VALUE Sensor … Sensor interfaces are defined Sensor Hub by each system, e.g. Hub IIO or HID sensor

VR needs not just advanced sensor processing, vision acceleration, computation and rendering - but also for all these subsystems to work efficiently together Audio Rendering

MEMS Application Sensors Sensor on CPUs, GPUs Fusion and DSPs

Precision timestamps Feature Tracking on all sensor samples

Advanced Camera EGLStream 3D Rendering and Video Control and stream Stream frames Composition generation between APIs On GPU Camera Control API

WORLDWIDE GAME USERS BY REGION AND PLATFORM (IN MILLIONS) 500

400

300

200

100

0 N.America Europe Asia Other Console PC Dedicated Handheld Mobile

Courtesy of PC Gaming Alliance www.pcgamingalliance.org © Copyright Khronos Group 2013 - Page 30 Leveraging Proven Native APIs into HTML5 • Khronos and W3C liaison - Leverage proven native API investments into the Web - Fast API development and deployment - Designed by the hardware community - Familiar foundation reduces developer learning curve

HTML WebCAM(!) Canvas WebVX? WebStream? Camera control and JavaScript Vision Sensor Fusion Processing video processing

Camera Control Native Path Rendering

JavaScript API shipping, Possible future Native APIs shipping acceleration being developed JavaScript APIs or or Khronos working group or work underway acceleration

Content JavaScript, HTML, CSS, ... Content downloaded from the Web. Middleware can make WebGL accessible to non-expert 3D programmers JavaScript Middleware

HTML5 Browser provides WebGL functionality alongside other HTML5 technologies - no plug-in required JavaScript CSS

OpenGL ES 2.0 OS Provided Drivers. WebGL on Windows OpenGL can use Google Angle to create conformant DX9/Angle OpenGL ES 2.0 over DX9

Much WebGL content uses three.js library: http://threejs.org/

- Microsoft – “where you have IE11, you have WebGL – turned on by default and working all the time” - Microsoft - WebGL also enabled for Windows applications - web app framework and web view - Apple - WebGL must be explicitly turned on MAC Safari and only exposed on iOS for iAds - Chrome OS - WebGL is the only cross-platform API to program the GPU - Google IO announcement - Chrome on Android will soon launch with WebGL © Copyright Khronos Group 2013 - Page 33 Cross-OS Portability

HTML5 provides cross HTML/CSS HTML/CSS HTML/CSS platform portability. GPU accessibility through WebGL available soon on ~90% mobile systems

Dalvik Preferred development SDK Objective C C# environments not (Java) designed for portability

Native code is portable- but apps must cope with C/C++ DirectX different available APIs and libraries

© Copyright Khronos Group 2013 - Page 34 WebGL First Wave Application Categories • Maps and Navigation • Modeling Tools and Repositories • Games • 3D Printing • Visualization • Music Videos and Promotion • Education • Photo Editors • Music Visualizers • Vision/Video Processing

© Copyright Khronos Group 2013 - Page 36 WebCL – Parallel Computing for the Web • JavaScript bindings to OpenCL APIs - Enables initiation of Kernels written in OpenCL C within the browser • Bindings stay close to the OpenCL standard - Maximum flexibility to provide a foundation for higher-level middleware - Minimal language modifications for 100% security and app portability - E.g. Mapping of CL memory objects into host memory space is not supported • API definition underway – public draft released - https://cvs.khronos.org/svn/repos/registry/trunk/public/webcl/spec/latest/index.html • Compelling use cases - Physics engines for WebGL games River Trail - Image and video editing in browser

JavaScript binding to OpenCL Data-parallel language APIs to enable initiation of extensions to JavaScript OpenCL C kernels for offload at run-time

© Copyright Khronos Group 2013 - Page 37 3D Needs a Transmission Format! • Compression and streaming of 3D assets becoming essential - Mobile and connected devices need access to increasingly large asset databases • 3D is the last media type to define a compressed format - 3D is more complex – diverse asset types and use cases • Needs to be royalty-free - Avoid an ‘internet video codec war’ scenario • Eventually enable hardware implementations of successful codecs - High-performance and low power – but pragmatic adoption strategy is key

Audio Video Images 3D MP3 H.264 JPEG ? ! An effective and widely adopted codec ignites previously unimagined opportunities for a media type © Copyright Khronos Group 2013 - Page 38 glTF Goals • Binary file format for efficient transmission for 3D assets - Reduce network bandwidth and minimize client processing overhead • Run-time neutral - DO NOT IMPLY OR MANDATE ANY RUN-TIME BEHAVIOR - Can be used by any app or run-time – usually WebGL accelerated • Scalable to handle compression and streaming - Though baseline format does not include compression • ‘Direct load efficiency’ for WebGL - Little or NO processing to drop glTF data into WebGL client • Carry conditioned data from any authoring format - Prototyping and optimizing efficient handling of COLLADA assets

A standards-based content pipeline for rich native and Web 3D Authoring Playback applications

OpenCOLLADA Tool Interop COLLADA2GLTF Importer/Exporter Translator and COLLADA Other Conformance Tests authoring On GitHUB formats

https://github.com/KhronosGroup/glTF

Web-based https://github.com/KhronosGroup/OpenCOLLADA Tools https://github.com/KhronosGroup/COLLADA-CTS

Pervasive WebGL deployment Three.js glTF Importer. Rest3D initiative

© Copyright Khronos Group 2013 - Page 40 Conclusion • Gaming and Animation is a complex and expanding application domain and multiple standards across multiple domains are needed to enable the market • Advances in SOC silicon processing and associated APIs are about to enable Generated Reality to truly meet user expectations on all platforms • Now is a great time to get involved with the standards initiatives that effect your business

• www.khronos.org • [email protected]