Khronos Overview Taiwan Feb 2012

Khronos Overview

Taiwan, February 2012

© Copyright Khronos Group, 2012 - Page 1 Khronos - Connecting Software to Silicon • Creating open, royalty-free API standards - Focus on graphics, dynamic media, compute and sensor hardware • Low-level - just above raw silicon - “Foundation” functionality needed on every platform • Safe forum for industry cooperation - „By the industry for the industry‟ - Open to any company to join - IP framework to protect

members and industry APIs enable software developers to turn silicon functionality into rich end user experiences

© Copyright Khronos Group, 2012 - Page 2 Khronos API Ecosystem - Trends High-end API Breakthrough apps embrace technology is created on mobility‟s strengths – not just treat high-end platforms phones as small PCs - will need complex, interoperating APIs e.g. Augmented Reality

As platforms diversify – mobile, Mobile is the new platform for apps TV, embedded – HTML5 will innovation. Mobile APIs are needed become increasingly important to unlock hardware potential while as a universal app platform conserving battery life

Advanced Audio Camera, Images and Streaming Media Computer Vision Surface Hardware accelerated Management vision functions 3D Digital Asset Exchange format Parallel Computing

Embedded and Unified Sensor and Mobile 3D Input Processing Vector 2D Cross platform Web desktop 3D Compute

Plugin-free 3D Web Content

Khronos creates royalty-free specifications to meet real market needs and helps drive industry adoption across multiple platforms

Academic Members - Participation in WGs Working Groups (WGs) Contributor Members One working group per API - Participation and vote in WGs

Promoter Members - Participation and vote in WGs - Board seat for strategy, budget and spec ratification

$$$ $

Conformance Tests Ratified SDKs, Sample, Ref Cards Educator Guidelines and Adopters Program Specifications and Man Pages and Courseware Feedback

Members Wider Industry $ Adopters Developers Educators Build conformant Develop applications Collaboration for Khronos implementation and products using the APIs API Education

WebCL Computer Vision OpenMAX AL 1.1 Working Group Working Group Spec Released Announced Announced

OpenSL ES 1.1 OpenGL 4.2 EGLStreams Spec Released Spec Released Spec Release

Mar11 Nov11

Jan11 Aug11 Dec11 WebGL 1.0 OpenCL 1.2 Spec Released Spec Release

StreamInput Working Group Announced

COLLADA 1.4 Conformance Tests Released

Tessellation and Compute „Shape 4.X realism‟

Geometry Shaders 3.X

Vertex and Fragment Shaders 2.X Fixed Function 1.X „Surface realism‟

Bringing state-of-the- art functionality to cross-platform graphics

OpenGL 4.1

OpenGL 3.3/4.0

OpenGL 3.2

OpenGL 3.1

OpenGL 2.0 OpenGL 2.1 OpenGL 3.0

2004 2005 2006 2007 2008 2009 2010 2011

DirectX 9.0c DirectX 10.0 DirectX 10.1 DirectX 11

© Copyright Khronos Group, 2012 - Page 9 OpenGL ES Adoption • OpenGL for embedded and mobile devices - Eliminates redundant and legacy features - Adds extensions to make it mobile-friendly • The dominant 3D API for mobile devices - Widely adopted for STB, DTV, automotive,… - Hundreds and hundreds of millions shipped • OpenGL ES 2.0 - Fully programmable pipeline • Runs high-end content and engines - UE3, Unity, Unigine, Rage

CPUs GPUs Multiple cores driving Emerging Increasingly general performance increases Intersection purpose data-parallel computing

Multi- Heterogeneous Graphics processor Computing APIs and programming Shading – e.g. OpenMP Languages

OpenCL is a programming framework for heterogeneous compute resources

• Platform Layer API - Query, select and initialize compute devices • Kernel Language Specification - Subset of ISO C99 with language extensions • Runtime API - Execute compute kernels – gather results • OpenCL has Embedded profile - No need for a separate “ES” spec

© Copyright Khronos Khronos 2009 Group, 2012 - Page 12 OpenCL 1.2 Announced in December • Significant updates - Khronos being responsive to developer requests - Updated OpenCL 1.2 conformance tests available • Backward compatible upgrade to OpenCL 1.1 - OpenCL 1.2 will run any OpenCL 1.0 and OpenCL 1.1 programs - OpenCL 1.2 platform can contain 1.0, 1.1 and 1.2 devices - Maintains embedded profile for mobile and embedded devices • Significant functionality added in response to developer demands - Partition Devices into sub devices - Separate compilation and linking of programs - Enhanced Image Support - Custom devices and built-in kernels - DX9 and DX11 Surface Sharing - Installable Client Drivers (optional)

© Copyright Khronos Group, 2012 - Page 13 Video and Audio Acceleration • OpenSL ES 1.1 and OpenMAX AL 1.1 launched in January 2011 - Additional functionality including improved buffer queuing - OpenMAX AL Digital TV Extension - technology-neutral access to Digital TV • OpenMAX AL is designed as application API - OpenMAX IL is for system integration – more flexibility, less portability • OpenMAX AL enables key image, camera and video use cases - Allows optimal hardware acceleration with application portability

Advanced image capture and photography

HD content playback with robust DRM

HD video teleconferencing

Augmented Reality

© Copyright Khronos Group, 2012 - Page 14 OpenSL ES – Advanced Audio • OpenSL ES does for audio what OpenGL ES does for graphics - Advanced audio functionality from simple playback to 3D audio • Object-based native audio API for simplicity and high performance - Reduces development time • Same API regardless of underlying implementation - Software or hardware accelerated • Cross OS portability - Preserves application investment

Advanced Audio Multimedia Video 3D Audio playback Audio Video Playback recording Audio Effects Radio and RDS Audio Advanced Recording Camera MIDI

Basic Image capture Buffer MIDI & display queues

Common Object-oriented API Framework

Camera video stream sent to the compositor

Camera 3D images used augmentations to track the composited with camera‟s video stream location and orientation

Camera 3D Augmentation Tracking Camera-to-scene transform locks Rendering the 3D rendering to the real world

OpenMAX AL Media Player EGLStreams enables and hides details of OpenGL ES GL_TEXTURE_EXTERNAL is the EGLStream video frame transport. Enables multiple is the EGLStream “Consumer” and “Producer” and controls buffering modes for different uses cases converts video format into RGB production of frames. eg: FIFO and explicit latch/release OpenGL ES texture

Camera OpenMAX AL OpenGL ES File MEDIA PLAYER EGLStream GL_TEXTURE_EXTERNAL URL Etc. Object ?

Apps request semantic sensor information StreamInput defines list of possible semantic requests “Am I in an elevator?” “Give me gestures and face position”

Standardized Node Universal Intercommunication Timestamps

Input Input Input Device Device Device

Filter Filter Node Node Advanced Sensors Everywhere Standard cameras, depth cameras Apps Need Sophisticated motion and position, touch, microphones Filter App Access to Sensor Data wireless controllers Node Without coding to specific systems or sensor hardware SHOULD NOT FORCE Sensor graph created to provide sensor information APPLICATIONS TO StreamInput defines graph creation API and node interconnects ACCESS INDIVIDUAL Low-level sensor processing encapsulated in nodes – unleashes fusion innovation SENSORS Apps gain „magical‟ situational awareness

© Copyright Khronos Group, 2012 - Page 21 Khronos Computer Vision Working Group • Vision Hardware Acceleration Layer - Enable hardware vendors to implement accelerated imaging and vision algorithms • CV HAL* can be used by high-level Application libraries or applications directly - Primary focus on enabling mobile and embedded systems OpenCV open Other higher-level • OpenCV is widely used open source source library CV libraries library for vision projects - Future version will leverage CV HAL CV HAL - * CV Hal is placeholder name

Open source sample Hardware vendor implementation? implementations

Semantics and fusion of Implement camera and positional sensors StreamInput vision sensor modules with OpenCV High-level computer vision library Accelerate OpenCV library with CV HAL Camera input from Functions Data and event OpenMAX AL or Accelerated computer interop with CL / other camera GL / ES for display subsystems CV HAL vision algorithms and compute Use OpenCL processing to implement CV HAL with parallel execution

Parallel computation

Audio Rendering

Positional Positional and Position and Tracking Sensors GPS Sensor Data Synchronization Semantics Application and sensor on CPU fusion Computer Vision and Tracking CV HAL

Video TAP to CPU Video stream to GPU Camera 3D Rendering and Camera EGLStream Processing Video Composition Control Camera, Preprocess and generate video streams Much more flexibility than just “overlay augmentations over background”

OpenGL ES 2.0 OpenGL ES Shipping - Android 2.2 OpenSL ES 1.0 OpenSL ES Shipping – Android 2.3 OpenMAX AL 1.0 OpenMAX AL Shipping - Android 4.0 EGL 1.4 EGL Shipping under SDK -> NDK Not yet OpenCL adopted Working group will StreamInput ship spec in 2012 Working group will CV HAL ship spec in 2012

© Copyright Khronos Group, 2012 - Page 25 Native APIs on Android • Khronos developing open cross vendor standards for key use cases - Khronos APIs have strong momentum in the silicon community • Drive adoption - add these APIs to Android NDK - Do not break/change existing Google APIs - Evangelize Google to possibly adopt into standard platform • Extended APIs can be used by: - Bundled apps, Market apps with API selection - Multiple APKs behind single multi-APK SKU

Evangelize Google and ISVs to adopt enabling APIs

Standard Extended NDK for Android SDK Key use cases and NDK E.g. video, camera, imaging, sensor and composition APIs

Traditional Web-content Rich Experiential Processing Multi-core CPUs Rich 2D and 3D GPU GPU Computing Multiple HD cameras Image and vision processing Video encode/decode Audio encode/decode Inertial and positional sensors

How can the Browser rapidly assimilate such diverse functionality?

© Copyright Khronos Group, 2012 - Page 27 Leveraging Native API Investment into HTML5 • HTML5 evolving into cross-platform programming platform - Gradually exposing complete system capabilities • Opportunity to synergize Web and native APIs development - Leverage native API investments, reduce developer learning cycles • Khronos and W3C creating close liaison

WebMAX? Device and WebAudio Sensor APIs Camera Advanced WebVL? Device control and JavaScript Vision Orientation video JavaScript Audio Processing Working processing Groups

CV HAL Native

Native APIs shipping JavaScript API shipping Possible future or working group underway or working group underway JavaScript APIs

© Copyright Khronos Group, 2012 - Page 28 WebGL Deployment • WebGL 1.0 Released at GDC March 2011 - Mozilla, Apple, Google and Opera working closely with GPU vendors • Typed array 1.0 spec ratified by Khronos in May - Supporting bulk data transfer between threads (workers) - Many use cases - background mesh loading, generation, deformation, physics ... • 1.0.1 release of WebGL spec and conformance suite imminent - 100% robust stance on security - Fixing bugs in 1.0.0 conformance suite - Implementations will report getContext("webgl") (not experimental)

WebGL is not enabled by default in Safari http://caniuse.com/#search=webgl © Copyright Khronos Group, 2012 - Page 29 Why Khronos for WebGL? • Unique forum where browser and GPU vendors can cooperate • Opened process to enable cooperation with web community

GPU Silicon Browser Vendors Vendors

Web Community

© Copyright Khronos Group, 2012 - Page 30 WebCL – Parallel Computing for the Web • Khronos launching new WebCL initiative - First announced in March 2011 - API definition already underway • JavaScript binding to OpenCL - Security is top priority • Many use cases - Physics engines to complement WebGL - Image and video editing in browser • Stay close to the OpenCL standard - Maximum flexibility - Foundation for higher-level middleware

Cross platform

desktop 3D

Unified Sensor and ! Input Processing

Plugin-free Embedded and APIs Native 3D Web Content Mobile 3D Web Graphics Sensors APIs Compute Imaging and vision „CV HAL‟ Hardware accelerated vision functions

Web ! Compute Heterogeneous Parallel Computing Camera, Imaging and Streaming Media

© Copyright Khronos Group, 2012 - Page 32 In Summary • APIs are key to enable compelling applications on advanced hardware – APIs developed on high-end hardware are now enabling mobile devices • APIs no longer exist alone – they are interoperating to form platform ecosystems for advanced content • Significant cooperation happening between native and Web APIs to bring advanced visual computing to HTML5 • Khronos is driving open standards for hardware acceleration Participate, change the industry AND get the inside edge for your products!

Connecting Software to Silicon