Khronos Apis Update

Khronos APIs Update

Neil Trevett Vice President Mobile Content, NVIDIA President, The Khronos Group

© Copyright Khronos Group, 2011 - Page 1 Khronos - Connecting Software to Silicon • Creating open, royalty-free acceleration API standards - Focus on graphics, dynamic media, compute and sensor hardware • Low-level - just above raw silicon - “Foundation” functionality needed on every platform • Safe forum for industry cooperation - „By the industry for the industry‟ - Open to any company to join

- IP framework to protect APIs enable software members and industry developers to turn silicon functionality into rich end user experiences

• Khronos APIs designed to enable healthy implementation-level innovation and competition in the open market

Khronos Family of Standards

StreamInput 3D Digital Asset Plugin-free Web Unified Sensor and

Exchange format 3D Web Content Compute Input Processing

accessibility

Authoring and and Authoring

A coordinated ecosystem of Cross platform Parallel Embedded and Vector 2D compute, graphics and media

desktop 3D Computing Mobile 3D standards and APIs

Application Application Acceleration Acceleration

Streaming Media Advanced Audio Surface Management

Khronos creates royalty-free specifications to meet real market needs and helps drive industry adoption across multiple platforms

© Copyright Khronos Group, 2011 - Page 4 Responding to Industry Trends High-end API Breakthrough apps embrace technology is created on mobility’s strengths – not just treat high-end platforms phones as small PCs - will need complex, interoperating APIs e.g. Augmented Reality

As platforms diversify – mobile, Mobile is the new platform for apps TV, embedded – HTML5 will innovation. Mobile APIs are needed become increasingly important to unlock hardware potential while as a universal app platform conserving battery life

© Copyright Khronos Group, 2011 - Page 5 AR – Key Use Case driving Khronos APIs • Many APIs need to work closely together - great goal! • Commercial uses of AR-related technologies continuing to evolve… • … but - the technologies needed here will enable amazing apps!!

Camera video stream sent to the compositor

Camera images used 3D to track the augmentations camera’s composited with location and video stream orientation

Camera 3D Augmentation Tracking Camera-to-scene transform locks Rendering the 3D rendering to the real world © Copyright Khronos Group, 2011 - Page 6 OpenGL ES – 3D Graphics • OpenGL for embedded and mobile devices - Eliminates redundant and legacy features - Adds extensions to make it mobile-friendly • The dominant 3D API for mobile devices - Widely adopted for STB, DTV, automotive,… - Hundreds and hundreds of millions shipped • Runs high-end content and engines - UE3, Unity, Unigine, Rage

• Platform Layer API - Query, select and initialize compute devices • Runtime API - Execute compute kernels – gather results • Kernel Language Specification - Subset of ISO C99 with language extensions • OpenCL has Embedded profile - No need for a separate “ES” spec

©Copyright Copyright Khronos Khronos 2009 Group, 2011 - Page 8 OpenMAX AL - Object Oriented Media • Object oriented processing of camera, images and video with AV sync - Connect to variety of input and output objects to PLAY and RECORD media • Object control interfaces - Sources: Mix control, Seek, Rate, Metadata Extraction, Camera Controls - Sinks: Encode control, Tuning, MIDI, Metadata Insertion • Video and image stream routing to other APIs - To CPU and GPU subsystems

Sinks Sources EGLStream to ES Analog Radio Data Tap to CPU Audio Mix Camera Display Window Audio Input DSrc OpenMAX AL DSnk Media Object URI URI Memory Memory

OpenMAX AL Media EGLStreams enables and hides details of OpenGL ES GL_TEXTURE_EXTERNAL is Player is the EGLStream video frame transport. Enables multiple the EGLStream “Consumer” and “Producer” and controls buffering modes for different uses cases converts video format into RGB production of frames. eg: FIFO and explicit latch/release OpenGL ES texture

Camera OpenMAX AL OpenGL ES File MEDIA PLAYER EGLStream GL_TEXTURE_EXTERNAL URL Etc. Object ?

© Copyright Khronos Group, 2011 - Page 10 OpenSL ES – Advanced Audio • OpenSL ES does for audio what OpenGL ES does for graphics - Advanced audio functionality from simple playback to 3D spatialized audio • Object-based native audio API for simplicity and high performance - Reduces development time • Same API regardless of underlying implementation - Software or hardware accelerated • Cross OS portability - Preserves application investment

Apps request semantic sensor information StreamInput defines list of possible semantic requests “Am I in an elevator?” “Give me gestures and face position”

Standardized Node Universal Intercommunication Timestamps

Input Input Input Device Device Device

Filter Filter Node Node Advanced Sensors Everywhere Standard cameras, depth cameras Apps Need Sophisticated motion and position, touch, microphones Filter App Access to Sensor Data wireless controllers Node Without coding to specific systems or sensor hardware SHOULD NOT FORCE APPLICATIONS TO Sensor graph created to provide sensor information StreamInput defines graph creation API and node interconnects ACCESS INDIVIDUAL Low-level sensor processing encapsulated in nodes – unleashes fusion innovation SENSORS Apps gain „magical‟ situational awareness

© Copyright Khronos Group, 2011 - Page 12 OpenCV as Potential Khronos Standard • OpenCV is widely use open source project for COMPUTER VISION Application • Khronos Hardware Abstraction Layer - Would enable hardware vendors to provide accelerated imaging High-level CV and vision modules algorithms library • Being sponsored by NVIDIA, Itseez and Willow Garage Hardware Abstraction Layer - Decision to initiate in few weeks time

Open source sample Hardware vendor implementation implementations

Can optionally use OpenCL to implement

Audio StreamInput Rendering

Positional Positional and Position and Tracking Sensors GPS Sensor Data Synchronization Semantics Application and sensor on CPU fusion Computer Vision and Tracking OpenCV

Video TAP to CPU Video stream to GPU Camera 3D Rendering and Camera EGLStream Processing Video Composition Control Camera, Preprocess and generate video streams Much more flexibility than just “overlay augmentations over background”

© Copyright Khronos Group, 2011 - Page 14 Leveraging Native API Investment into HTML5 • HTML5 evolving into cross-platform programming platform - Gradually exposing complete system capabilities • Opportunity to synergize Web and native APIs development - Leverage native API investments, reduce developer learning cycles • Khronos and W3C creating close liaison

WebMAX? Device and WebAudio Sensor APIs Camera HTML and Advanced Device control and Browser JavaScript Orientation video Composition JavaScript Audio Working processing Groups

StreamInput Native

Native APIs shipping JavaScript API shipping Possible future or working group underway or working group underway JavaScript APIs

© Copyright Khronos Group, 2011 - Page 15 WebGL and HTML Interaction • 3D is not trapped in a rectangular window - 3D can overlay and underlay HTML content - Easy to make 2D HTML HUDs or 3D user interfaces • Strong ties with other advanced HTML5 - WebGL can use HTML5

© Copyright Khronos Group, 2011 - Page 16 WebCL – Parallel Computing for the Web • Khronos launching new WebCL initiative - First announced in March 2011 - API definition already underway • JavaScript binding to OpenCL - Security is top priority • Many use cases - Physics engines to complement WebGL - Image and video editing in browser • Stay close to the OpenCL standard - Maximum flexibility - Foundation for higher-level middleware

© Copyright Khronos Group, 2011 - Page 17 In Summary • APIs are key to enable compelling applications on advanced hardware – APIs developed at high-end now enabling mobile and embedded devices • APIs no longer exist alone – they are interoperating to form platform ecosystems for advanced content • Significant cooperation happening between native and Web APIs to bring advanced visual computing to HTML5 • Khronos is driving open standards for hardware acceleration Join, change the industry AND get the inside edge for your products!