OpenSL ES & OpenMAX Multimedia

Erik Noreke Chair of OpenSL ES Working Group Chair of OpenMAX AL Working Group

© Copyright , 2011 - Page 1 Today’s Consumer Requirements

• Rich media applications and UI - Consumer decisions are driven by the multimedia experience - Drives smart phone market • Enhanced user experience - Games - Theater-like audio for video playback - Streaming media services - Mobile TV • Availability of third party applications - Consumers expected to purchase $35 billion of mobile applications by 2014

© Copyright Khronos Group, 2011 - Page 2 Application Developer Requirements

• Use case driven design - For ease of use • Comprehensive feature set - Including advanced audio and streaming media • Consistent APIs - Same APIs for both hardware and software implementations • Full, predictable functionality - No need to guess available or what is going to work • Portable - No need to port application for different platforms

© Copyright Khronos Group, 2011 - Page 3 Why Create Khronos Multimedia APIs?

Khronos APIs Open Source Framework • Conformance Tests • Free to change • Portable • Every implementation unique • IP Protection • GPL license • Profiles & Components • Plugins • Predictable set of functionality • Functionality dependent on implementation

© Copyright Khronos Group, 2011 - Page 4 Integration Layer

© Copyright Khronos Group, 2011 - Page 5 OpenMAX IL Portable Media Integration

• Low-level interface for audio, video, and imaging codecs • Developed for embedded and mobile devices • Enables interfacing with multimedia components in a unified manner • Allows components to be any combination of hardware or software completely transparently to the media chain • The goal of OpenMAX IL is to abstract multimedia components - Combat the problem of integration portability - Consistent functionality

© Copyright Khronos Group, 2011 - Page 6 OpenMAX IL – Component Networks

• Enables arbitrary multimedia pipelines by plugging blocks together - Componentized architecture abstracts multimedia functionality block interfaces • Wide variety of building blocks for imaging, video and audio functions - Encode, decode, apply an effect, capture, render, split, mix, etc • Enables blocks from different sources to work together - Blocks can be implemented in software or hardware

Portable & Re-usable streaming media building blocks

© Copyright Khronos Group, 2011 - Page 7 OpenMAX IL Example Graph

• Standardized component interfaces enable flexible media graphs • Includes multi-stream synchronization • Allows for custom plug-ins

OpenMAX Component Interfaces Enables components to be flexibly E.g. send dance beats to 3D connected in any graph topology application for visualization

AAC Audio Audio Audio Decoder Plug-in Renderer Speakers

*.mp4 / *.3gp Clock File Reader for AV Sync Time Data Video Video Video Scheduler Renderer Display Decoder MPEG4/ Decompressed H.264 Video Video

Example: MPEG-4 video synchronized with AAC audio decode © Copyright Khronos Group, 2011 - Page 8 &

Application Access Layer

© Copyright Khronos Group, 2011 - Page 9 The Need for Application Access • Hardware and low level access is powerful but complex - More power than most application developers require • Most application developers just want to playback and record media - Specifying where the content comes from - Specifying where the content should be rendered to - Manipulate a few playback controls - Have simple configurability • That are what OpenMAX AL and OpenSL ES provide… - Simple high-level multimedia APIs for playback and recording use cases - With hardware acceleration capabilities built in • …With Cross-platform portability - No need to rewrite an application for every platform

© Copyright Khronos Group, 2011 - Page 10 OpenMAX AL Focused on Streaming Media

• Media playback and recording - Playback and recording of everything from video files to radio to camera input • Full range of video effects and controls – including playback rate, post processing, and image manipulation - Apply advanced effects to captured images and video before sending to friends • Complete metadata support - Read, Edit, and Set the metadata for most types of content • OpenMAX AL gives easy access to streaming media - Allowing time to be spent on developing applications, not using frameworks • Designed as a complete set of multimedia functionality - Unlike any other streaming media API

© Copyright Khronos Group, 2011 - Page 11 OpenSL ES – Powerful Mobile Audio

• Full range of effects and controls, including advanced 3D effects such as Doppler and virtualization - Experience rich, enhanced sound from locations other than the handset, even moving, for the ultimate multimedia experience • Create theater quality audio experience - In a mobile device! • Full 3D audio functionality enhances any gaming experience - Perfect companion to OpenGL ES • Designed for implementation by either a hardware or software solution - Take full advantage of system capabilities

© Copyright Khronos Group, 2011 - Page 12 Object Oriented Media Access

• Simplified object-oriented streaming media - Built to enable common use cases - But also has access to advanced functionality • Media Objects enable PLAY and RECORD of content - Media Objects input and output to Devices - Perform some operation on an input and emit the result as output • Objects have control interfaces - Play, Record, Seek, Rate

Media Input Audio Mix

Camera (AL) Display Window (AL)

URI DSrc Player Object DSnk URI

Memory Memory

Content pipe Content pipe

© Copyright Khronos Group, 2011 - Page 13 OpenMAX AL & OpenSL ES

© Copyright Khronos Group, 2011 - Page 14 OpenSL ES and OpenMAX AL • Working groups collaborated to define the common API functionality • Independent, Compatible, Consistent and Distinct

Video 3D Audio playback Audio Video Playback recording Audio Effects Radio and RDS Audio Advanced Recording Camera MIDI

Basic Image capture Buffer MIDI & display queues

© Copyright Khronos Group, 2011 - Page 15 Why Profiles?

• Functionality divided into basic groups - Most common groupings • Profiles reduce fragmentation - Not all devices support all functionality - Only support the profiles you need • Applications can query available profiles - To adapt to the current device - Facilitates porting of applications • Profiles can be combined - To best suit any product • Optional features - Some features are optional in all profiles - APIs are consistent when feature is available

© Copyright Khronos Group, 2011 - Page 16 OpenSL ES and OpenMAX AL • Independent - But designed to work together - No dependency between the APIs – either one can exist by itself • Compatible - Working groups collaborated to make sure the APIs work together well • Consistent - Identical API architecture - Identical APIs for same functionality • Distinct - OpenMAX AL represents basic multimedia functionality (audio, video, and image) - OpenSL ES represents advanced audio functionality

© Copyright Khronos Group, 2011 - Page 17 OpenSL ES / OpenMAX AL – Implemented on top of OpenMAX IL

OR

URI DataSource Media Player Output Mix

File Reader Audio Audio Audio Audio Decoder Effect Mixer Sink Clock

© Copyright Khronos Group, 2011 - Page 18 Complete Multimedia Use-Case

Audio stream to DSP 3D Audio Rendering

Multimedia Application Processing on CPU

Video stream to GPU

OpenGL ES EGLStream GL_TEXURE_EXTERN OpenGL ES EGLSurface AL_OES texture

Much more flexibility than just “show a video”

© Copyright Khronos Group, 2011 - Page 19 Android NDK for Multimedia • NDK – Native Development Kit • Introduced in Android 2.3 Gingerbread Advanced Audio

Mobile 3D

© Copyright Khronos Group, 2011 - Page 20 Why the NDK?

• Better control of fuctionality and timing • Access to hardware acceleration • Multimedia is the primary reason for the NDK

• Increases application complexity • Requires compilation for each platform family - ARMv5TE - ARMv7-A - x86 instructions

© Copyright Khronos Group, 2011 - Page 21 Focus is on Standardized APIs

• Earlier internal APIs not stable enough…

• Avoid fragmentation • Well tested and documented • Focus on functionality - Not on implementation • Access to hardware acceleration

© Copyright Khronos Group, 2011 - Page 22 Native Multimedia APIs in Android 2.3

• OpenGL ES 2.0 • EGL 1.4 • OpenSL ES 1.0.1 - Partial implementation Mobile 3D - Almost complete Phone + Music profiles - Buffer queues from 1.1 as Android extension - Google version compliant as far as implemented - Other vendors offer fully compliant solution

Graphics Context Advanced Audio Management

© Copyright Khronos Group, 2011 - Page 23 Ice Cream Sandwich – Android 4.0 • Same OS for both phones and tablets • Better multitasking • Better support for hardware accelerated graphics • DRM • Partial OpenMAX AL 1.0.1 implementation - Streaming use-case only - Does not interoperate with OpenSL ES

Streaming Media

© Copyright Khronos Group, 2011 - Page 24 Android & OpenMAX IL

• Android uses OpenMAX IL in their Multimedia framework - Codecs only - No graph building • Main Multimedia framework is still Stagefright - Introduced in Éclair • Allows for both software implemented and hardware accelerated codecs • Not part of NDK - Not intended to be accessible from applications

© Copyright Khronos Group, 2011 - Page 25 What is Google saying?

© Copyright Khronos Group, 2011 - Page 26 What is Google saying?

© Copyright Khronos Group, 2011 - Page 27 What is Google saying?

© Copyright Khronos Group, 2011 - Page 28 Where is Android Headed?

• Fler enheter - TV? - Netbooks? - Chrome OS?

• What does the future hold? - Compliant OpenSL ES? - Better OpenMAX AL support? - OpenMAX AL EGL Stream Extension? - Support for OpenCL? - Content Pipes?

• Its anybody’s guess!

© Copyright Khronos Group, 2011 - Page 29 Any Questions?

© Copyright Khronos Group, 2011 - Page 30 Thank You

© Copyright Khronos Group, 2011 - Page 31