Opensl ES & Openmax Multimedia Apis

OpenSL ES & OpenMAX Multimedia APIs

Erik Noreke Chair of OpenSL ES Working Group Chair of OpenMAX AL Working Group

• Rich media applications and UI - Consumer decisions are driven by the multimedia experience - Drives smart phone market • Enhanced user experience - Games - Theater-like audio for video playback - Streaming media services - Mobile TV • Availability of third party applications - Consumers expected to purchase $35 billion of mobile applications by 2014

• Use case driven design - For ease of use • Comprehensive feature set - Including advanced audio and streaming media • Consistent APIs - Same APIs for both hardware and software implementations • Full, predictable functionality - No need to guess available or what is going to work • Portable - No need to port application for different platforms

Khronos APIs Open Source Framework • Conformance Tests • Free to change • Portable • Every implementation unique • IP Protection • GPL license • Profiles & Components • Plugins • Predictable set of functionality • Functionality dependent on implementation

• Low-level interface for audio, video, and imaging codecs • Developed for embedded and mobile devices • Enables interfacing with multimedia components in a unified manner • Allows components to be any combination of hardware or software completely transparently to the media chain • The goal of OpenMAX IL is to abstract multimedia components - Combat the problem of integration portability - Consistent functionality

• Enables arbitrary multimedia pipelines by plugging blocks together - Componentized architecture abstracts multimedia functionality block interfaces • Wide variety of building blocks for imaging, video and audio functions - Encode, decode, apply an effect, capture, render, split, mix, etc • Enables blocks from different sources to work together - Blocks can be implemented in software or hardware

Portable & Re-usable streaming media building blocks

• Standardized component interfaces enable flexible media graphs • Includes multi-stream synchronization • Allows for custom plug-ins

OpenMAX Component Interfaces Enables components to be flexibly E.g. send dance beats to 3D connected in any graph topology application for visualization

AAC Audio Audio Audio Decoder Plug-in Renderer Speakers

*.mp4 / *.3gp Clock File Reader for AV Sync Time Data Video Video Video Scheduler Renderer Display Decoder MPEG4/ Decompressed H.264 Video Video

Example: MPEG-4 video synchronized with AAC audio decode © Copyright Khronos Group, 2011 - Page 8 &

Application Access Layer

© Copyright Khronos Group, 2011 - Page 9 The Need for Application Access • Hardware and low level access is powerful but complex - More power than most application developers require • Most application developers just want to playback and record media - Specifying where the content comes from - Specifying where the content should be rendered to - Manipulate a few playback controls - Have simple configurability • That are what OpenMAX AL and OpenSL ES provide… - Simple high-level multimedia APIs for playback and recording use cases - With hardware acceleration capabilities built in • …With Cross-platform portability - No need to rewrite an application for every platform

• Media playback and recording - Playback and recording of everything from video files to radio to camera input • Full range of video effects and controls – including playback rate, post processing, and image manipulation - Apply advanced effects to captured images and video before sending to friends • Complete metadata support - Read, Edit, and Set the metadata for most types of content • OpenMAX AL gives easy access to streaming media - Allowing time to be spent on developing applications, not using frameworks • Designed as a complete set of multimedia functionality - Unlike any other streaming media API

• Full range of effects and controls, including advanced 3D effects such as Doppler and virtualization - Experience rich, enhanced sound from locations other than the handset, even moving, for the ultimate multimedia experience • Create theater quality audio experience - In a mobile device! • Full 3D audio functionality enhances any gaming experience - Perfect companion to OpenGL ES • Designed for implementation by either a hardware or software solution - Take full advantage of system capabilities

• Simplified object-oriented streaming media - Built to enable common use cases - But also has access to advanced functionality • Media Objects enable PLAY and RECORD of content - Media Objects input and output to Devices - Perform some operation on an input and emit the result as output • Objects have control interfaces - Play, Record, Seek, Rate

Media Input Audio Mix

Camera (AL) Display Window (AL)

URI DSrc Player Object DSnk URI

Memory Memory

Content pipe Content pipe

Video 3D Audio playback Audio Video Playback recording Audio Effects Radio and RDS Audio Advanced Recording Camera MIDI

Basic Image capture Buffer MIDI & display queues

• Functionality divided into basic groups - Most common groupings • Profiles reduce fragmentation - Not all devices support all functionality - Only support the profiles you need • Applications can query available profiles - To adapt to the current device - Facilitates porting of applications • Profiles can be combined - To best suit any product • Optional features - Some features are optional in all profiles - APIs are consistent when feature is available

© Copyright Khronos Group, 2011 - Page 16 OpenSL ES and OpenMAX AL • Independent - But designed to work together - No dependency between the APIs – either one can exist by itself • Compatible - Working groups collaborated to make sure the APIs work together well • Consistent - Identical API architecture - Identical APIs for same functionality • Distinct - OpenMAX AL represents basic multimedia functionality (audio, video, and image) - OpenSL ES represents advanced audio functionality

URI DataSource Media Player Output Mix

File Reader Audio Audio Audio Audio Decoder Effect Mixer Sink Clock

Audio stream to DSP 3D Audio Rendering

Multimedia Application Processing on CPU

Video stream to GPU

OpenGL ES EGLStream GL_TEXURE_EXTERN OpenGL ES EGLSurface AL_OES texture

Much more flexibility than just “show a video”

Mobile 3D

• Better control of fuctionality and timing • Access to hardware acceleration • Multimedia is the primary reason for the NDK

• Increases application complexity • Requires compilation for each platform family - ARMv5TE - ARMv7-A - x86 instructions

• Earlier internal APIs not stable enough…

• Avoid fragmentation • Well tested and documented • Focus on functionality - Not on implementation • Access to hardware acceleration

• OpenGL ES 2.0 • EGL 1.4 • OpenSL ES 1.0.1 - Partial implementation Mobile 3D - Almost complete Phone + Music profiles - Buffer queues from 1.1 as Android extension - Google version compliant as far as implemented - Other vendors offer fully compliant solution

Graphics Context Advanced Audio Management

© Copyright Khronos Group, 2011 - Page 23 Ice Cream Sandwich – Android 4.0 • Same OS for both phones and tablets • Better multitasking • Better support for hardware accelerated graphics • DRM • Partial OpenMAX AL 1.0.1 implementation - Streaming use-case only - Does not interoperate with OpenSL ES

Streaming Media

• Android uses OpenMAX IL in their Multimedia framework - Codecs only - No graph building • Main Multimedia framework is still Stagefright - Introduced in Éclair • Allows for both software implemented and hardware accelerated codecs • Not part of NDK - Not intended to be accessible from applications

• Fler enheter - TV? - Netbooks? - Chrome OS?

• What does the future hold? - Compliant OpenSL ES? - Better OpenMAX AL support? - OpenMAX AL EGL Stream Extension? - Support for OpenCL? - Content Pipes?

• Its anybody’s guess!