OpenMAX AL, OpenSL ES Native Multimedia in Android
Erik Noreke Chair of OpenMAX AL and OpenSL ES Working Groups
© Copyright Khronos Group, 2011 - Page 1 Why Create Khronos Multimedia APIs?
Khronos APIs Open Source Framework • Conformance Tests • Free to change • Portable • Every implementation unique • IP Protection • GPL license • Profiles & Components • Plugins • Predictable set of functionality • Functionality dependent on implementation
© Copyright Khronos Group, 2011 - Page 2 &
Application Access Layer
© Copyright Khronos Group, 2011 - Page 3 The Need for Application Access • Hardware and low level access is powerful but complex - More power than most application developers require • Most application developers just want to playback and record media - Specifying where the content comes from and is rendered to - Manipulate a few playback controls - Have simple configurability
© Copyright Khronos Group, 2011 - Page 4 The Need for Application Access • That are what OpenMAX AL and OpenSL ES provide… - Simple high-level multimedia APIs for playback and recording use cases - With hardware acceleration capabilities built in • …With Cross-platform portability - No need to rewrite an application for every platform
© Copyright Khronos Group, 2011 - Page 5 OpenMAX AL Focused on Streaming Media
• Media playback and recording - Playback and recording of everything from video files to radio to camera input • Full range of video effects and controls – including playback rate, post processing, and image manipulation - Apply advanced effects to captured images and video before sending to friends
© Copyright Khronos Group, 2011 - Page 6 OpenMAX AL Focused on Streaming Media
• OpenMAX AL gives easy access to streaming media - Allowing time to be spent on developing applications, not using frameworks • Designed as a complete set of multimedia functionality - Unlike any other streaming media API
© Copyright Khronos Group, 2011 - Page 7 OpenSL ES – Powerful Mobile Audio
• Full range of effects and controls, including advanced 3D effects such as Doppler and virtualization - Experience rich, enhanced sound from locations other than the handset, even moving, for the ultimate multimedia experience • Create theater quality audio experience - In a mobile device!
© Copyright Khronos Group, 2011 - Page 8 OpenSL ES – Powerful Mobile Audio
• Full 3D audio functionality enhances any gaming experience - Perfect companion to OpenGL ES • Designed for implementation by either a hardware or software solution - Take full advantage of system capabilities
© Copyright Khronos Group, 2011 - Page 9 Object Oriented Media Access
• Simplified object-oriented streaming media - Built to enable common use cases - But also has access to advanced functionality • Media Objects enable PLAY and RECORD of content - Media Objects input and output to Devices - Perform some operation on an input and emit the result as output • Objects have control interfaces - Play, Record, Seek, Rate
Media Input Audio Mix
Camera (AL) Display Window (AL)
URI DSrc Player Object DSnk URI
Memory Memory
Content pipe Content pipe
© Copyright Khronos Group, 2011 - Page 10 OpenMAX AL & OpenSL ES
© Copyright Khronos Group, 2011 - Page 11 OpenSL ES and OpenMAX AL • Working groups collaborated to define the common API functionality • Independent, Compatible, Consistent and Distinct
Video 3D Audio playback Audio Video Playback recording Audio Effects Radio and RDS Audio Advanced Recording Camera MIDI
Basic Image capture Buffer MIDI & display queues
© Copyright Khronos Group, 2011 - Page 12 Why Profiles?
• Functionality divided into basic groups - Most common groupings • Profiles reduce fragmentation - Not all devices support all functionality - Only support the profiles you need • Applications can query available profiles - To adapt to the current device - Facilitates porting of applications
© Copyright Khronos Group, 2011 - Page 13 Why Profiles?
• Profiles can be combined - To best suit any product • Optional features - Some features are optional in all profiles - APIs are consistent when feature is available
© Copyright Khronos Group, 2011 - Page 14 OpenSL ES and OpenMAX AL • Independent - But designed to work together - No dependency between the APIs – either one can exist by itself • Compatible - Working groups collaborated to make sure the APIs work together well
© Copyright Khronos Group, 2011 - Page 15 OpenSL ES and OpenMAX AL • Consistent - Identical API architecture - Identical APIs for same functionality • Distinct - OpenMAX AL represents basic multimedia functionality (audio, video, and image) - OpenSL ES represents advanced audio functionality
© Copyright Khronos Group, 2011 - Page 16 Complete Multimedia Use-Case
Audio stream to DSP 3D Audio Rendering
Multimedia Application Processing on CPU
Video stream to GPU
OpenGL ES EGLStream GL_TEXURE_EXTERN OpenGL ES EGLSurface AL_OES texture
Much more flexibility than just “show a video”
© Copyright Khronos Group, 2011 - Page 17 Android NDK for Multimedia • NDK – Native Development Kit • Introduced in Android 2.3 Gingerbread Advanced Audio
Mobile 3D
© Copyright Khronos Group, 2011 - Page 18 Why the NDK?
• Better control of fuctionality and timing • Access to hardware acceleration • Multimedia is the primary reason for the NDK
• Increases application complexity • Requires compilation for each platform family - ARMv5TE - ARMv7-A - x86 instructions
© Copyright Khronos Group, 2011 - Page 19 Focus is on Standardized APIs
• Earlier internal APIs not stable enough…
• Avoid fragmentation • Well tested and documented • Focus on functionality - Not on implementation • Access to hardware acceleration
© Copyright Khronos Group, 2011 - Page 20 Native Multimedia APIs in Android 2.3
• OpenGL ES 2.0 • EGL 1.4 • OpenSL ES 1.0.1 - Partial implementation Mobile 3D - Almost complete Phone + Music profiles - Buffer queues from 1.1 as Android extension - Google version compliant as far as implemented - Other vendors offer fully compliant solution
Graphics Context Advanced Audio Management
© Copyright Khronos Group, 2011 - Page 21 Ice Cream Sandwich – Android 4.0 • Same OS for both phones and tablets • Better multitasking • Better support for hardware accelerated graphics • DRM • Partial OpenMAX AL 1.0.1 implementation - Streaming use-case only - Does not interoperate with OpenSL ES
Streaming Media
© Copyright Khronos Group, 2011 - Page 22 Android & OpenMAX IL
• Android uses OpenMAX IL in their Multimedia framework - Codecs only - No graph building • Main Multimedia framework is still Stagefright - Introduced in Éclair • Allows for both software implemented and hardware accelerated codecs • Not part of NDK - Not intended to be accessible from applications
© Copyright Khronos Group, 2011 - Page 23 What is Google saying?
© Copyright Khronos Group, 2011 - Page 24 What is Google saying?
© Copyright Khronos Group, 2011 - Page 25 What is Google saying?
© Copyright Khronos Group, 2011 - Page 26 Where is Android Headed?
• More Devices - TVs? - Netbooks? - Chrome OS?
• What does the future hold? - Compliant OpenSL ES? - Better OpenMAX AL support? - OpenMAX AL EGL Stream Extension? - Support for OpenCL? - Content Pipes?
© Copyright Khronos Group, 2011 - Page 27 Thank You
© Copyright Khronos Group, 2011 - Page 28