OpenMAX AL, OpenSL ES Native Multimedia in Android Erik Noreke Chair of OpenMAX AL and OpenSL ES Working Groups © Copyright Khronos Group, 2011 - Page 1 Why Create Khronos Multimedia APIs? Khronos APIs Open Source Framework • Conformance Tests • Free to change • Portable • Every implementation unique • IP Protection • GPL license • Profiles & Components • Plugins • Predictable set of functionality • Functionality dependent on implementation © Copyright Khronos Group, 2011 - Page 2 & Application Access Layer © Copyright Khronos Group, 2011 - Page 3 The Need for Application Access • Hardware and low level access is powerful but complex - More power than most application developers require • Most application developers just want to playback and record media - Specifying where the content comes from and is rendered to - Manipulate a few playback controls - Have simple configurability © Copyright Khronos Group, 2011 - Page 4 The Need for Application Access • That are what OpenMAX AL and OpenSL ES provide… - Simple high-level multimedia APIs for playback and recording use cases - With hardware acceleration capabilities built in • …With Cross-platform portability - No need to rewrite an application for every platform © Copyright Khronos Group, 2011 - Page 5 OpenMAX AL Focused on Streaming Media • Media playback and recording - Playback and recording of everything from video files to radio to camera input • Full range of video effects and controls – including playback rate, post processing, and image manipulation - Apply advanced effects to captured images and video before sending to friends © Copyright Khronos Group, 2011 - Page 6 OpenMAX AL Focused on Streaming Media • OpenMAX AL gives easy access to streaming media - Allowing time to be spent on developing applications, not using frameworks • Designed as a complete set of multimedia functionality - Unlike any other streaming media API © Copyright Khronos Group, 2011 - Page 7 OpenSL ES – Powerful Mobile Audio • Full range of effects and controls, including advanced 3D effects such as Doppler and virtualization - Experience rich, enhanced sound from locations other than the handset, even moving, for the ultimate multimedia experience • Create theater quality audio experience - In a mobile device! © Copyright Khronos Group, 2011 - Page 8 OpenSL ES – Powerful Mobile Audio • Full 3D audio functionality enhances any gaming experience - Perfect companion to OpenGL ES • Designed for implementation by either a hardware or software solution - Take full advantage of system capabilities © Copyright Khronos Group, 2011 - Page 9 Object Oriented Media Access • Simplified object-oriented streaming media - Built to enable common use cases - But also has access to advanced functionality • Media Objects enable PLAY and RECORD of content - Media Objects input and output to Devices - Perform some operation on an input and emit the result as output • Objects have control interfaces - Play, Record, Seek, Rate Media Input Audio Mix Camera (AL) Display Window (AL) URI DSrc Player Object DSnk URI Memory Memory Content pipe Content pipe © Copyright Khronos Group, 2011 - Page 10 OpenMAX AL & OpenSL ES © Copyright Khronos Group, 2011 - Page 11 OpenSL ES and OpenMAX AL • Working groups collaborated to define the common API functionality • Independent, Compatible, Consistent and Distinct Video 3D Audio playback Audio Video Playback recording Audio Effects Radio and RDS Audio Advanced Recording Camera MIDI Basic Image capture Buffer MIDI & display queues © Copyright Khronos Group, 2011 - Page 12 Why Profiles? • Functionality divided into basic groups - Most common groupings • Profiles reduce fragmentation - Not all devices support all functionality - Only support the profiles you need • Applications can query available profiles - To adapt to the current device - Facilitates porting of applications © Copyright Khronos Group, 2011 - Page 13 Why Profiles? • Profiles can be combined - To best suit any product • Optional features - Some features are optional in all profiles - APIs are consistent when feature is available © Copyright Khronos Group, 2011 - Page 14 OpenSL ES and OpenMAX AL • Independent - But designed to work together - No dependency between the APIs – either one can exist by itself • Compatible - Working groups collaborated to make sure the APIs work together well © Copyright Khronos Group, 2011 - Page 15 OpenSL ES and OpenMAX AL • Consistent - Identical API architecture - Identical APIs for same functionality • Distinct - OpenMAX AL represents basic multimedia functionality (audio, video, and image) - OpenSL ES represents advanced audio functionality © Copyright Khronos Group, 2011 - Page 16 Complete Multimedia Use-Case Audio stream to DSP 3D Audio Rendering Multimedia Application Processing on CPU Video stream to GPU OpenGL ES EGLStream GL_TEXURE_EXTERN OpenGL ES EGLSurface AL_OES texture Much more flexibility than just “show a video” © Copyright Khronos Group, 2011 - Page 17 Android NDK for Multimedia • NDK – Native Development Kit • Introduced in Android 2.3 Gingerbread Advanced Audio Mobile 3D © Copyright Khronos Group, 2011 - Page 18 Why the NDK? • Better control of fuctionality and timing • Access to hardware acceleration • Multimedia is the primary reason for the NDK • Increases application complexity • Requires compilation for each platform family - ARMv5TE - ARMv7-A - x86 instructions © Copyright Khronos Group, 2011 - Page 19 Focus is on Standardized APIs • Earlier internal APIs not stable enough… • Avoid fragmentation • Well tested and documented • Focus on functionality - Not on implementation • Access to hardware acceleration © Copyright Khronos Group, 2011 - Page 20 Native Multimedia APIs in Android 2.3 • OpenGL ES 2.0 • EGL 1.4 • OpenSL ES 1.0.1 - Partial implementation Mobile 3D - Almost complete Phone + Music profiles - Buffer queues from 1.1 as Android extension - Google version compliant as far as implemented - Other vendors offer fully compliant solution Graphics Context Advanced Audio Management © Copyright Khronos Group, 2011 - Page 21 Ice Cream Sandwich – Android 4.0 • Same OS for both phones and tablets • Better multitasking • Better support for hardware accelerated graphics • DRM • Partial OpenMAX AL 1.0.1 implementation - Streaming use-case only - Does not interoperate with OpenSL ES Streaming Media © Copyright Khronos Group, 2011 - Page 22 Android & OpenMAX IL • Android uses OpenMAX IL in their Multimedia framework - Codecs only - No graph building • Main Multimedia framework is still Stagefright - Introduced in Éclair • Allows for both software implemented and hardware accelerated codecs • Not part of NDK - Not intended to be accessible from applications © Copyright Khronos Group, 2011 - Page 23 What is Google saying? © Copyright Khronos Group, 2011 - Page 24 What is Google saying? © Copyright Khronos Group, 2011 - Page 25 What is Google saying? © Copyright Khronos Group, 2011 - Page 26 Where is Android Headed? • More Devices - TVs? - Netbooks? - Chrome OS? • What does the future hold? - Compliant OpenSL ES? - Better OpenMAX AL support? - OpenMAX AL EGL Stream Extension? - Support for OpenCL? - Content Pipes? © Copyright Khronos Group, 2011 - Page 27 Thank You © Copyright Khronos Group, 2011 - Page 28 .