What’s new in GStreamer
GUADEC, Strasbourg 26 July 2014
Tim-Philipp Müller
• who are we ?
• what is GStreamer ? What is GStreamer ?
• set of libraries
• pipeline-based: elements, components, pads, data flow
• plugins, abstract API
• often wrap other libraries (decoders, encoders, filters, etc.) What is GStreamer ? (cont'd)
• low-level API and high-level API
• playbin, encodebin, gst-editing-services, gst-rtsp-server
• (and farstream, telepathy etc.)
• integration with other frameworks and projects
• e.g. GNOME, Enlightenment, WebKit, Firefox, Clutter, Android/iOS/OSX/Windows
• goal is to adapt to platform/framework (inputs, outputs, decoders, DSPs/GPUs etc.) What we are going to talk about
• what happened in the last year with GStreamer ?
• what's next ? Releases? Releases!
• stable releases 1.2 (September 2013) and 1.4 (July 2014) • 4 1.2 bugfix releases, had 10 for 1.0 → things are settling and stablizing • 1.5→1.6 development started
• as always: Android/iOS/OSX/Windows binaries at the same time as source releases PiTiVi
• 1.x based GNonLin and gst-editing-services releases
• PiTiVi beta release based on that • including GTK+3 goodness
• it seems to work for people! and new non-PiTiVi GES users
• also: PiTiVi crowdfunding project with the help of the GNOME Foundation gst-editing-services? Or how to make non-linear editing apps simple
• library based on GStreamer and heavily using GNonLin
• provides high-level abstraction of timelines, clips and operations to apply to them
• think of PiTiVi UI elements as non-UI, object oriented API (but can do more than that!) OpenGL, the journey to the 3rd dimension
• OpenGL (and GLES) support merged into gst-plugins-bad
• supported platforms: ALL!
• provides video sink element But that's not all! (OpenGL cont'd)
• GL shader based effects and filters • new ones just need shader code to be written
• mixer, overlay, colorspace conversion and scaling elements
• library API for integrating GStreamer into existing GL frameworks • yes, also Clutter (* proof of concept), Qt or SDL or integration into applications that use GL themselves
• ever wanted to render videos in GL? here's your solution Moar OpenGL (OpenGL cont'd)
• plays nicely with hardware codecs
• Raspberry Pi (OpenMAX IL)
• Android MediaCodec (WIP)
• VAAPI + Clutter → hardware accelerated video playback in Totem!
Finally you can use the hardware codecs on your ThinkPad! Where are my devices?
• new API for device probing & discovery
• what are devices anyway? cameras, microphones, speakers, ...
• list them, get notified about new ones and how to use them Replacement for GstPropertyProbe
• old API was tedious to use, not high-level enough
• did not provide enough information
• application had to know about platform specifics • ALSA, PulseAudio, OSS, ...
• no notifications, poll based GstDeviceMonitor
• create instance and set up filters • which type of devices, which media formats?
• get notifications about new ones and all that are there already
• provides a fully configured GStreamer element
• missing: • implementation for all platforms and subsystems • how to model things like CDs and DVDs More and Better Hardware Support
• VAAPI → Clutter → Totem as mentioned already
• video encoders on Android, soon zerocopy decoding • before you ask: no iOS because no suitable APIs
• Video4Linux2 based decoders and filters (and encoders WIP) • Samsung Exynos (odroid) and others
• OpenNI2 cameras (Kinect)
• gst-omx 1.2 released for OpenMAX IL integration New Codecs!
• H265 aka HEVC support • beware of the patent trolls
• VP9 support • used together with Opus in new WebM on e.g. YouTube
• experimental Daala support • new next next generation video codec by Xiph.org and Mozilla people Real-time communication & the Web
• WebRTC seems to be the next big Web thing
• allows low-latency, peer-to-peer audio/video streaming • just using a web browser
• defines JavaScript API and signalling
• supported by Chrome and Firefox (and soon Safari?) Real-time communication & the Web (cont'd)
• big, enterprisy server-side solution: Kurento • based on GStreamer • can do everything Real-time communication & the Web (cont'd)
• client-side only some proof of concept GStreamer implementations • all the basics are there • integration mostly blocked by missing integration of DTLS support in GLib
• help! For those who don't like C
• gst-python 1.2 release • overrides for PyGI to provide more Pythonic APIs
• Vala has 1.4 support
• gstreamer-sharp 1.x release • GI based, static bindings for the CLR
• in general: lots of annotation improvements for GI and new, more bindings friendly APIs • works from GJS too! see gnome-sound-recorder Other Stuff
• lots of little and bigger things, some examples
• playbin properties for custom audio/video filters
• more elements ported from 0.10, especially for OS X
• DVB S2/T2 support and many improvements to DVB Other Stuff (cont'd)
• major RTP/RTSP, adaptive streaming and MPEG TS improvements
• Wayland support WIP
• lots of performance improvements everywhere in the core
• more than 1800 bug reports resolved in the last year What's Next? Stereoscopic Video, or how to support 3D TVs
• all the basic infrastructure is there
• only signalling and integration is missing but work in progress
• conversion 2D←→3D, red/green, ...
• hopefully ready in time for 1.6 Dead technologies: Bluray
• maybe this time for real?
• anybody interested in this? i heard physical media are dead?
• basic playback not much work • can use existing libraries also used by VLC
• menus and everything much, much work and insane requirements Polishing
• documentation, documentation, documentation!
• also tests, tests, tests! • more unit/integration tests, also see gst-validate
• polish all the rough edges to make everything work better together
• OpenGL, VAAPI, Clutter and Totem • more seamless and efficient hardware integration in general • make proper use of our new APIs everywhere • improve bindings support even more • lots of low-hanging fruits Polishing (cont'd)
• all the little things people are struggling with
• more convenience APIs and fixing inconsistent behaviour
• abstract generic concepts into more base classes • aggregator (mixer), demuxer, muxer, overlay Better Windows Support? Oh no! The Enemy!
• fix the few known, bad bugs in existing elements
• performance improvements
• make use of new Windows APIs • MediaFoundation and DXVA
• allow compilation with MS Visual Studio Any other wishes?
... or want to help? don't know about GStreamer but want to learn?
Talk to us! Questions? Thank You!
PS: GStreamer conference in Düsseldorf, October 16-17
Pictures Zeitung Flugzeug News by lavanderiadesign Old Books by skittledog Road Ahead by Florian