What’s new in GStreamer GUADEC, Strasbourg 26 July 2014 Tim-Philipp Müller <[email protected]> Sebastian Dröge <[email protected]> Introduction • who are we ? • what is GStreamer ? What is GStreamer ? • set of libraries • pipeline-based: elements, components, pads, data flow • plugins, abstract API • often wrap other libraries (decoders, encoders, filters, etc.) What is GStreamer ? (cont'd) • low-level API and high-level API • playbin, encodebin, gst-editing-services, gst-rtsp-server • (and farstream, telepathy etc.) • integration with other frameworks and projects • e.g. GNOME, Enlightenment, WebKit, Firefox, Clutter, Android/iOS/OSX/Windows • goal is to adapt to platform/framework (inputs, outputs, decoders, DSPs/GPUs etc.) What we are going to talk about • what happened in the last year with GStreamer ? • what's next ? Releases? Releases! • stable releases 1.2 (September 2013) and 1.4 (July 2014) • 4 1.2 bugfix releases, had 10 for 1.0 → things are settling and stablizing • 1.5→1.6 development started • as always: Android/iOS/OSX/Windows binaries at the same time as source releases PiTiVi • 1.x based GNonLin and gst-editing-services releases • PiTiVi beta release based on that • including GTK+3 goodness • it seems to work for people! and new non-PiTiVi GES users • also: PiTiVi crowdfunding project with the help of the GNOME Foundation gst-editing-services? Or how to make non-linear editing apps simple • library based on GStreamer and heavily using GNonLin • provides high-level abstraction of timelines, clips and operations to apply to them • think of PiTiVi UI elements as non-UI, object oriented API (but can do more than that!) OpenGL, the journey to the 3rd dimension • OpenGL (and GLES) support merged into gst-plugins-bad • supported platforms: ALL! • provides video sink element But that's not all! (OpenGL cont'd) • GL shader based effects and filters • new ones just need shader code to be written • mixer, overlay, colorspace conversion and scaling elements • library API for integrating GStreamer into existing GL frameworks • yes, also Clutter (* proof of concept), Qt or SDL or integration into applications that use GL themselves • ever wanted to render videos in GL? here's your solution Moar OpenGL (OpenGL cont'd) • plays nicely with hardware codecs • Raspberry Pi (OpenMAX IL) • Android MediaCodec (WIP) • VAAPI + Clutter → hardware accelerated video playback in Totem! Finally you can use the hardware codecs on your ThinkPad! Where are my devices? • new API for device probing & discovery • what are devices anyway? cameras, microphones, speakers, ... • list them, get notified about new ones and how to use them Replacement for GstPropertyProbe • old API was tedious to use, not high-level enough • did not provide enough information • application had to know about platform specifics • ALSA, PulseAudio, OSS, ... • no notifications, poll based GstDeviceMonitor • create instance and set up filters • which type of devices, which media formats? • get notifications about new ones and all that are there already • provides a fully configured GStreamer element • missing: • implementation for all platforms and subsystems • how to model things like CDs and DVDs More and Better Hardware Support • VAAPI → Clutter → Totem as mentioned already • video encoders on Android, soon zerocopy decoding • before you ask: no iOS because no suitable APIs • Video4Linux2 based decoders and filters (and encoders WIP) • Samsung Exynos (odroid) and others • OpenNI2 cameras (Kinect) • gst-omx 1.2 released for OpenMAX IL integration New Codecs! • H265 aka HEVC support • beware of the patent trolls • VP9 support • used together with Opus in new WebM on e.g. YouTube • experimental Daala support • new next next generation video codec by Xiph.org and Mozilla people Real-time communication & the Web • WebRTC seems to be the next big Web thing • allows low-latency, peer-to-peer audio/video streaming • just using a web browser • defines JavaScript API and signalling • supported by Chrome and Firefox (and soon Safari?) Real-time communication & the Web (cont'd) • big, enterprisy server-side solution: Kurento • based on GStreamer • can do everything Real-time communication & the Web (cont'd) • client-side only some proof of concept GStreamer implementations • all the basics are there • integration mostly blocked by missing integration of DTLS support in GLib • help! For those who don't like C • gst-python 1.2 release • overrides for PyGI to provide more Pythonic APIs • Vala has 1.4 support • gstreamer-sharp 1.x release • GI based, static bindings for the CLR • in general: lots of annotation improvements for GI and new, more bindings friendly APIs • works from GJS too! see gnome-sound-recorder Other Stuff • lots of little and bigger things, some examples • playbin properties for custom audio/video filters • more elements ported from 0.10, especially for OS X • DVB S2/T2 support and many improvements to DVB Other Stuff (cont'd) • major RTP/RTSP, adaptive streaming and MPEG TS improvements • Wayland support WIP • lots of performance improvements everywhere in the core • more than 1800 bug reports resolved in the last year What's Next? Stereoscopic Video, or how to support 3D TVs • all the basic infrastructure is there • only signalling and integration is missing but work in progress • conversion 2D←→3D, red/green, ... • hopefully ready in time for 1.6 Dead technologies: Bluray • maybe this time for real? • anybody interested in this? i heard physical media are dead? • basic playback not much work • can use existing libraries also used by VLC • menus and everything much, much work and insane requirements Polishing • documentation, documentation, documentation! • also tests, tests, tests! • more unit/integration tests, also see gst-validate • polish all the rough edges to make everything work better together • OpenGL, VAAPI, Clutter and Totem • more seamless and efficient hardware integration in general • make proper use of our new APIs everywhere • improve bindings support even more • lots of low-hanging fruits Polishing (cont'd) • all the little things people are struggling with • more convenience APIs and fixing inconsistent behaviour • abstract generic concepts into more base classes • aggregator (mixer), demuxer, muxer, overlay Better Windows Support? Oh no! The Enemy! • fix the few known, bad bugs in existing elements • performance improvements • make use of new Windows APIs • MediaFoundation and DXVA • allow compilation with MS Visual Studio Any other wishes? ... or want to help? don't know about GStreamer but want to learn? Talk to us! Questions? Thank You! PS: GStreamer conference in Düsseldorf, October 16-17 Pictures Zeitung Flugzeug News by lavanderiadesign Old Books by skittledog Road Ahead by Florian.