Zero-Copy Video Streaming on Embedded Systems the Easy Way

Home , Dup (system call), Mesa (computer graphics), Video4Linux

Michael Tretter - [email protected] Embedded Linux Conference-Europe Philipp Zabel - [email protected] Oct 25, 2017

Slide 1 - © Pengutronix - http://www.pengutronix.de - 10/25/2017 Examples

● Presentation capture and streaming ● Augmented Reality ● UAV video downlink ● Intercom

CC BY-SA 4.0: Unknownlfcg – Own work

CC BY 4.0: kremlin.ru

Slide 2 - © Pengutronix - http://www.pengutronix.de - 10/25/2017 Agenda

● Video and graphics on embedded devices ● Hardware acceleration units and Zero-Copy buffer sharing ● Case study: i.MX6 ● The easy way ● Open Issues & Future Work

Slide 3 - © Pengutronix - http://www.pengutronix.de - 10/25/2017 Building Blocks

● Recording / Streaming ● Receiving / Projection / Compositing ● Lens correction / Warping ● Transcoding

CC BY-SA 4.0: DXR - Own work

Slide 4 - © Pengutronix - http://www.pengutronix.de - 10/25/2017 Embedded System Requirements

● Portable ● Soft real-time ● Energy efficient ● “High” data rates ● Lightweight

Limited processing power vs. Audio/Video use case

Slide 5 - © Pengutronix - http://www.pengutronix.de - 10/25/2017 Specialized Co-processors

● Graphics Processing Unit ● Supported or preferred format in ● Video encoder and decoder memory differ ● ● FPGA Copy and conversion between hardware units required ● Camera ● Display Controller ● Network Controller

Slide 6 - © Pengutronix - http://www.pengutronix.de - 10/25/2017 Zero-Copy

"Zero-copy" describes computer ● Copying in CPU is expensive operations in which the CPU does not ● CPU memory bandwidth smaller perform the task of copying data from than hardware acceleration units one memory area to another. ● CPU cache management (Wikipedia)

Slide 7 - © Pengutronix - http://www.pengutronix.de - 10/25/2017 Putting it all Together, Case study: i.MX6

● Memory bandwidth ● Up to quad-core Cortex A9, 1 GHz ● Up to 533 MHz DDR3 SDRAM ● CPU memcpy ~500 MiB/s (1066 MT/s @ 64-bit) ● 1080p30 YUYV: 120 MiB/s ● Realistically, up to 2.5 GiB/s on ● Cache management overhead i.MX6Q, more on i.MX6QP

Slide 8 - © Pengutronix - http://www.pengutronix.de - 10/25/2017 Putting it all Together, Case study: i.MX6

● GPU: Vivante GC2000 ● Display: IPUv3 display interface CPUs ● Camera: IPUv3 capture interface External ● GPU VPU: Chips&Media CODA960 Memory

Display AXI/ Interface AHB Capture Interface

Internal VPU SRAM

Slide 9 - © Pengutronix - http://www.pengutronix.de - 10/25/2017 Device Drivers and Interfaces

● GPU: Vivante GC2000 ● etnaviv (DRM) ● Display: IPUv3 display interface ● imx-drm (DRM, KMS) ● Camera: IPUv3 capture interface ● imx-media (Video4Linux2), staging ● VPU: Chips&Media CODA960 ● coda (Video4Linux)

● Userspace: Mesa/etnaviv (OpenGL)

● DMABuf

Slide 10 - © Pengutronix - http://www.pengutronix.de - 10/25/2017 Before DMABuf

Capture (IPUv3 CSI, V4L2) Encode (CODA960, V4L2) e c a p s R CPU copy W r e s U mmap mmap W R W l e n r e K

Slide 11 - © Pengutronix - http://www.pengutronix.de - 10/25/2017 DMABuf

Capture (IPUv3 CSI, V4L2) Encode (CODA960, V4L2) e c a p s r fd

e dup s U

W R W l e n r e K

Slide 12 - © Pengutronix - http://www.pengutronix.de - 10/25/2017 DMABuf

Decode (CODA960, V4L2) Display (IPUv3, DRM) e c a p s r fd

e dup s U

R W R l e n r e K

Slide 13 - © Pengutronix - http://www.pengutronix.de - 10/25/2017 Device Drivers and Interfaces (API)

● V4L2 ioctls: Import/export DMABuf handles VIDIOC_EXPBUF from/into video devices VIDIOC_QBUF

● DRM ioctls: Import/export DMABuf handles DRM_IOCTL_PRIME_HANDLE_TO_FD from/into GPU or display contoller DRM_IOCTL_PRIME_FD_TO_HANDLE devices, used by libdrm/Mesa

● EGL extensions: These sit on top of EGL_EXT_image_dma_buf_import DRM_IOCTL_PRIME_* EGL_EXT_image_dma_buf_import_modifiers EGL_MESA_image_dma_buf_export

Slide 14 - © Pengutronix - http://www.pengutronix.de - 10/25/2017 Device Drivers and Interfaces: V4L2

/* V4L2 DMABuf export */ /* V4L2 DMABuf import */ int video_fd = open("/dev/v4l/by-name/csi", int dmabuf_fd; O_RDWR); int video_fd = open("/dev/v4l/by-name/coda", O_RDWR); struct v4l2_requestbuffers reqbuf = { .count = 1, struct v4l2_requestbuffers reqbuf = { .type = V4L2_BUF_TYPE_VIDEO_CAPTURE, .count = 1, .memory = V4L2_MEMORY_MMAP, .type = V4L2_BUF_TYPE_VIDEO_OUTPUT, }; .memory = V4L2_MEMORY_DMABUF, ioctl(video_fd, VIDIOC_REQBUFS, &reqbuf); }; ioctl(video_fd, VIDIOC_REQBUFS, &reqbuf); struct v4l2_exportbuffer expbuf = { .type = V4L2_BUF_TYPE_VIDEO_CAPTURE, struct v4l2_buffer buf = { .index = 0, .type = V4L2_BUF_TYPE_VIDEO_OUTPUT, }; .memory = V4L2_MEMORY_DMABUF, ioctl(video_fd, VIDIOC_EXPBUF, &expbuf); .index = 0, .m.fd = dmabuf_fd, int dmabuf_fd = expbuf.fd; }; ioctl(video_fd, VIDIOC_QBUF, &buf); https://linuxtv.org/downloads/v4l-dvb-apis-new/uapi/v4l/vidioc-expbuf.html https://linuxtv.org/downloads/v4l-dvb-apis-new/uapi/v4l/dmabuf.html Slide 15 - © Pengutronix - http://www.pengutronix.de - 10/25/2017 Device Drivers and Interfaces: EGL/OpenGL ES

EGLint attrib_list[] = { int dmabuf_fd; EGL_WIDTH, 1920, int stride; EGL_HEIGHT, 1280, EGL_LINUX_DRM_FOURCC_EXT, eglExportDMABUFImageMESA( DRM_FORMAT_YUYV, egl_display, EGL_DMA_BUF_PLANE0_FD_EXT, dmabuf_fd, egl_image, EGL_DMA_BUF_PLANE0_FD_OFFSET_EXT, 0, &dmabuf_fd, EGL_DMA_BUF_PLANE0_FD_PITCH_EXT, 3840, &stride); EGL_NONE, }; EGLImageKHR egl_image = eglCreateImageKHR( egl_display, EGL_NO_CONTEXT, EGL_LINUX_DMA_BUF_EXT, NULL, attrib_list); glEGLImageTargetTexture2DOES( GL_TEXTURE_EXTERNAL_OES, egl_image); https://www.khronos.org/registry/EGL/extensions/EXT/EGL_EXT_image_dma_buf_import.txt https://www.khronos.org/registry/EGL/extensions/MESA/EGL_MESA_image_dma_buf_export.txt Slide 16 - © Pengutronix - http://www.pengutronix.de - 10/25/2017 The Easy Way

Slide 17 - © Pengutronix - http://www.pengutronix.de - 10/25/2017 GStreamer

“GStreamer is a library for constructing graphs of media-handling components. The applications it supports range from simple Ogg/Vorbis playback, audio/video streaming to complex audio (mixing) and video (non-linear editing) processing.

Applications can take advantage of advances in codec and filter technology transparently. Developers can add new codecs and filters by writing a simple plugin with a clean, generic interface.”

Slide 18 - © Pengutronix - http://www.pengutronix.de - 10/25/2017 GStreamer

gst-launch-1.0 playbin uri=file:///home/mtr/Videos/tears_of_steel_720p.mkv

playbin0 [>] current-uri="file:////home/mtr/Videos/tears_of_steel_720p.mkv" source=(GstFileSrc) source n-video=1 current-video=0 n-audio=1 current-audio=0 audio-sink=(GstPulseSink) pulsesink0 video-sink=(GstXvImageSink) xvimagesink0 video-stream-combiner=(GstInputSelector) inputselector0 audio-stream-combiner=(GstInputSelector) inputselector1 sample=((GstSample*) 0x5590b656c030)

GstPlaySink playsink [>] parent=(GstPlayBin) playbin0 flags=video+audio+text+soft-volume+deinterlace+soft-colorbalance sample=((GstSample*) 0x5590b656c110) video-sink=(GstXvImageSink) xvimagesink0 audio-sink=(GstPulseSink) pulsesink0 send-event-mode=first

GstBin abin [>] parent=(GstPlaySink) playsink

GstPlaySinkAudioConvert GstPulseSink aconv pulsesink0 [>] [>] parent=(GstBin) abin parent=(GstBin) abin GstQueue use-converters=TRUE enable-last-sample=FALSE aqueue audio/x-raw device="alsa_output.pci-0000_00_1b.0.analog-stereo" [>] format: F32LE current-device="alsa_output.pci-0000_00_1b.0.analog-stereo" parent=(GstBin) abin audio/x-raw layout: interleaved device-name="Built-in Audio Analog Stereo" current-level-buffers=44 format: F32LE GstAudioConvert GstAudioResample audio/x-raw rate: 44100 current-level-bytes=360448 layout: interleaved audio/x-raw conv audio/x-raw resample format: F32LE channels: 2 current-level-time=1022000000 rate: 44100 format: F32LE [>] format: F32LE [>] layout: interleaved src channel-mask: 0x0000000000000003 sink silent=TRUE channels: 2 layout: interleaved parent=(GstPlaySinkAudioConvert) aconv layout: interleaved parent=(GstPlaySinkAudioConvert) aconv sink rate: 44100 [>][bfb] [ >][ bfb] channel-mask: 0x0000000000000003 [>][bfb] rate: 44100 rate: 44100 channels: 2 channels: 2 channels: 2 sink src channel-mask: 0x0000000000000003 proxypad14 proxypad13 channel-mask: 0x0000000000000003 sink src channel-mask: 0x0000000000000003 sink src [>][bfb] audio/x-raw [ >][ bfb] [>][bfb][T] [>][bfb] [>][bfb] [ >][ bfb] [>][bfb] [ >][ bfb] format: F32LE layout: interleaved rate: 44100 channels: 2 GstIdentity identity audio/x-raw channel-mask: 0x0000000000000003 format: F32LE [>] proxypad15 parent=(GstPlaySinkAudioConvert) aconv layout: interleaved [>][bfb] rate: 44100 signal-handoffs=FALSE channels: 2 sink channel-mask: 0x0000000000000003 [>][bfb] sink src [>][bfb] [ >][ bfb] GstInputSelector inputselector0 video/x-raw [>] format: I420 parent=(GstPlayBin) playbin0 video/x-raw width: 1280 n-pads=1 GstURIDecodeBin format: I420 height: 534 uridecodebin0 video/x-raw active-pad=(GstSelectorPad) sink_0 GstBin width: 1280 interlace-mode: progressive GstStreamSynchronizer [>] format: I420 vbin height: 534 pixel-aspect-ratio: 1/1 streamsynchronizer0 parent=(GstPlayBin) playbin0 width: 1280 video/x-raw [>] [>] uri="file:////home/mtr/Videos/tears_of_steel_720p.mkv" height: 534 sink_0 interlace-mode: progressive chroma-site: mpeg2 format: I420 parent=(GstPlaySink) playsink parent=(GstPlaySink) playsink source=(GstFileSrc) source interlace-mode: progressive running-time=41000000 pixel-aspect-ratio: 1/1 colorimetry: bt709 width: 1280 caps=video/x-raw(ANY); audio/x-raw(ANY); text/x-raw(ANY); subpicture/x-dvd; subpictur… pixel-aspect-ratio: 1/1 t ags=(( Gst TagList *) 0x7fa4c40d7f20) chroma-site: mpeg2 framerate: 24/1 height: 534 chroma-site: mpeg2 active=TRUE src colorimetry: bt709 audio/x-raw interlace-mode: progressive GstQueue GstPlaySinkVideoConvert GstXvImageSink framerate: 24/1 format: F32LE sink_0 src_1 pixel-aspect-ratio: 1/1 GstBin vqueue vconv xvimagesink0 GstDecodeBin colorimetry: bt709 always- ok=FALSE [>][bfb] proxypad7 [>] [>] [>] audio/x-raw GstTee layout: interleaved [>][bfb] [>][bfb] chroma-site: mpeg2 vdbin decodebin0 video/x-raw framerate: 24/1 [>][bfb] [>][bfb] audiotee parent=(GstBin) vbin parent=(GstBin) vbin parent=(GstBin) vbin format: F32LE rate: 44100 colorimetry: bt709 [>] video/x-raw [>] format: I420 [>] sink_1 src_0 parent=(GstPlaySink) playsink format: I420 current-level-buffers=1 video/x-raw use-converters=TRUE max-lateness=20000000 parent=(GstURIDecodeBin) uridecodebin0 video_sink layout: interleaved parent=(GstPlaySink) playsink channels: 2 framerate: 24/1 video/x-raw current-level-bytes=1224960 use-balance=TRUE qos=TRUE GstFileSrc width: 1280 [>][bfb] [>][bfb] width: 1280 format: I420 video/x-raw caps=video/x-raw(ANY); audio/x-raw(ANY); text/x-raw(ANY); subpicture/x-dvd; subpictur… [>][bfb] rate: 44100 num-src-pads=1 channel-mask: 0x0000000000000003 format: I420 current-level-time=41000000 last-sample=((GstSample*) 0x5590b656c1f0) source height: 534 audio/x-raw format: I420 GstInputSelector channels: 2 width: 1280 height: 534 max-size-buffers=3 width: 1280 device-name="Intel(R) Textured Video" [>] format: F32LE sink interlace-mode: progressive inputselector1 proxypad6 video/x-raw GstVideoConvert video/x-raw video/x-raw height: 534 interlace-mode: progressive max-size-bytes=0 height: 534 width: 1280 double-buffer=FALSE parent=(GstURIDecodeBin) uridecodebin0 layout: interleaved channel-mask: 0x0000000000000003 [>][bfb] video/x-raw video/x-raw video/x-raw video/x-raw video/x-raw pixel-aspect-ratio: 1/1 [>] [>][bfb] format: I420 vdconv format: I420 GstDeinterlace format: I420 pixel-aspect-ratio: 1/1 max-size-time=0 interlace-mode: progressive GstVideoConvert height: 534 colorkey=526352 location="//home/mtr/Videos/tears_of_steel_720p.mkv" audio/mpeg audio/mpeg audio/mpeg src_0 rate: 44100 sink src_0 interlace-mode: progressive format: I420 format: I420 format: I420 format: I420 format: I420 chroma-site: mpeg2 audio/x-raw parent=(GstPlayBin) playbin0 width: 1280 [>] width: 1280 deinterlace width: 1280 pixel-aspect-ratio: 1/1 chroma-site: mpeg2 silent=TRUE pixel-aspect-ratio: 1/1 conv GstVideoScale GstVideoBalance GstVideoConvert interlace-mode: progressive window-width=1280 mpegversion: 4 mpegversion: 4 mpegversion: 4 [ >][ bfb] n-pads=1 channels: 2 audio_sink [>][bfb] [>][bfb] proxypad11 width: 1280 [>] width: 1280 width: 1280 width: 1280 width: 1280 colorimetry: bt709 format: F32LE height: 534 parent=(GstBin) vdbin height: 534 [>] height: 534 chroma-site: mpeg2 colorimetry: bt709 chroma-site: mpeg2 scale videobalance conv2 pixel-aspect-ratio: 1/1 window-height=534 GstTypeFindElement framed: true framed: true framed: true active-pad=(GstSelectorPad) sink_0 channel-mask: 0x0000000000000003 [>][bfb] proxypad10 height: 534 parent=(GstPlaySinkVideoConvert) vconv height: 534 height: 534 height: 534 height: 534 avdec_aac audio/x-raw src_0 framerate: 24/1 proxypad4 [>][bfb] interlace-mode: progressive qos=TRUE interlace-mode: progressive parent=(GstBin) vdbin interlace-mode: progressive framerate: 24/1 colorimetry: bt709 [>] [>] [>] chroma-site: mpeg2 src ANY sink typefind stream-format: raw stream-format: raw stream-format: raw layout: interleaved colorimetry: bt709 [ >][ bfb] interlace-mode: progressive qos=TRUE interlace-mode: progressive interlace-mode: progressive interlace-mode: progressive interlace-mode: progressive GstMatroskaDemux GstAacParse avdec_aac0 format: F32LE [ >][ bfb] [ >][ bfb] pixel-aspect-ratio: 1/1 pixel-aspect-ratio: 1/1 pixel-aspect-ratio: 1/1 src sink src framerate: 24/1 sink parent=(GstPlaySinkVideoConvert) vconv parent=(GstPlaySinkVideoConvert) vconv parent=(GstPlaySinkVideoConvert) vconv colorimetry: bt709 [ <][ bfb] [ <][ bfb] [>] matroskademux0 codec_data: 1210 codec_data: 1210 aacparse0 codec_data: 1210 audio/x-raw rate: 44100 framerate: 24/1 pixel-aspect-ratio: 1/1 pixel-aspect-ratio: 1/1 qos=TRUE pixel-aspect-ratio: 1/1 qos=TRUE pixel-aspect-ratio: 1/1 qos=TRUE pixel-aspect-ratio: 1/1 [>] layout: interleaved chroma-site: mpeg2 chroma-site: mpeg2 chroma-site: mpeg2 [>][bfb] [ >][ bfb] [>][bfb][T] [>][bfb] src framerate: 24/1 sink parent=(GstDecodeBin) decodebin0 [>] level: 2 level: 2 [>] level: 2 src_1 format: F32LE channels: 2 sink_0 sink chroma-site: mpeg2 chroma-site: mpeg2 chroma-site: mpeg2 chroma-site: mpeg2 chroma-site: mpeg2 parent=(GstDecodeBin) decodebin0 rate: 44100 src_1 src colorimetry: bt709 sink src colorimetry: bt709 sink src colorimetry: bt709 [ >][ bfb] [>][bfb] proxypad0 caps=video/x-matroska parent=(GstDecodeBin) decodebin0 base-profile: lc GstMultiQueue base-profile: lc parent=(GstDecodeBin) decodebin0 base-profile: lc [ >][ bfb] layout: interleaved channel-mask: 0x0000000000000003 running-time=1230000000 proxypad12 [ >][ bfb] proxypad8 colorimetry: bt709 sink src colorimetry: bt709 colorimetry: bt709 colorimetry: bt709 colorimetry: bt709 [ <][ bfb] channels: 2 rate: 44100 [ >][ bfb] [>][bfb] framerate: 24/1 [ >][ bfb] [ >][ bfb] framerate: 24/1 [>][bfb] [>][bfb] framerate: 24/1 ANY profile: lc multiqueue0 profile: lc profile: lc t ags=(( Gst TagList *) 0x7fa4b80676d0) [>][bfb] [>][bfb] framerate: 24/1 [ >][ bfb] [ >][ bfb] framerate: 24/1 sink src framerate: 24/1 sink src framerate: 24/1 sink src framerate: 24/1 proxypad9 [>] channel-mask: 0x0000000000000003 proxypad3 channels: 2 audio/x-matroska channels: 2 channels: 2 channels: 2 sink src proxypad5 active=TRUE [ >][ bfb] [>][bfb] [>][bfb] [>][bfb] [ >][ bfb] [ >][ bfb] [ >][ bfb] sink src video/x-matroska audio_0 rate: 44100 parent=(GstDecodeBin) decodebin0 sink src rate: 44100 [ >][ bfb] channel-mask: 0x0000000000000003 rate: 44100 [ >][ bfb] [>][bfb] video/x-h264 [ >][ bfb] always- ok=FALSE [<][bfb][t] [<][bfb] video/x-matroska-3d [ >][ bfb] max-size-bytes=2097152 [>][bfb] [>][bfb] a u d io / w e b m sink max-size-time=0 level: 3.1 video/x-raw proxypad2 [>][bfb] v i d eo / w e b m [<][bfb][T] format: I420 GstIdentity ANY video_0 video/x-h264 video/x-h264 video/x-h264 profile: main [ >][ bfb] identity width: 1280 [ >][ bfb] level: 3.1 level: 3.1 level: 3.1 codec_data: 014d401fffe1001d674d40... [>] profile: main sink_1 src_1 profile: main profile: main stream-format: avc height: 534 parent=(GstPlaySinkVideoConvert) vconv GstCapsFilter avdec_h264 codec_data: 014d401fffe1001d674d40... [ >][ bfb] [>][bfb][T] codec_data: 014d401fffe1001d674d40... interlace-mode: progressive signal-handoffs=FALSE codec_data: 014d401fffe1001d674d40... GstH264Parse capsfilter0 alignment: au stream-format: avc stream-format: avc avdec_h264-0 pixel-aspect-ratio: 1/1 stream-format: avc h264parse0 [>] width: 1280 [>] alignment: au sink_0 src_0 alignment: au alignment: au height: 534 chroma-site: mpeg2 [ >][ bfb] [>][bfb][T] [>] parent=(GstDecodeBin) decodebin0 parent=(GstDecodeBin) decodebin0 width: 1280 width: 1280 parent=(GstDecodeBin) decodebin0 width: 1280 caps=video/x-h264, alignment=(string)au, stream-format=(string){ avc, byte-stream }, … pixel-aspect-ratio: 1/1 colorimetry: bt709 sink src height: 534 height: 534 height: 534 framerate: 24/1 framerate: 24/1 [>][bfb] [ >][ bfb] pixel-aspect-ratio: 1/1 pixel-aspect-ratio: 1/1 pixel-aspect-ratio: 1/1 interlace-mode: progressive sink src framerate: 24/1 framerate: 24/1 sink src framerate: 24/1 sink src parsed: true [ >][ bfb] [ >][ bfb] [>][bfb] [>][bfb] interlace-mode: progressive [ >][ bfb] [>][bfb] parsed: true

Legend Element-States: [~] void-pending, [0] null, [-] ready, [=] paused, [>] playing Pad-Activation: [-] none, [>] push, [<] pull Pad-Flags: [b]locked, [f]lushing, [b] locking, [E ]OS; upper-case is set Pad-Task: [T] has start ed task, [t ] has paused task

Slide 19 - © Pengutronix - http://www.pengutronix.de - 10/25/2017 GStreamer

● Sink support ● Language bindings ● Wayland ● C++ ● WebRTC ● Python ● QML ● Rust ● ... ● ...

● Plugins ● Autoplugging ● V4L2 ● decodebin ● OpenGL ● encodebin ● Third party ● playsink ● ... ● ...

Slide 20 - © Pengutronix - http://www.pengutronix.de - 10/25/2017 Video4Linux2

● Elements: v4l2sink, v4l2videodec, v4l2videoenc, ... ● Support DMABuf import and export ● Recent feature: stable element names

● Nicolas Dufresne - Implementing Zero-Copy pipelines in GStreamer ● GStreamer Conference 2017 ● https://gstconf.ubicast.tv/videos/zero-copy-pipelines-in-gstreamer/

● Kernel subsystem for video cards ● Element: kmssink ● API and user space library ● Import DMABuf automatically ● Output via video card ● Depends on features of kms driver (e.g., no scaling on i.MX6)

Simple tool for testing

● Display server protocol ● Element: waylandsink ● DMABuf: linux_dmabuf_unstable_v1 ● Your mileage may vary ● Compositor decides ● Depends on compositor ● OpenGL upload for compositing ● Imported format might not be ● Display as overlay supported

● Camera: v4l2src ● CODA encode: v4l2h264enc

Capture VPU packetize Interface encode and stream

gst-launch-1.0 v4l2src io-mode=dmabuf1 device=/dev/v4l/by-name/csi ! \ v4l2h264enc output-io-mode=dmabuf-import2 ! \ rtph264pay ! \ udpsink

1 Still necessary in GStreamer 1.12, automatic in master 2 Still necessary, will be auto-negotiated in the future

● CODA decode: v4l2h264dec ● GPU, display: waylandsink

receive and VPU GPU Display extract decode composition Interface

gst-launch-1.0 udpsrc ! application/x-rtp,payload=96 ! \ rtph264depay ! \ h264parse ! \ v4l2h264dec1 io-mode=dmabuf2 ! \ waylandsink

1 Stable element names in master, for 1.12: v4l2videodec device=/dev/videoX 2 Still necessary in GStreamer 1.12, automatic in master

● Media input pipeline ● Currently needs manual configuration: media-ctl ● Pavel Machek: Cheap Complex Cameras (http://sched.co/ByYH)

media-ctl --links "'tc358743 1-000f':0->'imx6-mipi-csi2':0[1]" media-ctl --links "'imx6-mipi-csi2':1->'ipu1_csi0_mux':0[1]" media-ctl --links "'ipu1_csi0_mux':2->'ipu1_csi0':0[1]" media-ctl --links "'ipu1_csi0':2->'ipu1_csi0 capture':0[1]" media-ctl --set-dv "'tc358743 1-000f':0" media-ctl --set-v4l2 "'tc358743 1-000f':0[fmt:UYVY8_1X16/1920x1080]" media-ctl --set-v4l2 "'imx6-mipi-csi2':1[fmt:UYVY8_1X16/1920x1080]" media-ctl --set-v4l2 "'ipu1_csi0_mux':2[fmt:UYVY8_1X16/1920x1080]" media-ctl --set-v4l2 "'ipu1_csi0':0[fmt:UYVY8_1X16/1920x1080@1/60]" media-ctl --set-v4l2 "'ipu1_csi0':2[fmt:AYUV32/1920x1080@1/30]"

● Useful default media-controller configuration ● Mesa/etnaviv ● NV12 and YUYV texture import (GL_TEXTURE_2D) ● Direct sampling from linear buffers ● OpenCL support ● Weston: Atomic modesetting patchset for overlay plane support

● Camera pipeline configuration → Autoconfiguration? Device-tree default? ● Remaining proprietay blob: CODA VPU firmware ● V4l2 access as root → Pipewire?

● Modern embedded system use various coprocessors ● DMABuf is usable abstraction for zero-copy on Linux ● Let GStreamer manage all the ugly details

● Know your hardware ● Be aware of corner cases ● Check resulting GStreamer pipeline ● Zero-copy between driver blobs problematic or impossible → Avoid blobs!

● Questions?