Shifting Media app development into high gear Using virtual drivers to speed up development Helen Koike [email protected] Open First Creative Commons Attribution-ShareAlike 4.0 Summary ● Classic V4L2 API → Vivid Driver ● Media API → Vimc Driver ● Codecs → Vicodec Driver ● Vimc: ● Submodules ● Configfs API ● Future work Oct 2018 2 Summary ● Classic V4L2 API → Vivid Driver ● Media API → Vimc Driver ● Codecs → Vicodec Driver ● Vimc: ● Submodules ● Configfs API ● Future work Oct 2018 3 Oct 2018 4 Sepia Sensor Composer Scaler Greyscale Oct 2018 5 APP User space -------------------------------------------------------- /dev/video* -------------------------------------------------------- Kernel space Driver ---------------------------------------------------------------------------------------------------------------------------------- Physical device Oct 2018 6 APP Read/Write/MMAP User space -------------------------------------------------------- /dev/video* -------------------------------------------------------- Driver Video stream Kernel space ---------------------------------------------------------------------------------------------------------------------------------- Physical device Oct 2018 7 APP Read/Write/MMAP User space IOCTLs -------------------------------------------------------- /dev/video* -------------------------------------------------------- Driver Standard Custom Video stream General configs Controls Controls * Img fmt * Contrast * DRV DEF 1 Kernel * Buffers * Brightness * DRV DEF 2 space * Video std * Gamma * DRV DEF 3 * Frame rate . ---------------------------------------------------------------------------------------------------------------------------------- Physical device Oct 2018 8 APP User space -------------------------------------------------------- /dev/video* -------------------------------------------------------- Kernel space Driver ---------------------------------------------------------------------------------------------------------------------------------- Physical device Oct 2018 9 APP User space -------------------------------------------------------- /dev/video* -------------------------------------------------------- Kernel space Driver ---------------------------------------------------------------------------------------------------------------------------------- Physical device Oct 2018 10 APP User space -------------------------------------------------------- /dev/video* -------------------------------------------------------- The Virtual Video Test Driver (VIVID) Kernel space ---------------------------------------------------------------------------------------------------------------------------------- Physical device Oct 2018 11 APP User space -------------------------------------------------------- /dev/video* -------------------------------------------------------- The Virtual Video Test Driver (VIVID) Kernel space ---------------------------------------------------------------------------------------------------------------------------------- Physical device Oct 2018 12 Vivid driver Oct 2018 13 Vivid driver: current state ● Merged in 3.17 by Hans Verkuil ● Good coverage of the API ● Error injection ● Up to 4K resolution ● ... More info: https://linuxtv.org/downloads/v4l-dvb-apis/v4l-drivers/vivid.html Oct 2018 14 Summary ● Classic V4L2 API → Vivid Driver ● Media API → Vimc Driver ● Codecs → Vicodec Driver ● Vimc: ● Submodules ● Configfs API ● Future work Oct 2018 15 Media API - Motivation ● Embedded systems: great variety of devices ● Hardware complexity ● Similar configuration in different components of the device Oct 2018 16 contrast Sepia Sensor Composer Scaler Greyscale contrast Oct 2018 17 APP Read/Write/MMAP User space IOCTLs -------------------------------------------------------- /dev/video* -------------------------------------------------------- Driver Standard Custom Video stream General configs Controls Controls * Img fmt * Contrast * DRV DEF 1 Kernel * Buffers * Brightness * DRV DEF 2 space * Video std * Gamma * DRV DEF 3 * Frame rate . ---------------------------------------------------------------------------------------------------------------------------------- Physical device Oct 2018 18 /dev/video0 Sepia Sensor Composer Scaler Greyscale Oct 2018 19 /dev/video0 Sepia Sensor Composer Scaler Greyscale Oct 2018 20 /dev/video0 Sepia /dev/v4l-subdev0 Sensor Composer Scaler /dev/v4l-subdev2 /dev/v4l-subdev1 Greyscale Oct 2018 21 APP contrast Subdev API IOCTLs Sepia /dev/v4l- subdev0 APP R/W/MMAP APP IOCTLs IOCTLs Scaler /dev/v4l- /dev/video0 subdev2 APP IOCTLs contrast Greyscale /dev/v4l- subdev1 Oct 2018 22 What is part of the device? What is the topology? APP ? ? ? ? ? ? ? ? ? ? ? ? /dev/v4l-subdev0 /dev/v4l-subdev98 /dev/v4l-subdev11 /dev/v4l-subdev14 /dev/video2 /dev/video0 /dev/v4l-subdev6 /dev/video42 /dev/v4l-subdev21 /dev/v4l-subdev31 Oct 2018 23 /dev/video0 Sepia /dev/v4l-subdev0 Sensor Composer Scaler /dev/v4l-subdev2 /dev/v4l-subdev1 Greyscale Oct 2018 24 /dev/video0 out Sepia /dev/v4l-subdev0 Sensor Composer Scaler /dev/v4l-subdev2 /dev/v4l-subdev1 Greyscale Oct 2018 25 APP contrast Media API IOCTLs Sepia /dev/v4l- subdev0 APP R/W/MMAP APP APP IOCTLs IOCTLs IOCTLs Scaler /dev/media0 /dev/v4l- /dev/video0 subdev2 APP IOCTLs contrast Greyscale /dev/v4l- subdev1 Oct 2018 26 APP /dev/video0 IOCTLs out /dev/media0 Sepia /dev/v4l-subdev0 Sensor Composer Scaler /dev/v4l-subdev2 /dev/v4l-subdev1 Greyscale Oct 2018 27 APP /dev/video0 IOCTLs out /dev/media0 Sepia /dev/v4l-subdev0 Sensor Composer Scaler /dev/v4l-subdev2 /dev/v4l-subdev1 Greyscale Oct 2018 28 APP User space -------------------------------------------------------- /dev/media* -------------------------------------------------------- Kernel space Driver ---------------------------------------------------------------------------------------------------------------------------------- Physical device Oct 2018 29 APP User space -------------------------------------------------------- /dev/media* -------------------------------------------------------- Kernel space Driver ---------------------------------------------------------------------------------------------------------------------------------- Physical device Oct 2018 30 APP User space -------------------------------------------------------- /dev/media* -------------------------------------------------------- The Virtual Media Controller Driver (VIMC) Kernel space ---------------------------------------------------------------------------------------------------------------------------------- Physical device Oct 2018 31 APP User space -------------------------------------------------------- /dev/media* -------------------------------------------------------- The Virtual Media Controller Driver (VIMC) Kernel space ? ---------------------------------------------------------------------------------------------------------------------------------- Physical device Oct 2018 32 Vimc Driver ● Proposed by Laurent Pinchart for Outreachy in 2015 ● Merged in Kernel 4.12 ● Moving slowly Oct 2018 33 Vimc Driver: current state ● Basic set of emulation units: – capture (no output e.g. HDMI/S-Video) – sensor (generates images in different formats) – debayer (converts bayer to non-bayer format) – scaler (scaling down is not supported) ● Several optimizations and controls are still missing ● Hard-coded topology (re-compilation required) Oct 2018 34 Oct 2018 35 Vimc Driver Oct 2018 36 Summary ● Classic V4L2 API → Vivid Driver ● Media API → Vimc Driver ● Codecs → Vicodec Driver ● Vimc: ● Submodules ● Configfs API ● Future work Oct 2018 37 Codecs (encoder/decoder) ● Encoder: – raw → compressed (e.g. RGB → H.264) ● Decoder: – compressed → raw (e.g. H.264 → RGB) Oct 2018 38 Stateful vs Stateless codecs ● Stateful codecs: – Driver keeps the current state (definition of state depends on the image format) – Requires /dev/video* ● Stateless codecs: – Userspace keeps the state sending it in every frame – Requires /dev/video* /dev/media* Oct 2018 39 APP User space --------------------------------------------- /dev/video* (/dev/media*) ---------------------------------------------- The Virtual Codec Driver (Vicodec) Kernel space Encoder Decoder Encoder Decoder ---------------------------------------------------------------------------------------------------------------------------------- Physical device Encoder Decoder Oct 2018 40 Vicodec Driver: current state ● Relatively new (4.18 by Tom aan de Wiel) ● FWHT codec (Fast Walsh Hadamard Transform) ● Supports stateful only ● Stateless support: proposed as an Outreachy project Oct 2018 41 Use cases for virtual drivers ● Hardware is not ready / Board bring-up phase ● Convenient to make tests on a PC instead of an embedded device ● Test application in different hardware – e.g. Gstreamer Oct 2018 42 Use cases for virtual drivers (cont) ● Loop back device – Adding layers of effects – Test streams ● Test APIs ● Reference code ● Improve compliance tests Oct 2018 43 Summary ● Classic V4L2 API → Vivid Driver ● Media API → Vimc Driver ● Codecs → Vicodec Driver ● Vimc: ● Submodules ● Configfs API ● Future work Oct 2018 44 Developer APP User space --------------- VIMC Configfs API --------------------- /dev/media0 --------------------- /dev/media1 ----------- Kernel space create create Oct 2018 45 Vimc: submodules Subdev1.ko Implementation of subdevice Subdev2.ko drivers are modularized and doesn’t need to alter Vimc’s core code. SubdevN.ko Oct 2018 46 vimc-sensor.ko vimc-scaler.ko vimc-debayer.ko Standard APP Developer vimc-capture.ko