Shifting Media app development into high gear Using virtual drivers to speed up development
Helen Koike [email protected] Open First Creative Commons Attribution-ShareAlike 4.0 Summary
● Classic V4L2 API → Vivid Driver ● Media API → Vimc Driver ● Codecs → Vicodec Driver
● Vimc: ● Submodules ● Configfs API ● Future work Oct 2018 2 Summary
● Classic V4L2 API → Vivid Driver ● Media API → Vimc Driver ● Codecs → Vicodec Driver
● Vimc: ● Submodules ● Configfs API ● Future work Oct 2018 3 Oct 2018 4 Sepia
Sensor Composer Scaler
Greyscale
Oct 2018 5 APP
User space
------/dev/video* ------
Kernel space Driver
------Physical device
Oct 2018 6 APP
Read/Write/MMAP User space
------/dev/video* ------
Driver Video stream
Kernel space
------Physical device
Oct 2018 7 APP
Read/Write/MMAP User space IOCTLs
------/dev/video* ------
Driver Standard Custom Video stream General configs Controls Controls * Img fmt * Contrast * DRV DEF 1 Kernel * Buffers * Brightness * DRV DEF 2 space * Video std * Gamma * DRV DEF 3 * Frame rate ......
------Physical device
Oct 2018 8 APP
User space
------/dev/video* ------
Kernel space Driver
------Physical device
Oct 2018 9 APP
User space
------/dev/video* ------
Kernel space Driver
------Physical device
Oct 2018 10 APP
User space
------/dev/video* ------
The Virtual Video Test Driver (VIVID)
Kernel space
------Physical device
Oct 2018 11 APP
User space
------/dev/video* ------
The Virtual Video Test Driver (VIVID)
Kernel space
------Physical device
Oct 2018 12 Vivid driver
Oct 2018 13 Vivid driver: current state
● Merged in 3.17 by Hans Verkuil ● Good coverage of the API ● Error injection ● Up to 4K resolution ● ... More info: https://linuxtv.org/downloads/v4l-dvb-apis/v4l-drivers/vivid.html
Oct 2018 14 Summary
● Classic V4L2 API → Vivid Driver ● Media API → Vimc Driver ● Codecs → Vicodec Driver
● Vimc: ● Submodules ● Configfs API ● Future work Oct 2018 15 Media API - Motivation
● Embedded systems: great variety of devices ● Hardware complexity ● Similar configuration in different components of the device
Oct 2018 16 contrast Sepia
Sensor Composer Scaler
Greyscale contrast
Oct 2018 17 APP
Read/Write/MMAP User space IOCTLs
------/dev/video* ------
Driver Standard Custom Video stream General configs Controls Controls * Img fmt * Contrast * DRV DEF 1 Kernel * Buffers * Brightness * DRV DEF 2 space * Video std * Gamma * DRV DEF 3 * Frame rate ......
------Physical device
Oct 2018 18 /dev/video0
Sepia
Sensor Composer Scaler
Greyscale
Oct 2018 19 /dev/video0
Sepia
Sensor Composer Scaler
Greyscale
Oct 2018 20 /dev/video0
Sepia /dev/v4l-subdev0 Sensor Composer Scaler /dev/v4l-subdev2
/dev/v4l-subdev1
Greyscale
Oct 2018 21 APP contrast Subdev API IOCTLs Sepia /dev/v4l- subdev0
APP R/W/MMAP APP IOCTLs IOCTLs Scaler /dev/v4l- /dev/video0 subdev2
APP
IOCTLs contrast Greyscale /dev/v4l- subdev1 Oct 2018 22 What is part of the device? What is the topology? APP
? ?
? ? ? ? ? ? ? ? ? ?
/dev/v4l-subdev0 /dev/v4l-subdev98 /dev/v4l-subdev11 /dev/v4l-subdev14 /dev/video2 /dev/video0 /dev/v4l-subdev6 /dev/video42 /dev/v4l-subdev21 /dev/v4l-subdev31 Oct 2018 23 /dev/video0
Sepia /dev/v4l-subdev0 Sensor Composer Scaler /dev/v4l-subdev2
/dev/v4l-subdev1
Greyscale
Oct 2018 24 /dev/video0
out
Sepia /dev/v4l-subdev0 Sensor Composer Scaler /dev/v4l-subdev2
/dev/v4l-subdev1
Greyscale
Oct 2018 25 APP contrast Media API IOCTLs Sepia /dev/v4l- subdev0
APP R/W/MMAP APP APP IOCTLs IOCTLs IOCTLs Scaler /dev/media0 /dev/v4l- /dev/video0 subdev2
APP
IOCTLs contrast Greyscale /dev/v4l- subdev1 Oct 2018 26 APP /dev/video0 IOCTLs out /dev/media0
Sepia /dev/v4l-subdev0 Sensor Composer Scaler /dev/v4l-subdev2
/dev/v4l-subdev1
Greyscale
Oct 2018 27 APP /dev/video0 IOCTLs out /dev/media0
Sepia /dev/v4l-subdev0 Sensor Composer Scaler /dev/v4l-subdev2
/dev/v4l-subdev1
Greyscale
Oct 2018 28 APP
User space
------/dev/media* ------
Kernel space Driver
------Physical device
Oct 2018 29 APP
User space
------/dev/media* ------
Kernel space Driver
------Physical device
Oct 2018 30 APP
User space
------/dev/media* ------
The Virtual Media Controller Driver (VIMC)
Kernel space
------Physical device
Oct 2018 31 APP
User space
------/dev/media* ------
The Virtual Media Controller Driver (VIMC)
Kernel space
?
------Physical device
Oct 2018 32 Vimc Driver
● Proposed by Laurent Pinchart for Outreachy in 2015 ● Merged in Kernel 4.12 ● Moving slowly
Oct 2018 33 Vimc Driver: current state
● Basic set of emulation units: – capture (no output e.g. HDMI/S-Video) – sensor (generates images in different formats) – debayer (converts bayer to non-bayer format) – scaler (scaling down is not supported) ● Several optimizations and controls are still missing ● Hard-coded topology (re-compilation required) Oct 2018 34 Oct 2018 35 Vimc Driver
Oct 2018 36 Summary
● Classic V4L2 API → Vivid Driver ● Media API → Vimc Driver ● Codecs → Vicodec Driver
● Vimc: ● Submodules ● Configfs API ● Future work Oct 2018 37 Codecs (encoder/decoder)
● Encoder: – raw → compressed (e.g. RGB → H.264) ● Decoder: – compressed → raw (e.g. H.264 → RGB)
Oct 2018 38 Stateful vs Stateless codecs
● Stateful codecs: – Driver keeps the current state (definition of state depends on the image format) – Requires /dev/video* ● Stateless codecs: – Userspace keeps the state sending it in every frame – Requires /dev/video* /dev/media* Oct 2018 39 APP
User space
------/dev/video* (/dev/media*) ------
The Virtual Codec Driver (Vicodec)
Kernel space
Encoder Decoder Encoder Decoder
------Physical device Encoder Decoder
Oct 2018 40 Vicodec Driver: current state
● Relatively new (4.18 by Tom aan de Wiel) ● FWHT codec (Fast Walsh Hadamard Transform) ● Supports stateful only ● Stateless support: proposed as an Outreachy project
Oct 2018 41 Use cases for virtual drivers
● Hardware is not ready / Board bring-up phase ● Convenient to make tests on a PC instead of an embedded device ● Test application in different hardware – e.g. Gstreamer
Oct 2018 42 Use cases for virtual drivers (cont)
● Loop back device – Adding layers of effects – Test streams ● Test APIs ● Reference code ● Improve compliance tests
Oct 2018 43 Summary
● Classic V4L2 API → Vivid Driver ● Media API → Vimc Driver ● Codecs → Vicodec Driver
● Vimc: ● Submodules ● Configfs API ● Future work Oct 2018 44 Developer APP User space
------VIMC Configfs API ------/dev/media0 ------/dev/media1 ------
Kernel space create
create
Oct 2018 45 Vimc: submodules
Subdev1.ko
Implementation of subdevice Subdev2.ko drivers are modularized and doesn’t need to alter Vimc’s core code.
SubdevN.ko
Oct 2018 46 vimc-sensor.ko vimc-scaler.ko vimc-debayer.ko Standard APP Developer vimc-capture.ko ...
Emulate different topologies to test the app in several scenarios
custom-sd1.ko custom-sd2-ko Manufacturer custom-sd3.ko ...
Oct 2018 47 Summary
● Classic V4L2 API → Vivid Driver ● Media API → Vimc Driver ● Codecs → Vicodec Driver
● Vimc: ● Submodules ● Configfs API ● Future work Oct 2018 48 Configfs API: Topology
‘Sensor A’ ● vimc-sensor.ko Entities 0 – Name – Submodule 0 ● ‘Debayer A’ Pads vimc-debayer.ko 1 – Source – Sink
0 ● ‘Raw Capture 0’ Links vimc-capture.ko
Oct 2018 49 mkdir “MEDIA_NAME”
Oct 2018 50 mkdir “SUBMOD:NAME”
‘Sensor A’ vimc-sensor.ko 0
0 ‘Debayer A’ vimc-debayer.ko 1
0 ‘Raw Capture 0’ vimc-capture.ko
Oct 2018 51 mkdir “NAME1:PAD->NAME2:PAD”
‘Sensor A’ vimc-sensor.ko 0
0 ‘Debayer A’ vimc-debayer.ko 1
0 ‘Raw Capture 0’ vimc-capture.ko
Oct 2018 52 Summary
● Classic V4L2 API → Vivid Driver ● Media API → Vimc Driver ● Codecs → Vicodec Driver
● Vimc: ● Submodules ● Configfs API ● Future work Oct 2018 53 Vimc Driver: future work
● API in Configfs (WIP) ● More controls: – LKCAMP – Linux kernel study group ● Image generation directly from capture node (WIP) – LKCAMP ● Frame rate control
Oct 2018 54 Vimc Driver: future work
● Optimizations of img processing in the pipeline – Calculate final format based on the full pipeline configuration – Multi-threaded processing ● Submodules for output: HDMI / S-Video ● More standard submodules ● Add more V4L2 mechanisms / controls / options
Oct 2018 55 Thank you!
Helen Koike [email protected] This work is licensed under https://creativecommons.org/licenses/by-sa/4.0/ Userspace tools
● v4l-utils media-ctl -d /dev/media0 --print-dot > /tmp/out.dot && xdot /tmp/out.dot media-ctl -d /dev/media0 -V "'Sensor A':0[fmt:RGB888_1X24/600x600]" media-ctl -v -d /dev/media0 --links "'Debayer A':1->'Scaler':0 [0]"
● Yavta (Yet Another V4L2 Test Application) yavta --format RGB24 --size 600x600 /dev/video0
Oct 2018 57 Summary
● Classic V4L2 API → Vivid Driver ● Media API → Vimc Driver ● Codecs → Vicodec Driver
● Vimc: ● Submodules ● Configfs API ● Future work Oct 2018 58