Creating Digital Products Part 2 : Software

Cambridge Game Creators London – 2015-05-21

Laurent Pinchart [email protected] Problem Definition Problem Definition Problem Definition Linux System Boot Loader Up & Running Kernel Initialization

0 1 4 15 time

Graphics Use Cases Fixed Logo

Boot Loader Fixed Logo

Linux Kernel Animation, Feedback

System Initialization Full Graphics Stack

Up & Running Traditional Linux Graphics Architecture Qt GTK+ EFL Apps

Xlib

Xorg X server

DDX

Mesa 3D

libdrm

FBDEV DRM/KMS Kernel

Hardware

X11 Graphics Stack ● DDX – Device Dependent X driver. Handles initialization, manages the display, performs 2D accelerated rendering. ● DRM – Direct Rendering Manager. Subsystem of the Linux kernel responsible for interfacing with GPUs. ● EFL, GTK+, Qt – Open-source graphical toolkits. ● EGL – Interface between rendering APIs and the underlying native platform windowing system. ● FBDEV – Frame Buffer Device. Subsystem of the Linux kernel responsible for accessing the display framebuffer. ● GLES – GL Embedded System. Embedded profile of the OpenGL API. ● KMS – Kernel Mode Setting. Subsystem of the Linux kernel responsible for interfacing with display controllers. Handles mode setting and hardware composition. ● Mesa (3D) – Open-source implementation of the EGL, GL, GLES, … APIs. ● Xlib – Reference implementation of the client side of the X11 protocol. ● Xorg – Reference implementation of the server side of the system.

Acronyms X11 Client . . . X11 Client

Compositor

X Server

FBDEV evdev DRM/KMS

X11 Composition X11 Client . . . X11 Client

Compositor

X Server

1

FBDEV evdev DRM/KMS

X11 Composition X11 Client . . . X11 Client

2 3

Compositor

X Server

1

FBDEV evdev DRM/KMS

X11 Composition X11 Client . . . X11 Client

2 3 4 Compositor

X Server

1

FBDEV evdev DRM/KMS

X11 Composition X11 Client . . . X11 Client

2 3 4 Compositor 5 X Server

1

FBDEV evdev DRM/KMS

X11 Composition X11 Client . . . X11 Client

2 3 4 Compositor 5 X Server

1 6

FBDEV evdev DRM/KMS

X11 Composition Linux Kernel Display APIs FBDEV vs. DRM

APIs Frame Buffer Display (memory) Device

Device Model – FBDEV Planes CRTC Encoder Connector (memory)

Device Model – DRM/KMS Source: http://xkcd.com/619/ Features DRM FB

Dynamic Allocation Yes No

Multiple Buffers Yes panning

Import dmabuf No

dmabuf Export mmap mmap

Memory Management DRM FB RGB Formats 4CC 4CC

Enumeration Planes No

Negotiation No No

Atomicity Yes No

Mode Setting DRM FB

Overlays Yes No

Rotation Yes No

Scaling Yes No

Cropping/Panning Yes Yes

Transformations g tin 8 t 1 F e . U s .2 B e 2 s A d s x r o e fe M m c u M f D n in T u , ic fe L T B s , , b e m & S n to c M M m a R E M u l A yn D G K D P S

2000 2008 2009 2011 2012 2013-...

Activity – DRM/KMS 4 .9 .3 1 x u ts in a m , L g r V in o E k F D n C B la C F B 4

1996 2000 2012

Activity – FBDEV ts a rm o F C C 4

2012

Source: http://valdodge.com/2010/07/ Activity – FBDEV 3000

2500

2000

1500 drm fb drm+bug 1000

500

0

Mailing List Traffic 2000

1800

1600

1400

1200 drm 1000 fb 800

600

400

200 fb

0 drm

Cumulative Changes - API 70000

60000

50000

40000 drm fb 30000

20000

10000

fb

0 drm

Cumulative Changes - Core 1800000

1600000

1400000

1200000

1000000 drm

800000 fb

600000

400000

200000 fb

0 drm

Cumulative Changes - Drivers Source: http://xkcd.com/927/ Use Cases Use Cases - FBDEV (that's it...)

Use Cases - FBDEV Everything

Use Cases – DRM/KMS Beyond X11 X11 Client . . . X11 Client

2 3 4 Compositor 5 X Server

1 6

DRM/KMS evdev FBDEV

X11 Composition by Shmuel Csaba Otto Traian; GNU FDL 1.2+ and CC-BY-SA 3.0+; created 2013-08-28; updated 2014-02-27

GNOME Video

Clutter 1.17.4 W2EGL EGL glue libwayland-client code

Inkscape

GTK+ 3.12 W2EGL EGL glue libwayland-client code

KDE Plasma

Qt 5.3 W2EGL EGL glue libwayland-client code

EGL

libwayland-server

Wayland Compositor libwayland-EGL libEGL-mesa-drivers Weston, Mutter, KWin, Clayland, libEGL-mesa EGL libGBM Enlightenment, ... D

3 OpenGL|ES libGLES-mesa

libOpenVG-mesa libinput a OpenVG s e

M Graphics device drivers libGL-mesa-DRI ① ④ ioctl libDRM

evdev KMS DRM Kernel

CPU & main memory

USB, PS/2, ... GPU & graphic memory framebuffer Wayland Graphics Stack Wayland Wayland Client . . . Client

2 3

Wayland Compositor

1 4

evdev DRM/KMS

Wayland Composition Single GUI Application

Toolkit (Qt, SDL, ...)

DRM/KMS evdev FBDEV

Custom Application The DRM and KMS APIs DRM ? KMS

APIs ● Memory Management ● Command Stream ● Vertical Blanking ● Version, Authentication, Master, ...

DRM ● Device Model ● Frame Buffer ● Modes ● Page Flip ● Planes ● Cursor, Gamma, ...

KMS drm kms armada exynos i810 i915 mga msm nouveau omap qxl r128 radeon savage sis tegra via

DRM/KMS API Planes CRTC Encoder Connector (memory)

Connector

Encoder Connector

Device Model Planes CRTC Encoder Connector (memory)

Encoder Connector

Memory SoC Off-Chip

Device Model - SoC Frame Buffer

KMS – Scanout Root Plane

Composition

Overlay Plane(s)

KMS – Composition Frame Buffer

● width ● height GEM ● format Object(s) ● pitches CRTC ● offsets

Memory Properties

KMS – Frame Buffer Frame Buffer

GEM Object

● width ● height CRTC ● bpp Memory ● pitch ● size

Properties

DRM/KMS – GEM Object Process A Process B 3 Send FD SCM_RIGHTS

Local Global Global Local 1 2 4 Handle FD FD Handle

GEM Object

DRM – Handles sync back porch

active area Active Area

front porch

KMS – Modes (1/2) hdisplay hsync_start hsync_end

htotal sync back porch

active area Active Area

front porch

KMS – Modes (2/2) fb x mode

m

o

d

e

. Active v y d

i

Area s p crtc

l

a

y

mode.hdisplay *connectors struct drm_mode_set { num_connectors struct drm_framebuffer *fb; struct drm_crtc *crtc; struct drm_display_mode *mode; uint32_t x; uint32_t y; struct drm_connector **connectors; size_t num_connectors; };

KMS – Mode Setting KMS Programming Example Code Ahead

Error handling omitted for readability

Disclaimer #include #include #include #include int main(int argc, char **argv) { return 0; }

Skeleton /* * Open * * #include * * int drmOpen(const char *name, const char *busid); */ int fd; fd = drmOpen("imx-drm", NULL);

Open /* * Get resources * * #include * * drmModeResPtr drmModeGetResources(int fd); * void drmModeFreeResources(drmModeResPtr ptr); */ drmModeResPtr resources; uint32_t crtc_id; uint32_t connector_id; resources = drmModeGetResources(fd); crtc_id = resources->crtcs[0]; connector_id = resources->connectors[0];

Get Resources /* * Get modes * * #include * * drmModeConnectorPtr drmModeGetConnector(int fd, * uint32_t connectorId); * void drmModeFreeConnector(drmModeConnectorPtr ptr); */ drmModeConnectorPtr connector; drmModeModeInfo mode; uint32_t width; uint32_t height; connector = drmModeGetConnector(fd, connector_id); mode = connector->modes[0]; width = mode.hdisplay; height = mode.vdisplay;

Get Modes /* * Create buffer * * #include * #include * * #define DRM_IOCTL_MODE_CREATE_DUMB ... * * int drmIoctl(int fd, unsigned long request, void *arg); */ struct drm_mode_create_dumb bo_create; memset(&bo_create, 0, sizeof(bo_create)); bo_create.bpp = 32; bo_create.width = width; bo_create.height = height; drmIoctl(fd, DRM_IOCTL_MODE_CREATE_DUMB, &bo_create);

Create Buffer /* * Fill buffer * * #include * #include * #include * * #define DRM_IOCTL_MODE_MAP_DUMB ... * * int drmIoctl(int fd, unsigned long request, void *arg); */ struct drm_mode_map_dumb bo_map; memset(&bo_map, 0, sizeof(bo_map)); bo_map.handle = bo_create.handle; drmIoctl(fd, DRM_IOCTL_MODE_MAP_DUMB, &bo_map);

Fill Buffer (1/2) /* * Fill buffer * * #include * * void *mmap(void *addr, size_t length, int prot, * int flags, int fd, off_t offset) * int munmap(void *addr, size_t length); */ void *mem; mem = mmap(0, bo_create.size, PROT_READ | PROT_WRITE, MAP_SHARED, fd, bo_map.offset); fill_pattern(DRM_FORMAT_ARGB8888, mem, width, height, bo_create.pitch); munmap(mem, bo_create.size);

Fill Buffer (2/2) /* * Create frame buffer * * #include * #include * * int drmModeAddFB2(int fd, uint32_t width, * uint32_t height, uint32_t pixel_format, * uint32_t bo_handles[4], uint32_t pitches[4], * uint32_t offsets[4], uint32_t *buf_id, * uint32_t flags); */ uint32_t handles[4] = { bo_create.handle, }; uint32_t pitches[4] = { bo_create.pitch, }; uint32_t offsets[4] = { 0, }; uint32_t fb_id; drmModeAddFB2(fd, width, height, DRM_FORMAT_ARGB8888, handles, pitches, offsets, &fb_id, 0);

Create Frame Buffer /* * Set the mode * * #include * * int drmModeSetCrtc(int fd, uint32_t crtcId, * uint32_t bufferId, uint32_t x, uint32_t y, * uint32_t *connectors, int count, * drmModeModeInfoPtr mode); */ drmModeSetCrtc(fd, crtc_id, fb_id, 0, 0, &connector_id, 1, &mode);

Set the Mode /* * Get plane resources * * #include * * drmModePlaneResPtr drmModeGetPlaneResources(int fd); * void drmModeFreePlaneResources(drmModePlaneResPtr ptr); */ drmModePlaneResPtr planes; uint32_t plane_id; planes = drmModeGetPlaneResources(fd); plane_id = planes->planes[0];

Get Plane Resources /* * Set the plane * * #include * * int drmModeSetPlane(int fd, * uint32_t plane_id, uint32_t crtc_id, * uint32_t fb_id, uint32_t flags, * uint32_t crtc_x, uint32_t crtc_y, * uint32_t crtc_w, uint32_t crtc_h, * uint32_t src_x, uint32_t src_y, * uint32_t src_w, uint32_t src_h); */ drmModeSetPlane(fd, plane_id, crtc_id, fb_id, 0, width / 4, height / 4, width / 2, height / 2, 0, 0, (width / 2) << 16, (height / 2) << 16);

Set the Plane /* * List plane properties * * #include * #include * * drmModeObjectPropertiesPtr drmModeObjectGetProperties( * int fd, uint32_t object_id, uint32_t object_type); * void drmModeFreeObjectProperties( * drmModeObjectPropertiesPtr ptr); */ drmModeObjectPropertiesPtr properties; properties = drmModeObjectGetProperties(fd, plane_id, DRM_MODE_OBJECT_PLANE);

List Plane Properties /* * Find alpha property * * #include * * drmModePropertyPtr drmModeGetProperty(int fd, * uint32_t propertyId); * void drmModeFreeProperty(drmModePropertyPtr ptr); */ drmModePropertyPtr property; unsigned int i; for (i = 0; i < properties->count_props; ++i) { property = drmModeGetProperty(fd, properties->props[i]); if (!strcmp(property->name, "alpha")) break; }

Find Alpha Property /* * Turn global transparency on * * #include * * int drmModeObjectSetProperty(int fd, uint32_t object_id, * uint32_t object_type, uint32_t property_id, * uint64_t value); */ drmModeObjectSetProperty(fd, plane_id, DRM_MODE_OBJECT_PLANE, property->prop_id, 128);

Turn Transparency On ● Atomic Update ● Page Flip ● Cursor ● Events ● ...

And Much More... 3D Acceleration by Shmuel Csaba Otto Traian; GNU FDL 1.3+ and CC-BY-SA 3.0+; created 2014-03-02, last updated 2014-03-02

geomery texture sound OpenGL commands and/or 3D shaders written in GLSL data data data (vertex, tesselation control, tessellation evaluation, computer geometry, fragment and compute shaders) game game engine

system calls subroutine SDL calls Mesa 3D Library Subroutines

GNU C Library Subroutines DRM Library Subroutines

System Call Interface (SCI)

other Process Memory Linux kernel KMS DRM scheduler manager subsystems Linux kernel

System Display Graphics Hardware CPU Devices GPU RAM Screen controller RAM

OpenGL Applications

Mesa

libGL u t n a s 0 o 5 5 e a 0 e 1 1 v r 2 9 9 d u i i r w a o s r n

Mesa Classic Applications

Mesa libGL

EGL OpenCL VDPAU OpenGL State Trackers

gallium o e v n n i p i e o 5 v r . p e 1 . a Pipe . d 9 d n m i e t

a Drivers v e r e l r l f

Gallium 3D by Shmuel Csaba Otto Traian; GNU FDL 1.3+ and CC-BY-SA 4.0 international; created 2014-03-23; modified 2014-03-25 by Matthew Raymond

Wayland-client X11-client or GLX libwayland-client libX or libXCB 3D Game Engine X-server X.Org Server

- DIX driver d n

a wlglamor DDX driver l y t

a n Glamor OpenGL 3.3 w e

b i OpenGL-based 2D rendering acceleration library,

i l translates "X rendering" into OpenGL and EGL l c OpenGL 4.4 EGL OpenGL EGL or EGL libwayland-server OpenGL EGL OpenGL OpenVG 4.4 1.5 | ES 3.0 1.1 Wayland Compositor libEGL-mesa EGL GLX libGL-mesa-glx Weston, Mutter, KWin, Clayland, libGBM Enlightenment, ... D 3 OpenGL|ES 2.0 libGL-nvidia-glx OpenGL 3.3

a OpenVGlibGLES-mesa libGL-fglrx-glx

s libGL-mesa-swx11 libinput libOpenVG-mesa e

M libGL-mesa-DRI User space graphics drivers Proprietary user space graphics drivers ① ④ ioctl

libDRM integrate libDRM into driver?

evdev KMS DRM Linux kernel proprietary BLOB

CPU & main memory USB, PS/2, ... Display controller framebuffer GPU & graphic memory

Free vs. Proprietary Free Proprietary

Performance +(+) ++(+)

Features +(+) ++

Security + - open/ Kernel open closed vendor Evolutions yes lock-in

Free vs. Proprietary i.MX6 i.MX6 ● http://nouveau.freedesktop.org/wiki/ ● http://limadriver.org/ ● https://github.com/freedreno ● https://github.com/kusma/tegra-re ● https://github.com/etnaviv/etna_viv ● https://github.com/hermanhermitage/ videocoreiv/

Reverse Engineering Reverse Engineering Reverse Engineering commit b13a0a9e2922cbfbf7b400f4f0fd0acd19f941df Author: Alexandre Courbot Date: Sat Jul 26 18:41:41 2014 +0900

drm/nouveau/gk20a: reclocking support

Add support for reclocking on GK20A, using a statically-defined pstates table. The algorithms for calculating the coefficients and setting the clocks are directly taken from the ChromeOS kernel.

Signed-off-by: Alexandre Courbot Signed-off-by: Ben Skeggs

Reverse Engineering Assembling the Pieces Distributions Distributions Distributions Meta-Distributions Credit: http://free-electrons.com/pub/conferences/2012/lsm/buildroot/ Meta-Distributions ● FBDEV, DRM/KMS ● Bare, X11, Wayland ● Mainline, BSP ● Distribution

Summary Resources • http://free-electrons.com/docs/ • http://phd.mupuf.org/ • [email protected]

Resources [email protected]

Contact Thank you.