Gstreamer & Syslink

GStreamer & SysLink

Víctor Manuel Jáquez Leal

2011-10-22 Sat

Víctor Manuel Jáquez Leal () GStreamer & SysLink 2011-10-22 Sat 1 / 1 Outline

Víctor Manuel Jáquez Leal () GStreamer & SysLink 2011-10-22 Sat 2 / 1 Parallel computing

Serial computing is now dead. Parallel computing is a revolution that is now upon us. Although it started more than 40 years ago.

Víctor Manuel Jáquez Leal () GStreamer & SysLink 2011-10-22 Sat 3 / 1 Why Parallel computing?

To achieve performance guarantee. It generally leads to lower power consumption.

Víctor Manuel Jáquez Leal () GStreamer & SysLink 2011-10-22 Sat 4 / 1 Parallel computing challenges

Programming for serial computing is already dicult. Programming for parallel computing will only exacerbate this diculty.

Víctor Manuel Jáquez Leal () GStreamer & SysLink 2011-10-22 Sat 5 / 1 Asymmetric Multiprocessing Might be dierent type of processors. Might have private memory. Might run dierent OS.

Parallel computing approaches

Symmetric Multiprocessing Identical processors. A single shared main memory. A single OS instance.

Víctor Manuel Jáquez Leal () GStreamer & SysLink 2011-10-22 Sat 6 / 1 Parallel computing approaches

Symmetric Multiprocessing Asymmetric Multiprocessing Identical processors. Might be dierent type of A single shared main processors. memory. Might have private memory. A single OS instance. Might run dierent OS.

Víctor Manuel Jáquez Leal () GStreamer & SysLink 2011-10-22 Sat 6 / 1 Symmetric Multiprocessing SMP

It is the most common conguration. It has bandwidth issues. It lacks of fault-tolerant features. The cache coherency mechanism consumes too much power.

Víctor Manuel Jáquez Leal () GStreamer & SysLink 2011-10-22 Sat 7 / 1 Asymmetric Multiprocessing AMP

A problem in one kernel/CPU will not aect the whole system. Applications must use a dierent API. Requires more hardware resource management. Wider kernel memory consumption.

Víctor Manuel Jáquez Leal () GStreamer & SysLink 2011-10-22 Sat 8 / 1 Another AMP Conguration

One master processor. Multiple remote processors. slave processors are treated as devices (under /dev). One shared memory but registering private regions.

Víctor Manuel Jáquez Leal () GStreamer & SysLink 2011-10-22 Sat 9 / 1 Inter-Processors Communication IPC

An IPC mechanism is key for the AMP architecture. Sharing memory regions between processors. Data interchange. Passing messages between processors. Task synchronization.

Víctor Manuel Jáquez Leal () GStreamer & SysLink 2011-10-22 Sat 10 / 1 Examples of such multi-core SoCs

Texas Instruments's OMAP processors Qualcomm's SnapDragon processors Broadcom's SiByte family of processors Freescale's 8641D dual-core PowerPC-based SOC.

Víctor Manuel Jáquez Leal () GStreamer & SysLink 2011-10-22 Sat 11 / 1 What is SysLink?

SysLink is the latest generation AMP/IPC driver developed by Texas Instrument. It provides mechanisms to control and communicate with remote processors.

Víctor Manuel Jáquez Leal () GStreamer & SysLink 2011-10-22 Sat 12 / 1 Warning!!!

Objective

This talk will describe the main SysLink components and how we could map those concepts into GStreamer elements. No OpenMAX No extra libraries (syslink-userspace / libmemmgr / etc.)

Víctor Manuel Jáquez Leal () GStreamer & SysLink 2011-10-22 Sat 13 / 1 Objective

This talk will describe the main SysLink components and how we could map those concepts into GStreamer elements. No OpenMAX No extra libraries (syslink-userspace / libmemmgr / etc.) Warning!!!

Víctor Manuel Jáquez Leal () GStreamer & SysLink 2011-10-22 Sat 13 / 1 Project

Gitorious repository page https://gitorious.org/vj-pandaboard/syslink/

These slides are also here https://gitorious.org/vjaquez-misc/syslink-talk/

Víctor Manuel Jáquez Leal () GStreamer & SysLink 2011-10-22 Sat 14 / 1 History of TI's AMP/IPC implementations (1/6)

OMAP2: Reference implementation. OMAP3: DSP/Bridge OMAP2/OMAP3/DaVinci: DSPLink OMAP4: SysLink OMAP4/OMAP5: rpmsg / remoteproc

Víctor Manuel Jáquez Leal () GStreamer & SysLink 2011-10-22 Sat 15 / 1 History of TI's AMP/IPC implementations (2/6)

OMAP2: Reference implementation. dsp-gateway by Nokia. Full open source.

Víctor Manuel Jáquez Leal () GStreamer & SysLink 2011-10-22 Sat 16 / 1 History of TI's AMP/IPC implementations (3/6)

OMAP3: DSP/Bridge Full featured implementation. Only supports DSP (C64x+). Linux side into stagering repository. DSP side is restricted.

Víctor Manuel Jáquez Leal () GStreamer & SysLink 2011-10-22 Sat 17 / 1 History of TI's AMP/IPC implementations (4/6)

OMAP2/OMAP3/DaVinci: DSPLink A stripped implementation of DSP/Bridge. It was not submitted to upstream.

Víctor Manuel Jáquez Leal () GStreamer & SysLink 2011-10-22 Sat 18 / 1 History of TI's AMP/IPC implementations (5/6)

OMAP4: SysLink Supports other types of cores (ARM Cortex-M3 and C64x) Handled in downstream (Ubuntu)

Víctor Manuel Jáquez Leal () GStreamer & SysLink 2011-10-22 Sat 19 / 1 History of TI's AMP/IPC implementations (6/6)

OMAP4/OMAP5: rpmsg / remoteproc Developed completely in upstream.

Víctor Manuel Jáquez Leal () GStreamer & SysLink 2011-10-22 Sat 20 / 1 Dynamic memory management: . . . Dynamic loading: . . . Power Management: . . . Zero-copy shared memory: . . . 2D memory buers allocation: . . . Remote function calls: . . .

Key features of SysLink

Messaging: . . .

Víctor Manuel Jáquez Leal () GStreamer & SysLink 2011-10-22 Sat 21 / 1 Dynamic loading: . . . Power Management: . . . Zero-copy shared memory: . . . 2D memory buers allocation: . . . Remote function calls: . . .

Key features of SysLink

Messaging: . . . Dynamic memory management: . . .

Víctor Manuel Jáquez Leal () GStreamer & SysLink 2011-10-22 Sat 21 / 1 Power Management: . . . Zero-copy shared memory: . . . 2D memory buers allocation: . . . Remote function calls: . . .

Key features of SysLink

Messaging: . . . Dynamic memory management: . . . Dynamic loading: . . .

Víctor Manuel Jáquez Leal () GStreamer & SysLink 2011-10-22 Sat 21 / 1 Zero-copy shared memory: . . . 2D memory buers allocation: . . . Remote function calls: . . .

Key features of SysLink

Messaging: . . . Dynamic memory management: . . . Dynamic loading: . . . Power Management: . . .

Víctor Manuel Jáquez Leal () GStreamer & SysLink 2011-10-22 Sat 21 / 1 2D memory buers allocation: . . . Remote function calls: . . .

Key features of SysLink

Messaging: . . . Dynamic memory management: . . . Dynamic loading: . . . Power Management: . . . Zero-copy shared memory: . . .

Víctor Manuel Jáquez Leal () GStreamer & SysLink 2011-10-22 Sat 21 / 1 Remote function calls: . . .

Key features of SysLink

Messaging: . . . Dynamic memory management: . . . Dynamic loading: . . . Power Management: . . . Zero-copy shared memory: . . . 2D memory buers allocation: . . .

Víctor Manuel Jáquez Leal () GStreamer & SysLink 2011-10-22 Sat 21 / 1 Key features of SysLink

Messaging: . . . Dynamic memory management: . . . Dynamic loading: . . . Power Management: . . . Zero-copy shared memory: . . . 2D memory buers allocation: . . . Remote function calls: . . .

Víctor Manuel Jáquez Leal () GStreamer & SysLink 2011-10-22 Sat 21 / 1 SysLink v2 architecture

Víctor Manuel Jáquez Leal () GStreamer & SysLink 2011-10-22 Sat 22 / 1 SharedRegion: Handles shared memory areas. Gate: Protects memory regions from remote processors. MessageQ: Variable length message passing across processors. Notify: Registers callbacks execute when a remote event occur. HeapBuf: Manages xed size buers within the shared memory. HeapMem: Manages variable size buers within the shared memory.

SysLink v2 components

MultiProc: Identies each available processor.

Víctor Manuel Jáquez Leal () GStreamer & SysLink 2011-10-22 Sat 23 / 1 Gate: Protects memory regions from remote processors. MessageQ: Variable length message passing across processors. Notify: Registers callbacks execute when a remote event occur. HeapBuf: Manages xed size buers within the shared memory. HeapMem: Manages variable size buers within the shared memory.

SysLink v2 components

MultiProc: Identies each available processor. SharedRegion: Handles shared memory areas.

Víctor Manuel Jáquez Leal () GStreamer & SysLink 2011-10-22 Sat 23 / 1 MessageQ: Variable length message passing across processors. Notify: Registers callbacks execute when a remote event occur. HeapBuf: Manages xed size buers within the shared memory. HeapMem: Manages variable size buers within the shared memory.

SysLink v2 components

MultiProc: Identies each available processor. SharedRegion: Handles shared memory areas. Gate: Protects memory regions from remote processors.

Víctor Manuel Jáquez Leal () GStreamer & SysLink 2011-10-22 Sat 23 / 1 Notify: Registers callbacks execute when a remote event occur. HeapBuf: Manages xed size buers within the shared memory. HeapMem: Manages variable size buers within the shared memory.

SysLink v2 components

Víctor Manuel Jáquez Leal () GStreamer & SysLink 2011-10-22 Sat 23 / 1 HeapBuf: Manages xed size buers within the shared memory. HeapMem: Manages variable size buers within the shared memory.

SysLink v2 components

Víctor Manuel Jáquez Leal () GStreamer & SysLink 2011-10-22 Sat 23 / 1 HeapMem: Manages variable size buers within the shared memory.

SysLink v2 components

MultiProc: Identies each available processor. SharedRegion: Handles shared memory areas. Gate: Protects memory regions from remote processors. MessageQ: Variable length message passing across processors. Notify: Registers callbacks execute when a remote event occur. HeapBuf: Manages xed size buers within the shared memory.

Víctor Manuel Jáquez Leal () GStreamer & SysLink 2011-10-22 Sat 23 / 1 SysLink v2 components

Víctor Manuel Jáquez Leal () GStreamer & SysLink 2011-10-22 Sat 23 / 1 Remote Command Messaging RCM

It handles remote function calls. RCM client It manages the allocation, sending, receiving of RCM messages to/from the RCM server. RCM Server It processes inbound messages received from an RCM client. After processing a message, the server will return the message to the client.

Víctor Manuel Jáquez Leal () GStreamer & SysLink 2011-10-22 Sat 24 / 1 RCM Usage

Víctor Manuel Jáquez Leal () GStreamer & SysLink 2011-10-22 Sat 25 / 1 OMAP4430 Pandaboard

Dual-core ARM Cortex-A9 MPCore. Dual-core ARM Cortex-M3. DSP C64x (not much info). 3D graphic acceleration PowerVR SGX540.

Víctor Manuel Jáquez Leal () GStreamer & SysLink 2011-10-22 Sat 26 / 1 Ducati

The two Cortex-M3 processors comprise Ducati. Sys M3: runs the Notify driver. App M3: executes the entire multimedia software.

Víctor Manuel Jáquez Leal () GStreamer & SysLink 2011-10-22 Sat 27 / 1 Distributed Codec Engine DCE

libdce It is library to remotely invoke the hardware accelerated codecs on IVA-HD via SysLink/RCM. DCE AppM3 It is similar in purpose to remote CE for the OMAP4430. CE is a framework that enables applications to instantiate and work with codecs and algorithms using a common API. It is synchronous (sends a buer and waits for the output). It is open source (TI's licence).

Víctor Manuel Jáquez Leal () GStreamer & SysLink 2011-10-22 Sat 28 / 1 Distributed OpenMAX DOMX

It is a RPC shim for an IL client to use a component on Ducati DOMX AppM3 It is asynchronous. It is closed (distributed binary only). Tightly coupled with OpenMAX IL API.

Víctor Manuel Jáquez Leal () GStreamer & SysLink 2011-10-22 Sat 29 / 1 DMM/TILER

It organizes video/image memory in a 2D fashion.

It facilitates Also Allocating. Rotating Freeing. Mirroring Mapping. the 2D blocks or its rectangular of 2D blocks in the container(s). subsections. To be replaced by GEM (The Graphics Execution Manager) https://lwn.net/Articles/283798

Víctor Manuel Jáquez Leal () GStreamer & SysLink 2011-10-22 Sat 30 / 1 Row Stride

The stride is the number of bytes from one row of pixels in memory to the next row of pixels in memory. If padding bytes are present, the stride is wider than the width of the image. DMM/TILER 2D buers require 4KiB row-stride GstStrideTransform (2009 thread) http://gstreamer.freedesktop.org/wiki/NewGstVideo video/x-raw-yuv-strided ??? https://gitorious.org/gstreamer-omap/gst-plugins- base/trees/v0.10.35+ti/gst/stride Use GstMeta in 0.11 ???

Víctor Manuel Jáquez Leal () GStreamer & SysLink 2011-10-22 Sat 31 / 1 2D TILER Buer Pool

GstDucatiBuerPool (1D) https://gitorious.org/gstreamer-omap/gst- ducati/blobs/master/src/gstducatibuerpool.c But for 2D ?? Use GstBuerPool ???

Víctor Manuel Jáquez Leal () GStreamer & SysLink 2011-10-22 Sat 32 / 1 Sync vs Async output buers

DCE is sync No problem here. DOMX is async How to handle it? GstPad tastks and queue in a thread ?

Víctor Manuel Jáquez Leal () GStreamer & SysLink 2011-10-22 Sat 33 / 1 rpmsg / remoteproc

rpmsg A virtio-based messaging bus that allows kernel drivers to communicate with remote processors available on the system. remoteproc A generic driver that maintains the state of the remote processor(s). Firmware loading.

Víctor Manuel Jáquez Leal () GStreamer & SysLink 2011-10-22 Sat 34 / 1 Conclusion

Is it possible to live without OpenMAX? How to handle GEM/TILER buers? strides? buer pool? Asynchronous output buers?

Víctor Manuel Jáquez Leal () GStreamer & SysLink 2011-10-22 Sat 35 / 1 Image credits

ASMP vs SMP http://ohlandl.ipv7.net/CPU/ASMP_SMP.html Who's Afraid of Asymmetric Multiprocessing? http://rtcmagazine.com/articles/view/100404 Ducati for dummies http://www.omappedia.org/wiki/Ducati_For_Dummies SysLink project http://omappedia.org/wiki/Syslink_Project Work in Progress (CC BY-NC 2.0) blumpy https://secure.ickr.com/photos/10537966@N00/325853852/ Pandaboard (C) Tiago Maluta https://secure.ickr.com/photos/22248939@N04/5877274222/

Víctor Manuel Jáquez Leal () GStreamer & SysLink 2011-10-22 Sat 36 / 1 Thanks!

Questions? [email protected] :)

Víctor Manuel Jáquez Leal () GStreamer & SysLink 2011-10-22 Sat 37 / 1