Practical Real-Time Programming in User-Space on linux.conf.au 2008

Lennart Poettering [email protected]

January 2008

Lennart Poettering Practical Real-Time Programming in User-Space on Linux Who Am I?

Software Engineer at , Inc. Developer of PulseAudio, and a few other projects http://0pointer.de/lennart/ [email protected] IRC: mezcalero

Lennart Poettering Practical Real-Time Programming in User-Space on Linux Whenever an RT process is able to run, it runs.

What is real-time (RT) programming?

Lennart Poettering Practical Real-Time Programming in User-Space on Linux What is real-time (RT) programming? Whenever an RT process is able to run, it runs.

Lennart Poettering Practical Real-Time Programming in User-Space on Linux ABS, Medical Systems, Finance, Audio, . . . Everywhere where small latencies are required Fixed rate playback audio/video; UI feedback

Where is this used?

Lennart Poettering Practical Real-Time Programming in User-Space on Linux Everywhere where small latencies are required Fixed rate playback audio/video; UI feedback

Where is this used? ABS, Medical Systems, Finance, Audio, . . .

Lennart Poettering Practical Real-Time Programming in User-Space on Linux Fixed rate playback audio/video; UI feedback

Where is this used? ABS, Medical Systems, Finance, Audio, . . . Everywhere where small latencies are required

Lennart Poettering Practical Real-Time Programming in User-Space on Linux Where is this used? ABS, Medical Systems, Finance, Audio, . . . Everywhere where small latencies are required Fixed rate playback audio/video; UI feedback

Lennart Poettering Practical Real-Time Programming in User-Space on Linux Our focus: Desktop and Audio on Linux

Lennart Poettering Practical Real-Time Programming in User-Space on Linux Hard vs. Soft Real-time

Lennart Poettering Practical Real-Time Programming in User-Space on Linux Soft RT

Linux as an RT kernel

Lennart Poettering Practical Real-Time Programming in User-Space on Linux Linux as an RT kernel Soft RT

Lennart Poettering Practical Real-Time Programming in User-Space on Linux Scheduling latency: HZ, CPU Load

Lennart Poettering Practical Real-Time Programming in User-Space on Linux Making a process real-time

Lennart Poettering Practical Real-Time Programming in User-Space on Linux Real-time priorities

Lennart Poettering Practical Real-Time Programming in User-Space on Linux Priority inversion Priority inheritance, PTHREAD PRIO INHERIT

Avoid locks!

Lennart Poettering Practical Real-Time Programming in User-Space on Linux Priority inheritance, PTHREAD PRIO INHERIT

Avoid locks! Priority inversion

Lennart Poettering Practical Real-Time Programming in User-Space on Linux Avoid locks! Priority inversion Priority inheritance, PTHREAD PRIO INHERIT

Lennart Poettering Practical Real-Time Programming in User-Space on Linux Wait-free programming Lock-free algorithms are difficult, wait-free even more so

Lock-free programming

Lennart Poettering Practical Real-Time Programming in User-Space on Linux Lock-free algorithms are difficult, wait-free even more so

Lock-free programming Wait-free programming

Lennart Poettering Practical Real-Time Programming in User-Space on Linux Lock-free programming Wait-free programming Lock-free algorithms are difficult, wait-free even more so

Lennart Poettering Practical Real-Time Programming in User-Space on Linux Multi-Reader, Multi-Writer lock-free queues

Lock-free reference counting

Lennart Poettering Practical Real-Time Programming in User-Space on Linux Lock-free reference counting Multi-Reader, Multi-Writer lock-free queues

Lennart Poettering Practical Real-Time Programming in User-Space on Linux APIs and portability: libatomic ops vs. glibc vs. glib vs. sync Emulating atomic ops: In kernel easy, in user-space hard

Atomic operations

Lennart Poettering Practical Real-Time Programming in User-Space on Linux Emulating atomic ops: In kernel easy, in user-space hard

Atomic operations APIs and portability: libatomic ops vs. glibc vs. glib vs. sync

Lennart Poettering Practical Real-Time Programming in User-Space on Linux Atomic operations APIs and portability: libatomic ops vs. glibc vs. glib vs. sync Emulating atomic ops: In kernel easy, in user-space hard

Lennart Poettering Practical Real-Time Programming in User-Space on Linux Memory Barriers

Lennart Poettering Practical Real-Time Programming in User-Space on Linux Beware of thread local pools like GSlice! Alternative: free() lists

Lock-free memory allocation: difficult

Lennart Poettering Practical Real-Time Programming in User-Space on Linux Alternative: free() lists

Lock-free memory allocation: difficult Beware of thread local pools like GSlice!

Lennart Poettering Practical Real-Time Programming in User-Space on Linux Lock-free memory allocation: difficult Beware of thread local pools like GSlice! Alternative: free() lists

Lennart Poettering Practical Real-Time Programming in User-Space on Linux Zero-Copy Minimize copies, cache pressure

Reference-counted memory handling

Lennart Poettering Practical Real-Time Programming in User-Space on Linux Minimize copies, cache pressure

Reference-counted memory handling Zero-Copy

Lennart Poettering Practical Real-Time Programming in User-Space on Linux Reference-counted memory handling Zero-Copy Minimize copies, cache pressure

Lennart Poettering Practical Real-Time Programming in User-Space on Linux To mlockall() or not mlockall()? madvise() instead? Or temporary mlock()?

Memory locking

Lennart Poettering Practical Real-Time Programming in User-Space on Linux madvise() instead? Or temporary mlock()?

Memory locking To mlockall() or not mlockall()?

Lennart Poettering Practical Real-Time Programming in User-Space on Linux Memory locking To mlockall() or not mlockall()? madvise() instead? Or temporary mlock()?

Lennart Poettering Practical Real-Time Programming in User-Space on Linux hrtimers on x86 only, for now nanosleep(), itimers, POSIX Timers are fine But what about poll()? Combine itimers + ppoll() with POSIX real-time signals Signals are evil! Beware of old kernels with ppoll()!

Timers and sleeping: high-resolution timers

Lennart Poettering Practical Real-Time Programming in User-Space on Linux nanosleep(), itimers, POSIX Timers are fine But what about poll()? Combine itimers + ppoll() with POSIX real-time signals Signals are evil! Beware of old kernels with ppoll()!

Timers and sleeping: high-resolution timers hrtimers on x86 only, for now

Lennart Poettering Practical Real-Time Programming in User-Space on Linux But what about poll()? Combine itimers + ppoll() with POSIX real-time signals Signals are evil! Beware of old kernels with ppoll()!

Timers and sleeping: high-resolution timers hrtimers on x86 only, for now nanosleep(), itimers, POSIX Timers are fine

Lennart Poettering Practical Real-Time Programming in User-Space on Linux Combine itimers + ppoll() with POSIX real-time signals Signals are evil! Beware of old kernels with ppoll()!

Timers and sleeping: high-resolution timers hrtimers on x86 only, for now nanosleep(), itimers, POSIX Timers are fine But what about poll()?

Lennart Poettering Practical Real-Time Programming in User-Space on Linux Signals are evil! Beware of old kernels with ppoll()!

Timers and sleeping: high-resolution timers hrtimers on x86 only, for now nanosleep(), itimers, POSIX Timers are fine But what about poll()? Combine itimers + ppoll() with POSIX real-time signals

Lennart Poettering Practical Real-Time Programming in User-Space on Linux Beware of old kernels with ppoll()!

Timers and sleeping: high-resolution timers hrtimers on x86 only, for now nanosleep(), itimers, POSIX Timers are fine But what about poll()? Combine itimers + ppoll() with POSIX real-time signals Signals are evil!

Lennart Poettering Practical Real-Time Programming in User-Space on Linux Timers and sleeping: high-resolution timers hrtimers on x86 only, for now nanosleep(), itimers, POSIX Timers are fine But what about poll()? Combine itimers + ppoll() with POSIX real-time signals Signals are evil! Beware of old kernels with ppoll()!

Lennart Poettering Practical Real-Time Programming in User-Space on Linux Futexes Semaphores good, Conditions bad, Mutexes with PI good

Mutexes, Semaphores, Conditions

Lennart Poettering Practical Real-Time Programming in User-Space on Linux Semaphores good, Conditions bad, Mutexes with PI good

Mutexes, Semaphores, Conditions Futexes

Lennart Poettering Practical Real-Time Programming in User-Space on Linux , Conditions bad, Mutexes with PI good

Mutexes, Semaphores, Conditions Futexes Semaphores good

Lennart Poettering Practical Real-Time Programming in User-Space on Linux , Mutexes with PI good

Mutexes, Semaphores, Conditions Futexes Semaphores good, Conditions bad

Lennart Poettering Practical Real-Time Programming in User-Space on Linux Mutexes, Semaphores, Conditions Futexes Semaphores good, Conditions bad, Mutexes with PI good

Lennart Poettering Practical Real-Time Programming in User-Space on Linux FIFOs and eventfd() ... use locking Compromise: Wrap eventfd()/FIFOs in atomic ops

Using poll() and futexes?

Lennart Poettering Practical Real-Time Programming in User-Space on Linux ... use locking Compromise: Wrap eventfd()/FIFOs in atomic ops

Using poll() and futexes? FIFOs and eventfd()

Lennart Poettering Practical Real-Time Programming in User-Space on Linux Compromise: Wrap eventfd()/FIFOs in atomic ops

Using poll() and futexes? FIFOs and eventfd() ... use locking

Lennart Poettering Practical Real-Time Programming in User-Space on Linux Using poll() and futexes? FIFOs and eventfd() ... use locking Compromise: Wrap eventfd()/FIFOs in atomic ops

Lennart Poettering Practical Real-Time Programming in User-Space on Linux (theoretically) Allows sleeping on timers, futexes, fds and is lock-free

Better solution: kevent

Lennart Poettering Practical Real-Time Programming in User-Space on Linux Allows sleeping on timers, futexes, fds and is lock-free

Better solution: kevent (theoretically)

Lennart Poettering Practical Real-Time Programming in User-Space on Linux and is lock-free

Better solution: kevent (theoretically) Allows sleeping on timers, futexes, fds

Lennart Poettering Practical Real-Time Programming in User-Space on Linux Better solution: kevent (theoretically) Allows sleeping on timers, futexes, fds and is lock-free

Lennart Poettering Practical Real-Time Programming in User-Space on Linux RTLD NOW and $BIND NOW

Lazy binding and relocations

Lennart Poettering Practical Real-Time Programming in User-Space on Linux Lazy binding and relocations RTLD NOW and $BIND NOW

Lennart Poettering Practical Real-Time Programming in User-Space on Linux Watchdog thread vs. RLIMIT CPU and SIGXCPU RLIMIT RTPRIO RLIMIT RTTIME

Robust RT and security

Lennart Poettering Practical Real-Time Programming in User-Space on Linux RLIMIT RTPRIO RLIMIT RTTIME

Robust RT and security Watchdog thread vs. RLIMIT CPU and SIGXCPU

Lennart Poettering Practical Real-Time Programming in User-Space on Linux RLIMIT RTTIME

Robust RT and security Watchdog thread vs. RLIMIT CPU and SIGXCPU RLIMIT RTPRIO

Lennart Poettering Practical Real-Time Programming in User-Space on Linux Robust RT and security Watchdog thread vs. RLIMIT CPU and SIGXCPU RLIMIT RTPRIO RLIMIT RTTIME

Lennart Poettering Practical Real-Time Programming in User-Space on Linux It’s difficult!

How to partition your code best between RT threads and non RT threads?

Lennart Poettering Practical Real-Time Programming in User-Space on Linux How to partition your code best between RT threads and non RT threads? It’s difficult!

Lennart Poettering Practical Real-Time Programming in User-Space on Linux Any other kind of blocking I/O Unbounded algorithms, i.e. slower than O(n) Code that needs locking – unless there is only one piece of code that locks it

Avoid in RT:

Disk I/O

Lennart Poettering Practical Real-Time Programming in User-Space on Linux Unbounded algorithms, i.e. slower than O(n) Code that needs locking – unless there is only one piece of code that locks it

Avoid in RT:

Disk I/O Any other kind of blocking I/O

Lennart Poettering Practical Real-Time Programming in User-Space on Linux Code that needs locking – unless there is only one piece of code that locks it

Avoid in RT:

Disk I/O Any other kind of blocking I/O Unbounded algorithms, i.e. slower than O(n)

Lennart Poettering Practical Real-Time Programming in User-Space on Linux Avoid in RT:

Disk I/O Any other kind of blocking I/O Unbounded algorithms, i.e. slower than O(n) Code that needs locking – unless there is only one piece of code that locks it

Lennart Poettering Practical Real-Time Programming in User-Space on Linux Context switches; Latency due to buffering; Code becomes a lot more difficult to understand.

Splitting code up into non-RT and RT threads comes at a cost:

Lennart Poettering Practical Real-Time Programming in User-Space on Linux ; Latency due to buffering; Code becomes a lot more difficult to understand.

Splitting code up into non-RT and RT threads comes at a cost: Context switches

Lennart Poettering Practical Real-Time Programming in User-Space on Linux ; Code becomes a lot more difficult to understand.

Splitting code up into non-RT and RT threads comes at a cost: Context switches; Latency due to buffering

Lennart Poettering Practical Real-Time Programming in User-Space on Linux Splitting code up into non-RT and RT threads comes at a cost: Context switches; Latency due to buffering; Code becomes a lot more difficult to understand.

Lennart Poettering Practical Real-Time Programming in User-Space on Linux But this is controversial.

Thus: in some cases it even makes sense to do non-trivial calculations in the RT thread. As long as it is bounded by 1/HZ in completion time. Additional benefit: your code can decide when it is best to execute the non-trivial calculations.

Lennart Poettering Practical Real-Time Programming in User-Space on Linux Thus: in some cases it even makes sense to do non-trivial calculations in the RT thread. As long as it is bounded by 1/HZ in completion time. Additional benefit: your code can decide when it is best to execute the non-trivial calculations. But this is controversial.

Lennart Poettering Practical Real-Time Programming in User-Space on Linux Make sure to run a watchdog of some kind. Possibly a shell with a high real-time priority

Debugging RT

Lennart Poettering Practical Real-Time Programming in User-Space on Linux Possibly a shell with a high real-time priority

Debugging RT Make sure to run a watchdog of some kind.

Lennart Poettering Practical Real-Time Programming in User-Space on Linux Debugging RT Make sure to run a watchdog of some kind. Possibly a shell with a high real-time priority

Lennart Poettering Practical Real-Time Programming in User-Space on Linux Logging in RT programs Tracing memory allocations Tracing mutexes Idea: A module for valgrind that looks for stuff that should not be done in RT threads

Profile your code!

Lennart Poettering Practical Real-Time Programming in User-Space on Linux Tracing memory allocations Tracing mutexes Idea: A module for valgrind that looks for stuff that should not be done in RT threads

Profile your code! Logging in RT programs

Lennart Poettering Practical Real-Time Programming in User-Space on Linux Tracing mutexes Idea: A module for valgrind that looks for stuff that should not be done in RT threads

Profile your code! Logging in RT programs Tracing memory allocations

Lennart Poettering Practical Real-Time Programming in User-Space on Linux Idea: A module for valgrind that looks for stuff that should not be done in RT threads

Profile your code! Logging in RT programs Tracing memory allocations Tracing mutexes

Lennart Poettering Practical Real-Time Programming in User-Space on Linux Profile your code! Logging in RT programs Tracing memory allocations Tracing mutexes Idea: A module for valgrind that looks for stuff that should not be done in RT threads

Lennart Poettering Practical Real-Time Programming in User-Space on Linux Any questions?

That’s all, folks.

Lennart Poettering Practical Real-Time Programming in User-Space on Linux That’s all, folks. Any questions?

Lennart Poettering Practical Real-Time Programming in User-Space on Linux