Practical Real-Time Programming in User-Space on Linux linux.conf.au 2008
Lennart Poettering [email protected]
January 2008
Lennart Poettering Practical Real-Time Programming in User-Space on Linux Who Am I?
Software Engineer at Red Hat, Inc. Developer of PulseAudio, Avahi and a few other Free Software projects http://0pointer.de/lennart/ [email protected] IRC: mezcalero
Lennart Poettering Practical Real-Time Programming in User-Space on Linux Whenever an RT process is able to run, it runs.
What is real-time (RT) programming?
Lennart Poettering Practical Real-Time Programming in User-Space on Linux What is real-time (RT) programming? Whenever an RT process is able to run, it runs.
Lennart Poettering Practical Real-Time Programming in User-Space on Linux ABS, Medical Systems, Finance, Audio, . . . Everywhere where small latencies are required Fixed rate playback audio/video; UI feedback
Where is this used?
Lennart Poettering Practical Real-Time Programming in User-Space on Linux Everywhere where small latencies are required Fixed rate playback audio/video; UI feedback
Where is this used? ABS, Medical Systems, Finance, Audio, . . .
Lennart Poettering Practical Real-Time Programming in User-Space on Linux Fixed rate playback audio/video; UI feedback
Where is this used? ABS, Medical Systems, Finance, Audio, . . . Everywhere where small latencies are required
Lennart Poettering Practical Real-Time Programming in User-Space on Linux Where is this used? ABS, Medical Systems, Finance, Audio, . . . Everywhere where small latencies are required Fixed rate playback audio/video; UI feedback
Lennart Poettering Practical Real-Time Programming in User-Space on Linux Our focus: Desktop and Audio on Linux
Lennart Poettering Practical Real-Time Programming in User-Space on Linux Hard vs. Soft Real-time
Lennart Poettering Practical Real-Time Programming in User-Space on Linux Soft RT
Linux as an RT kernel
Lennart Poettering Practical Real-Time Programming in User-Space on Linux Linux as an RT kernel Soft RT
Lennart Poettering Practical Real-Time Programming in User-Space on Linux Scheduling latency: HZ, CPU Load
Lennart Poettering Practical Real-Time Programming in User-Space on Linux Making a process real-time
Lennart Poettering Practical Real-Time Programming in User-Space on Linux Real-time priorities
Lennart Poettering Practical Real-Time Programming in User-Space on Linux Priority inversion Priority inheritance, PTHREAD PRIO INHERIT
Avoid locks!
Lennart Poettering Practical Real-Time Programming in User-Space on Linux Priority inheritance, PTHREAD PRIO INHERIT
Avoid locks! Priority inversion
Lennart Poettering Practical Real-Time Programming in User-Space on Linux Avoid locks! Priority inversion Priority inheritance, PTHREAD PRIO INHERIT
Lennart Poettering Practical Real-Time Programming in User-Space on Linux Wait-free programming Lock-free algorithms are difficult, wait-free even more so
Lock-free programming
Lennart Poettering Practical Real-Time Programming in User-Space on Linux Lock-free algorithms are difficult, wait-free even more so
Lock-free programming Wait-free programming
Lennart Poettering Practical Real-Time Programming in User-Space on Linux Lock-free programming Wait-free programming Lock-free algorithms are difficult, wait-free even more so
Lennart Poettering Practical Real-Time Programming in User-Space on Linux Multi-Reader, Multi-Writer lock-free queues
Lock-free reference counting
Lennart Poettering Practical Real-Time Programming in User-Space on Linux Lock-free reference counting Multi-Reader, Multi-Writer lock-free queues
Lennart Poettering Practical Real-Time Programming in User-Space on Linux APIs and portability: libatomic ops vs. glibc vs. glib vs. sync Emulating atomic ops: In kernel easy, in user-space hard
Atomic operations
Lennart Poettering Practical Real-Time Programming in User-Space on Linux Emulating atomic ops: In kernel easy, in user-space hard
Atomic operations APIs and portability: libatomic ops vs. glibc vs. glib vs. sync
Lennart Poettering Practical Real-Time Programming in User-Space on Linux Atomic operations APIs and portability: libatomic ops vs. glibc vs. glib vs. sync Emulating atomic ops: In kernel easy, in user-space hard
Lennart Poettering Practical Real-Time Programming in User-Space on Linux Memory Barriers
Lennart Poettering Practical Real-Time Programming in User-Space on Linux Beware of thread local pools like GSlice! Alternative: free() lists
Lock-free memory allocation: difficult
Lennart Poettering Practical Real-Time Programming in User-Space on Linux Alternative: free() lists
Lock-free memory allocation: difficult Beware of thread local pools like GSlice!
Lennart Poettering Practical Real-Time Programming in User-Space on Linux Lock-free memory allocation: difficult Beware of thread local pools like GSlice! Alternative: free() lists
Lennart Poettering Practical Real-Time Programming in User-Space on Linux Zero-Copy Minimize copies, cache pressure
Reference-counted memory handling
Lennart Poettering Practical Real-Time Programming in User-Space on Linux Minimize copies, cache pressure
Reference-counted memory handling Zero-Copy
Lennart Poettering Practical Real-Time Programming in User-Space on Linux Reference-counted memory handling Zero-Copy Minimize copies, cache pressure
Lennart Poettering Practical Real-Time Programming in User-Space on Linux To mlockall() or not mlockall()? madvise() instead? Or temporary mlock()?
Memory locking
Lennart Poettering Practical Real-Time Programming in User-Space on Linux madvise() instead? Or temporary mlock()?
Memory locking To mlockall() or not mlockall()?
Lennart Poettering Practical Real-Time Programming in User-Space on Linux Memory locking To mlockall() or not mlockall()? madvise() instead? Or temporary mlock()?
Lennart Poettering Practical Real-Time Programming in User-Space on Linux hrtimers on x86 only, for now nanosleep(), itimers, POSIX Timers are fine But what about poll()? Combine itimers + ppoll() with POSIX real-time signals Signals are evil! Beware of old kernels with ppoll()!
Timers and sleeping: high-resolution timers
Lennart Poettering Practical Real-Time Programming in User-Space on Linux nanosleep(), itimers, POSIX Timers are fine But what about poll()? Combine itimers + ppoll() with POSIX real-time signals Signals are evil! Beware of old kernels with ppoll()!
Timers and sleeping: high-resolution timers hrtimers on x86 only, for now
Lennart Poettering Practical Real-Time Programming in User-Space on Linux But what about poll()? Combine itimers + ppoll() with POSIX real-time signals Signals are evil! Beware of old kernels with ppoll()!
Timers and sleeping: high-resolution timers hrtimers on x86 only, for now nanosleep(), itimers, POSIX Timers are fine
Lennart Poettering Practical Real-Time Programming in User-Space on Linux Combine itimers + ppoll() with POSIX real-time signals Signals are evil! Beware of old kernels with ppoll()!
Timers and sleeping: high-resolution timers hrtimers on x86 only, for now nanosleep(), itimers, POSIX Timers are fine But what about poll()?
Lennart Poettering Practical Real-Time Programming in User-Space on Linux Signals are evil! Beware of old kernels with ppoll()!
Timers and sleeping: high-resolution timers hrtimers on x86 only, for now nanosleep(), itimers, POSIX Timers are fine But what about poll()? Combine itimers + ppoll() with POSIX real-time signals
Lennart Poettering Practical Real-Time Programming in User-Space on Linux Beware of old kernels with ppoll()!
Timers and sleeping: high-resolution timers hrtimers on x86 only, for now nanosleep(), itimers, POSIX Timers are fine But what about poll()? Combine itimers + ppoll() with POSIX real-time signals Signals are evil!
Lennart Poettering Practical Real-Time Programming in User-Space on Linux Timers and sleeping: high-resolution timers hrtimers on x86 only, for now nanosleep(), itimers, POSIX Timers are fine But what about poll()? Combine itimers + ppoll() with POSIX real-time signals Signals are evil! Beware of old kernels with ppoll()!
Lennart Poettering Practical Real-Time Programming in User-Space on Linux Futexes Semaphores good, Conditions bad, Mutexes with PI good
Mutexes, Semaphores, Conditions
Lennart Poettering Practical Real-Time Programming in User-Space on Linux Semaphores good, Conditions bad, Mutexes with PI good
Mutexes, Semaphores, Conditions Futexes
Lennart Poettering Practical Real-Time Programming in User-Space on Linux , Conditions bad, Mutexes with PI good
Mutexes, Semaphores, Conditions Futexes Semaphores good
Lennart Poettering Practical Real-Time Programming in User-Space on Linux , Mutexes with PI good
Mutexes, Semaphores, Conditions Futexes Semaphores good, Conditions bad
Lennart Poettering Practical Real-Time Programming in User-Space on Linux Mutexes, Semaphores, Conditions Futexes Semaphores good, Conditions bad, Mutexes with PI good
Lennart Poettering Practical Real-Time Programming in User-Space on Linux FIFOs and eventfd() ... use locking Compromise: Wrap eventfd()/FIFOs in atomic ops
Using poll() and futexes?
Lennart Poettering Practical Real-Time Programming in User-Space on Linux ... use locking Compromise: Wrap eventfd()/FIFOs in atomic ops
Using poll() and futexes? FIFOs and eventfd()
Lennart Poettering Practical Real-Time Programming in User-Space on Linux Compromise: Wrap eventfd()/FIFOs in atomic ops
Using poll() and futexes? FIFOs and eventfd() ... use locking
Lennart Poettering Practical Real-Time Programming in User-Space on Linux Using poll() and futexes? FIFOs and eventfd() ... use locking Compromise: Wrap eventfd()/FIFOs in atomic ops
Lennart Poettering Practical Real-Time Programming in User-Space on Linux (theoretically) Allows sleeping on timers, futexes, fds and is lock-free
Better solution: kevent
Lennart Poettering Practical Real-Time Programming in User-Space on Linux Allows sleeping on timers, futexes, fds and is lock-free
Better solution: kevent (theoretically)
Lennart Poettering Practical Real-Time Programming in User-Space on Linux and is lock-free
Better solution: kevent (theoretically) Allows sleeping on timers, futexes, fds
Lennart Poettering Practical Real-Time Programming in User-Space on Linux Better solution: kevent (theoretically) Allows sleeping on timers, futexes, fds and is lock-free
Lennart Poettering Practical Real-Time Programming in User-Space on Linux RTLD NOW and $BIND NOW
Lazy binding and relocations
Lennart Poettering Practical Real-Time Programming in User-Space on Linux Lazy binding and relocations RTLD NOW and $BIND NOW
Lennart Poettering Practical Real-Time Programming in User-Space on Linux Watchdog thread vs. RLIMIT CPU and SIGXCPU RLIMIT RTPRIO RLIMIT RTTIME
Robust RT and security
Lennart Poettering Practical Real-Time Programming in User-Space on Linux RLIMIT RTPRIO RLIMIT RTTIME
Robust RT and security Watchdog thread vs. RLIMIT CPU and SIGXCPU
Lennart Poettering Practical Real-Time Programming in User-Space on Linux RLIMIT RTTIME
Robust RT and security Watchdog thread vs. RLIMIT CPU and SIGXCPU RLIMIT RTPRIO
Lennart Poettering Practical Real-Time Programming in User-Space on Linux Robust RT and security Watchdog thread vs. RLIMIT CPU and SIGXCPU RLIMIT RTPRIO RLIMIT RTTIME
Lennart Poettering Practical Real-Time Programming in User-Space on Linux It’s difficult!
How to partition your code best between RT threads and non RT threads?
Lennart Poettering Practical Real-Time Programming in User-Space on Linux How to partition your code best between RT threads and non RT threads? It’s difficult!
Lennart Poettering Practical Real-Time Programming in User-Space on Linux Any other kind of blocking I/O Unbounded algorithms, i.e. slower than O(n) Code that needs locking – unless there is only one piece of code that locks it
Avoid in RT:
Disk I/O
Lennart Poettering Practical Real-Time Programming in User-Space on Linux Unbounded algorithms, i.e. slower than O(n) Code that needs locking – unless there is only one piece of code that locks it
Avoid in RT:
Disk I/O Any other kind of blocking I/O
Lennart Poettering Practical Real-Time Programming in User-Space on Linux Code that needs locking – unless there is only one piece of code that locks it
Avoid in RT:
Disk I/O Any other kind of blocking I/O Unbounded algorithms, i.e. slower than O(n)
Lennart Poettering Practical Real-Time Programming in User-Space on Linux Avoid in RT:
Disk I/O Any other kind of blocking I/O Unbounded algorithms, i.e. slower than O(n) Code that needs locking – unless there is only one piece of code that locks it
Lennart Poettering Practical Real-Time Programming in User-Space on Linux Context switches; Latency due to buffering; Code becomes a lot more difficult to understand.
Splitting code up into non-RT and RT threads comes at a cost:
Lennart Poettering Practical Real-Time Programming in User-Space on Linux ; Latency due to buffering; Code becomes a lot more difficult to understand.
Splitting code up into non-RT and RT threads comes at a cost: Context switches
Lennart Poettering Practical Real-Time Programming in User-Space on Linux ; Code becomes a lot more difficult to understand.
Splitting code up into non-RT and RT threads comes at a cost: Context switches; Latency due to buffering
Lennart Poettering Practical Real-Time Programming in User-Space on Linux Splitting code up into non-RT and RT threads comes at a cost: Context switches; Latency due to buffering; Code becomes a lot more difficult to understand.
Lennart Poettering Practical Real-Time Programming in User-Space on Linux But this is controversial.
Thus: in some cases it even makes sense to do non-trivial calculations in the RT thread. As long as it is bounded by 1/HZ in completion time. Additional benefit: your code can decide when it is best to execute the non-trivial calculations.
Lennart Poettering Practical Real-Time Programming in User-Space on Linux Thus: in some cases it even makes sense to do non-trivial calculations in the RT thread. As long as it is bounded by 1/HZ in completion time. Additional benefit: your code can decide when it is best to execute the non-trivial calculations. But this is controversial.
Lennart Poettering Practical Real-Time Programming in User-Space on Linux Make sure to run a watchdog of some kind. Possibly a shell with a high real-time priority
Debugging RT
Lennart Poettering Practical Real-Time Programming in User-Space on Linux Possibly a shell with a high real-time priority
Debugging RT Make sure to run a watchdog of some kind.
Lennart Poettering Practical Real-Time Programming in User-Space on Linux Debugging RT Make sure to run a watchdog of some kind. Possibly a shell with a high real-time priority
Lennart Poettering Practical Real-Time Programming in User-Space on Linux Logging in RT programs Tracing memory allocations Tracing mutexes Idea: A module for valgrind that looks for stuff that should not be done in RT threads
Profile your code!
Lennart Poettering Practical Real-Time Programming in User-Space on Linux Tracing memory allocations Tracing mutexes Idea: A module for valgrind that looks for stuff that should not be done in RT threads
Profile your code! Logging in RT programs
Lennart Poettering Practical Real-Time Programming in User-Space on Linux Tracing mutexes Idea: A module for valgrind that looks for stuff that should not be done in RT threads
Profile your code! Logging in RT programs Tracing memory allocations
Lennart Poettering Practical Real-Time Programming in User-Space on Linux Idea: A module for valgrind that looks for stuff that should not be done in RT threads
Profile your code! Logging in RT programs Tracing memory allocations Tracing mutexes
Lennart Poettering Practical Real-Time Programming in User-Space on Linux Profile your code! Logging in RT programs Tracing memory allocations Tracing mutexes Idea: A module for valgrind that looks for stuff that should not be done in RT threads
Lennart Poettering Practical Real-Time Programming in User-Space on Linux Any questions?
That’s all, folks.
Lennart Poettering Practical Real-Time Programming in User-Space on Linux That’s all, folks. Any questions?
Lennart Poettering Practical Real-Time Programming in User-Space on Linux