Practical Real-Time Programming in User-Space on Linux linux.conf.au 2008 Lennart Poettering [email protected] January 2008 Lennart Poettering Practical Real-Time Programming in User-Space on Linux Who Am I? Software Engineer at Red Hat, Inc. Developer of PulseAudio, Avahi and a few other Free Software projects http://0pointer.de/lennart/ [email protected] IRC: mezcalero Lennart Poettering Practical Real-Time Programming in User-Space on Linux Whenever an RT process is able to run, it runs. What is real-time (RT) programming? Lennart Poettering Practical Real-Time Programming in User-Space on Linux What is real-time (RT) programming? Whenever an RT process is able to run, it runs. Lennart Poettering Practical Real-Time Programming in User-Space on Linux ABS, Medical Systems, Finance, Audio, . Everywhere where small latencies are required Fixed rate playback audio/video; UI feedback Where is this used? Lennart Poettering Practical Real-Time Programming in User-Space on Linux Everywhere where small latencies are required Fixed rate playback audio/video; UI feedback Where is this used? ABS, Medical Systems, Finance, Audio, . Lennart Poettering Practical Real-Time Programming in User-Space on Linux Fixed rate playback audio/video; UI feedback Where is this used? ABS, Medical Systems, Finance, Audio, . Everywhere where small latencies are required Lennart Poettering Practical Real-Time Programming in User-Space on Linux Where is this used? ABS, Medical Systems, Finance, Audio, . Everywhere where small latencies are required Fixed rate playback audio/video; UI feedback Lennart Poettering Practical Real-Time Programming in User-Space on Linux Our focus: Desktop and Audio on Linux Lennart Poettering Practical Real-Time Programming in User-Space on Linux Hard vs. Soft Real-time Lennart Poettering Practical Real-Time Programming in User-Space on Linux Soft RT Linux as an RT kernel Lennart Poettering Practical Real-Time Programming in User-Space on Linux Linux as an RT kernel Soft RT Lennart Poettering Practical Real-Time Programming in User-Space on Linux Scheduling latency: HZ, CPU Load Lennart Poettering Practical Real-Time Programming in User-Space on Linux Making a process real-time Lennart Poettering Practical Real-Time Programming in User-Space on Linux Real-time priorities Lennart Poettering Practical Real-Time Programming in User-Space on Linux Priority inversion Priority inheritance, PTHREAD PRIO INHERIT Avoid locks! Lennart Poettering Practical Real-Time Programming in User-Space on Linux Priority inheritance, PTHREAD PRIO INHERIT Avoid locks! Priority inversion Lennart Poettering Practical Real-Time Programming in User-Space on Linux Avoid locks! Priority inversion Priority inheritance, PTHREAD PRIO INHERIT Lennart Poettering Practical Real-Time Programming in User-Space on Linux Wait-free programming Lock-free algorithms are difficult, wait-free even more so Lock-free programming Lennart Poettering Practical Real-Time Programming in User-Space on Linux Lock-free algorithms are difficult, wait-free even more so Lock-free programming Wait-free programming Lennart Poettering Practical Real-Time Programming in User-Space on Linux Lock-free programming Wait-free programming Lock-free algorithms are difficult, wait-free even more so Lennart Poettering Practical Real-Time Programming in User-Space on Linux Multi-Reader, Multi-Writer lock-free queues Lock-free reference counting Lennart Poettering Practical Real-Time Programming in User-Space on Linux Lock-free reference counting Multi-Reader, Multi-Writer lock-free queues Lennart Poettering Practical Real-Time Programming in User-Space on Linux APIs and portability: libatomic ops vs. glibc vs. glib vs. sync Emulating atomic ops: In kernel easy, in user-space hard Atomic operations Lennart Poettering Practical Real-Time Programming in User-Space on Linux Emulating atomic ops: In kernel easy, in user-space hard Atomic operations APIs and portability: libatomic ops vs. glibc vs. glib vs. sync Lennart Poettering Practical Real-Time Programming in User-Space on Linux Atomic operations APIs and portability: libatomic ops vs. glibc vs. glib vs. sync Emulating atomic ops: In kernel easy, in user-space hard Lennart Poettering Practical Real-Time Programming in User-Space on Linux Memory Barriers Lennart Poettering Practical Real-Time Programming in User-Space on Linux Beware of thread local pools like GSlice! Alternative: free() lists Lock-free memory allocation: difficult Lennart Poettering Practical Real-Time Programming in User-Space on Linux Alternative: free() lists Lock-free memory allocation: difficult Beware of thread local pools like GSlice! Lennart Poettering Practical Real-Time Programming in User-Space on Linux Lock-free memory allocation: difficult Beware of thread local pools like GSlice! Alternative: free() lists Lennart Poettering Practical Real-Time Programming in User-Space on Linux Zero-Copy Minimize copies, cache pressure Reference-counted memory handling Lennart Poettering Practical Real-Time Programming in User-Space on Linux Minimize copies, cache pressure Reference-counted memory handling Zero-Copy Lennart Poettering Practical Real-Time Programming in User-Space on Linux Reference-counted memory handling Zero-Copy Minimize copies, cache pressure Lennart Poettering Practical Real-Time Programming in User-Space on Linux To mlockall() or not mlockall()? madvise() instead? Or temporary mlock()? Memory locking Lennart Poettering Practical Real-Time Programming in User-Space on Linux madvise() instead? Or temporary mlock()? Memory locking To mlockall() or not mlockall()? Lennart Poettering Practical Real-Time Programming in User-Space on Linux Memory locking To mlockall() or not mlockall()? madvise() instead? Or temporary mlock()? Lennart Poettering Practical Real-Time Programming in User-Space on Linux hrtimers on x86 only, for now nanosleep(), itimers, POSIX Timers are fine But what about poll()? Combine itimers + ppoll() with POSIX real-time signals Signals are evil! Beware of old kernels with ppoll()! Timers and sleeping: high-resolution timers Lennart Poettering Practical Real-Time Programming in User-Space on Linux nanosleep(), itimers, POSIX Timers are fine But what about poll()? Combine itimers + ppoll() with POSIX real-time signals Signals are evil! Beware of old kernels with ppoll()! Timers and sleeping: high-resolution timers hrtimers on x86 only, for now Lennart Poettering Practical Real-Time Programming in User-Space on Linux But what about poll()? Combine itimers + ppoll() with POSIX real-time signals Signals are evil! Beware of old kernels with ppoll()! Timers and sleeping: high-resolution timers hrtimers on x86 only, for now nanosleep(), itimers, POSIX Timers are fine Lennart Poettering Practical Real-Time Programming in User-Space on Linux Combine itimers + ppoll() with POSIX real-time signals Signals are evil! Beware of old kernels with ppoll()! Timers and sleeping: high-resolution timers hrtimers on x86 only, for now nanosleep(), itimers, POSIX Timers are fine But what about poll()? Lennart Poettering Practical Real-Time Programming in User-Space on Linux Signals are evil! Beware of old kernels with ppoll()! Timers and sleeping: high-resolution timers hrtimers on x86 only, for now nanosleep(), itimers, POSIX Timers are fine But what about poll()? Combine itimers + ppoll() with POSIX real-time signals Lennart Poettering Practical Real-Time Programming in User-Space on Linux Beware of old kernels with ppoll()! Timers and sleeping: high-resolution timers hrtimers on x86 only, for now nanosleep(), itimers, POSIX Timers are fine But what about poll()? Combine itimers + ppoll() with POSIX real-time signals Signals are evil! Lennart Poettering Practical Real-Time Programming in User-Space on Linux Timers and sleeping: high-resolution timers hrtimers on x86 only, for now nanosleep(), itimers, POSIX Timers are fine But what about poll()? Combine itimers + ppoll() with POSIX real-time signals Signals are evil! Beware of old kernels with ppoll()! Lennart Poettering Practical Real-Time Programming in User-Space on Linux Futexes Semaphores good, Conditions bad, Mutexes with PI good Mutexes, Semaphores, Conditions Lennart Poettering Practical Real-Time Programming in User-Space on Linux Semaphores good, Conditions bad, Mutexes with PI good Mutexes, Semaphores, Conditions Futexes Lennart Poettering Practical Real-Time Programming in User-Space on Linux , Conditions bad, Mutexes with PI good Mutexes, Semaphores, Conditions Futexes Semaphores good Lennart Poettering Practical Real-Time Programming in User-Space on Linux , Mutexes with PI good Mutexes, Semaphores, Conditions Futexes Semaphores good, Conditions bad Lennart Poettering Practical Real-Time Programming in User-Space on Linux Mutexes, Semaphores, Conditions Futexes Semaphores good, Conditions bad, Mutexes with PI good Lennart Poettering Practical Real-Time Programming in User-Space on Linux FIFOs and eventfd() ... use locking Compromise: Wrap eventfd()/FIFOs in atomic ops Using poll() and futexes? Lennart Poettering Practical Real-Time Programming in User-Space on Linux ... use locking Compromise: Wrap eventfd()/FIFOs in atomic ops Using poll() and futexes? FIFOs and eventfd() Lennart Poettering Practical Real-Time Programming in User-Space on Linux Compromise: Wrap eventfd()/FIFOs in atomic ops Using poll() and futexes? FIFOs and eventfd() ... use locking Lennart Poettering Practical Real-Time Programming in User-Space on Linux Using poll() and futexes? FIFOs and eventfd() ... use locking Compromise: Wrap eventfd()/FIFOs