Juha Erkkilä Real-Time Audio Servers on BSD Unix Derivatives Master's Thesis in Information Technology June 17, 2005 University of Jyväskylä Department of Mathematical Information Technology Jyväskylä Author: Juha Erkkilä Contact information: [email protected].fi Title: Real-Time Audio Servers on BSD Unix Derivatives Työn nimi: Reaaliaikaiset äänipalvelinsovellukset BSD Unix -johdannaisjärjestelmissä Project: Master's Thesis in Information Technology Page count: 146 Abstract: This paper covers real-time and interprocess communication features of 4.4BSD Unix derived operating systems, and especially their applicability for real- time audio servers. The research ground of bringing real-time properties to tradi- tional Unix operating systems (such as 4.4BSD) is covered. Included are some design ideas used in BSD-variants, such as using multithreaded kernels, and schedulers that can provide real-time guarantees to processes. Factors affecting the design of real- time audio servers are considered, especially the suitability of various interprocess communication facilities as mechanisms to pass audio data between applications. To test these mechanisms on a real operating system, an audio server and a client utilizing these techniques is written and tested on an OpenBSD operating system. The performance of the audio server and OpenBSD is analyzed, with attempts to identify some bottlenecks of real-time operation in the OpenBSD system. Suomenkielinen tiivistelmä: Tämä tutkielma kattaa reaaliaikaisuus- ja prosessien väliset kommunikaatio-ominaisuudet, keskittyen 4.4BSD Unix -johdannaisiin käyt- töjärjestelmiin, ja erityisesti siihen kuinka hyvin nämä soveltuvat reaaliaikaisille äänipalvelinsovelluksille. Tutkimusalueeseen sisältyy reaaliaikaisuusominaisuuk- sien tuominen perinteisiin Unix-käyttöjärjestelmiin (kuten 4.4BSD:hen). Mukana on suunnitteluideoita, joita on käytetty joissakin BSD-varianteissa, kuten säikeis- tetyt kernelit, ja skedulerit, jotka voivat tarjota reaaliaikaisuustakeita prosesseille. Huomioon otettavina seikkoina ovat myös reaaliaikaisten äänipalvelimien suun- nitteluun vaikuttavat tekijät, erityisesti erilaisten prosessien välisten kommunikaa- tiomekanismien soveltuvuus sovellusten väliseen äänidatan välitykseen. Näiden mekanismien testaamista varten tutkielmaa varten on kirjoitettu yksi palvelinsovel- lus sekä asiakassovellus, joita testataan OpenBSD-käyttöjärjestelmän päällä. Ana- lyysin kohteena ovat testipalvelinsovelluksen ja OpenBSD:n suorituskyky, tarkoituk- sena on erityisesti löytää reaaliaikaisen suorituskyvyn pullonkauloja OpenBSD- käyttöjärjestelmästä. Keywords: real-time distributed BSD unix kernel audio scheduling Avainsanat: reaaliaikainen hajautettu BSD unix kernel ääni skedulointi No copyright, this work is in public domain. Preface You might already know this, but in case you're reading some treeware edition, you might want to learn that this work can also be found on a server that speaks that famous “http”-protocol too. At address http://www.cc.jyu.fi/∼erkkila/thesis/ you should be able to find the LATEX sources, the test software code that has been typeset into the appendices, test scripts, and some other related things. If you ever find them useful, that's great, as that's more than I could ever expect ;-) As is pretty much obvious, this work is heavily inspired by the work that the whole BSD community is doing. While the main reasons for that are actually quite political, it is also because of the way that the community handles politics: it's not in what they say, but in what they do. Thus, if this work inspires anyone to hack on something, anything, it would be interesting to know about that, as that would be the best reward I can ever get. i Glossary API Application Programming Interface. Whatever rules a programmer must abide in order to use some other software component from another software compo- nent Asynchronous I/O With asynchronous I/O, I/O operations proceed independently from the execution of the process that initiated them BSD Unix Berkeley Software Distribution Unix, the branch of Unix that was devel- oped in University of Berkeley, California. OpenBSD is a derivative of BSD Unix, like other systems such as NetBSD, FreeBSD, and Macintosh OS X Exokernel A kernel of an operating system model, where the kernel is reduced to merely controlling access to hardware resources. All higher level functional- ity (such as device drivers and the abstractions they offer) are provided by application libraries Fork The act of a software project splitting off from another project. Also the project that does the forking. The term comes from the Unix fork() system call that creates a new process from the current one GUI Graphical User Interface IPC Interprocess communication. Means exchange of data between processes. Fa- cilities for IPC in practically all Unix system include signals, (anonymous and named) pipes, files, sockets, and shared memory IPL Interrupt Priority Level. Hardware and software interrupts are associated with a level, and kernel can delay interrupt handling (“mask” interrupts) until it is ready to deal with interrupts that have happened on some specific level. This way access to critical data in the kernel can be protected Kernel The core of an operating system. A kernel handles accessing hardware, and provides a uniform interface to hardware for application software Kqueue A generic kernel event notification mechanism, common in BSD variants. It provides an extensible framework for various kernel events, and a notification interface for applications interested in those events ii JACK Jack Audio Connection Kit. A low-latency audio server project for Unix systems Preemptive scheduling Preemptive scheduling means a kind of multitasking, where scheduler may interrupt the execution of a process in order to switch to another process. Preemptive kernel means that a kernel can allow this to happen also when the process is executing in kernel mode QNX A fault-tolerant, real-time, distributed operating system that is based on a microkernel architecture Scheduler The part of the operating system that apportions CPU time to processes SMP Symmetric Multi-Processing. Means that operating system and its processes run on more than one CPU, where CPUs have symmetrical relation to each other, that is, no CPU is treated special SMT Simultaneous multithreading. A performance improvement technique in hardware, which means that a CPU can have several threads executing in parallel. Does not usually provide any benefit unless the operating system knows how to utilize this feature. The system could see deal with it as a variation of SMP SUSv3 Single Unix Specification version 3, the most prevalent standard for the interfaces of Unix systems. Current version combines SUSv2 and the POSIX standard (the older Unix standard published by IEEE), and also in many ways supersedes POSIX System V The version of Unix created by AT&T, first released in 1983. Historically the two major flavors of Unix were considered to be System V and BSD, from which most of the Unix flavors were derived from Thread A thread is a kind of process that shares some execution context with other threads. A process can be viewed as a thread that has its execution context totally isolated from other processes. What this means in practice may be partially system-dependent Unix Originally a system developed at Bell Labs at AT&T. In this paper the term is used to mean a family of operating systems. Unix is a trademarked term, so its use is controlled and for correctness, all occurrences of it in this paper should possibly be replaced with “Unix-like”. This, however, is ridiculous iii X Window System The mostly commonly implemented graphics system in most Unix operating systems. It has client-server architecture, where server accesses the underlying graphics hardware and clients send requests to the server to draw things to the screen. Often referred to simply as “X” iv Contents Preface i Glossary ii 1 Introduction 1 1.1 Real-Time audio servers . 2 1.2 4.4BSD Unix and its derivatives . 5 2 System design in BSD Unix derivatives 8 2.1 Processes . 8 2.2 Interrupts . 10 2.3 Scheduler . 11 2.4 I/O model and IPC . 12 2.5 Audio subsystems in BSD kernels . 13 2.5.1 NetBSD/OpenBSD kernel audio internals . 15 3 Real-Time audio servers 20 3.1 Real-Time systems . 20 3.2 Are audio servers necessary? . 21 3.3 Server requirements . 22 3.4 Acceptable latencies . 24 3.5 The context switching problem . 25 3.5.1 Working around the context switches . 29 3.6 Audio libraries . 30 3.6.1 Audio API emulation . 31 3.7 Working around real-time requirements . 33 3.8 Audio protocols . 35 3.9 Threaded programming . 37 4 Real-Time Unix design 39 4.1 Traditional Unix kernel design . 40 4.2 Semi-preemptibility . 41 4.3 Preemption in kernels with fine-grained locking . 43 v 4.4 Message passing kernels with LWKTs . 48 4.5 Monolithic kernels on microkernels . 52 4.6 Real-Time features of operating systems . 54 4.6.1 Memory locking . 54 4.6.2 Clocks and timers . 55 5 Real-Time scheduling 56 5.1 Traditional Unix scheduling and real-time applications . 56 5.2 SUS scheduling policies . 59 5.3 Scheduler implementations . 61 5.3.1 SVR4 scheduler . 61 5.3.2 Schedulers in BSD systems . 63 5.3.3 Some problems in scheduler implementations . 64 5.3.4 SMP and SMT . 64 5.3.5 Scheduling of client-server systems . 65 5.3.6 Security and scheduler partitioning . 68 5.4 Research on scheduling algorithms . 69 5.4.1 Fair-Share scheduling algorithms . 71 5.4.2 Approaches based on partitioning . 73 6 Interprocess communication 76 6.1 Signals . 76 6.1.1 Real-Time signals . 78 6.2 Pipes and sockets . 79 6.2.1 Anonymous and named pipes . 79 6.2.2 Sockets . 80 6.2.3 Synchronous and asynchronous I/O . 82 6.2.4 Kernel event queues . 84 6.3 Shared memory . 86 6.3.1 Techniques in Mach IPC . 90 6.3.2 Solaris doors . 91 6.4 Data on a network . 92 6.4.1 Stream and datagram sockets .