DEGREE PROJECT IN INFORMATION AND COMMUNICATION TECHNOLOGY, SECOND CYCLE, 30 CREDITS STOCKHOLM, SWEDEN 2017 Real-time audio processing for an embedded Linux system using a dual-kernel approach NITIN KULKARNI KTH ROYAL INSTITUTE OF TECHNOLOGY SCHOOL OF ELECTRICAL ENGINEERING Real-time audio processing for an embedded Linux system using a dual-kernel approach Masters Thesis submitted in partial fulfillment of the requirements for the Degree of Master of Science in ICT Innovation - Embedded Systems at KTH Royal Institute of Technology, Stockholm NITIN KULKARNI Master’s Thesis Report Supervisor: Dr. Stefano Zambon Examiner: Dr. Carlo Fischione TRITA-EE 2017:140 Acknowledgment I would like to sincerely thank my parents and friends for their unwavering support and love all throughout this journey. I would like to thank Prof. Carlo Fischione for his guidance as my examiner and providing me this opportunity to work on Real-time embedded Linux. I am indebted to my colleague Sharan Yagneswar for making time to provide the right feedback and questions that helped me fine tune the project. Finally, I would like to thank all of them who have contributed in my graduate programme at Eindhoven University of Technology and at KTH Royal Institute of Technology, Stockholm. Stockholm, October 2017 Nitin Kulkarni Abstract Professional audio processing systems such as digital musical instruments, audio mixers, etc. must operate with very tight constraints on overall processing latency and CPU performance. Consequently, traditional implementations are still mostly based on specialized hardware like Digital Signal Processors (DSP) and Real-Time Operating Systems (RTOS) to meet such requirements. However, such systems are minimalistic in nature and they lack many features (e.g. network connectivity, wide hardware support, etc.) that a general-purpose operating system such as Linux offers. Linux is a very popular choice for the operating system used in embedded devices, and many developers have started to use it for designing real-time systems with relaxed timing constraints. However, none of the available solutions using a standard Linux kernel can satisfy the low-latency requirements of professional audio systems. In this thesis, a dual kernel approach is employed to enable an embedded Linux system to process audio with low roundtrip latency. The solution is developed using the Xenomai framework for real-time computation, which is based on a technique known as interrupt pipeline (I-pipe). I-Pipe enables interrupt virtualization through a micro-kernel running between the Linux kernel and the interrupt controller hard- ware. The designed system includes an x86 Atom System-on-Chip (SoC), an XMOS microcontroller and audio converters to and from the analog domain. Custom kernel drivers and libraries have been developed to expose the audio programming func- tionalities to programs running in user-space. As a result, the system can achieve robust real-time performance appropriate for professional audio applications, and at the same time it has all the advantages of a traditional Linux solution such as compatibility with external devices and ease of programming. The real-time capa- bility is measured by evaluating the performance in terms of worst case response time of the real-time tasks in comparison to the same metrics obtained under a standard Linux kernel. The overall roundtrip latency of audio processing is showed to be improved by almost an order of magnitude (around 2.5ms instead of 20ms). Keywords: Linux Device Driver, Scheduling, Interrupt Latencies, RTDM, SPI, XMOS, Audio Processing, Round-trip Latency. Sammanfattning Profesionella system för ljudbearbetning, som digitala musikinstrument, mixerbord, etc, arbetar med väldigt hårda krav på tidfördröjning och CPU-prestanda. Som en konsekvens har dessa system traditionellt implementerats på specialiserad hårdvara som specifika DSP-processor och speciella realtidsoperativsystem. Den typen av system är till sin natur minimalistiska och saknar många funktioner (till exem- pel nätverk och brett stöd för olika hårdvaror) som mer generella operativsystem, som Linux, kan erbjuda. Linux är ett väldigt populärt val av operativsystem för inbyggda system och många utvecklare har även börjat använda det till realtidssys- tem med mindre hårda tidskrav. Det finns dock idag inte någon lösning med en standard-linuxkärna som kan tillfredsställda de krav på låg fördröjning som krävs för användning i profesionella ljudsystem. I det här examensarbetet används en dubbelkärneuppsättning för att ge ett inbyggt Linuxsystem möjlighet att bearbeta digitalt ljud med låg fördröjning. Lös- ningen använder Xenomai-ramverket för realtidsberäkningar baserat på en teknik kallad interrupt pipeline (I-pipe). I-pipe ger möjlighet att virtualisera interrupt genom en mikrokärna som körs som ett lager mellan Linuxkärnan och hårdvarans interruptcontroller. Det resulterande systemet inkluderar ett x86 Atom-enchipssystem, en XMOS microcontroller, och ljudkonverterare till och från analoga ljud in- och utgångar. Drivrutiner och bibliotek utvecklas för att ge direkt tillgång till ljudfunktioner från applikationer. Systemet ges därmed robust realtidsprestanda som gör det lämpligt för profesionella ljudtillämpningar samtidigt som det behåller alla fördelar från ett traditionellt Linuxsystem, som kompabilitet med extern hårdvara och en- klare applikationsutveckling. Systemets realtidsprestanda utvärderas som den max- imala uppmätta tidfördröjning vid realtidsberäkningar jämfört med motsvarande beräkningar på en standardlinuxkärna. Resultaten visade på en förbättring på näs- tan en storleksordning (ca 2,5ms mot 20ms). Nyckelord: RTDM, Linuxdrivrutin, Schemaläggning, avbrottsfördröjning, inter- ruptfördröjning, ljudbearbetning, tidsfördröjning, XMOS, SPI. Contents List of Figures List of Tables List of Abbreviations 1 Introduction 1 1.1 Problem Statement . 3 1.2 Goals . 3 1.3 Related Work . 4 1.4 The Approach . 5 1.5 Outline . 6 2 Background 7 2.1 Real Time Systems . 7 2.1.1 Basic concepts of Real-time systems . 8 2.1.2 Classification of Real-time systems . 11 2.1.3 Real-time operating system . 12 2.2 Audio Processing Systems . 13 2.2.1 A typical audio processing chain . 13 2.2.2 Audio processing - Software architecture . 15 2.2.3 Audio processing - use cases . 17 2.3 Linux . 17 2.3.1 User view of Linux . 18 2.3.2 Software architecture of Linux kernel . 19 2.3.3 Benefits of using Linux . 22 2.3.4 Drawbacks of Linux . 23 2.4 Approaches to Real-time Linux . 24 2.4.1 PREEMPT_RT . 25 2.4.2 LITMUSRT ............................ 27 2.4.3 RTLinux . 27 2.4.4 RTAI . 28 2.5 Xenomai . 31 2.5.1 Xenomai Architecture . 32 2.5.2 Real-time Driver Model . 36 2.5.3 Xenomai Services . 38 2.6 XMOS . 39 3 Xenomai setup for an Intel Atom board 41 3.1 Intel Joule specifications . 41 3.2 Yocto build system . 42 3.2.1 OpenEmbedded . 43 3.2.2 Building a Customized Linux Image using Yocto . 44 3.3 Applying I-pipe and Xenomai patches . 45 3.3.1 Applying I-pipe Patch . 46 3.4 Xenomai kernel configuration . 46 3.5 Installing Xenomai libraries and validating the setup . 47 4 Development of a custom Real Time Driver for Audio over SPI 49 4.1 Audio over SPI driver . 49 4.1.1 Protocol between Joule and XMOS . 50 4.2 Driver Architecture . 52 4.2.1 Chained GPIO interrupts . 52 4.2.2 Hardware controllers involved . 53 4.3 Driver Implementation . 53 4.3.1 Writing the RTDM driver . 53 4.3.2 Modifications to platform drivers . 54 4.3.3 User space wrapper . 56 5 Benchmarking 57 5.1 Benchmarking metrics . 57 5.1.1 Round-trip latency . 57 5.1.2 Scheduling latency . 58 5.1.3 Driver Interrupt servicing latency . 60 5.1.4 CPU Usage . 61 6 Conclusion 63 6.1 Summary and outlook . 63 6.2 Future Work . 64 6.2.1 Asynchronous Driver . 64 Bibliography 65 List of Figures 1.1 Major features of Linux and RTOS indicating the differences between the two types of operating systems. 2 2.1 Typical real time audio processing system [1]. 7 2.2 Interrupt to task latency. 9 2.3 Preemptive scheduling in a real-time system. 10 2.4 A typical audio processing system. 13 2.5 Generic software architecture for handling audio services. 16 2.6 User view of Linux . 18 2.7 Differenc between Monolithic and Micro-kernel architecture. 20 2.8 Software architecture of Linux operating system. 20 2.9 Interrupt abstraction based real-time Linux[2]. 25 2.10 Adeos architecture[3]. 29 2.11 Adeos interrupt pipe[3]. 30 2.12 IRQ handling through the I-pipe enabled kernel[4]. 32 2.13 Xenomai enabled system architecture. 33 2.14 Dual kernel Cobalt architecture of Xenomai. 34 2.15 Single kernel Mercury architecture of Xenomai. 35 2.16 Xenomai API support for different skins. 35 2.17 RTDM layer interaction with other system layers [5]. 36 2.18 xCORE architecture. [6]. 39 3.1 Yocto Layers. [7]. 44 4.1 Communication between Intel Joule and XMOS microcontroller with buffers. 50 4.2 Sequence diagram of the communication protocol between XMOS and Joule. 51 4.3 The Audio over SPI driver architecture. 52 5.1 Histogram of worst case scheduling latencies in standard Linux. 59 5.2 Histogram of worst case scheduling latencies in dual kernel Linux. 60 5.3 Probability of interrupt servicing latency in normal Linux kernel. 61 5.4 Probability of interrupt servicing latency in Xenomai enabled Linux. 61 List of Tables 2.1 Key differences between Hard real-time systems and Soft real-time systems. 12 2.2 Overview of Real-time frameworks for Linux . 25 2.3 Device description fields for device registration. 37 3.1 Intel Joule Specifications. 42 5.1 Round trip latencies comparison. 58 5.2 Scheduling latencies for single and dual kernel Linux. 59 5.3 CPU usage comparison between Xenomai Linux and normal Linux. 62 List of Abbreviations ADC ................ Analog to Digital Converter AES ................. Audio Engineering Society API .................