Communication in Microkernel-Based Operating Systems Dissertation zur Erlangung des akademischen Grades Doktoringenieur (Dr.-Ing.) vorgelegt an der Technischen Universit¨atDresden Fakult¨atInformatik eingereicht von Diplom-Informatiker Ronald Aigner geboren am 20. Mai 1976 in Dresden Gutachter: Prof. Dr. rer. nat. Hermann H¨artig Technische Universit¨atDresden Prof. Dr. Klaus Meyer-Wegener Friedrich Alexander Universit¨atErlangen-N¨urnberg Tag der Verteidigung: 21. Januar 2011 Dresden im Mai 2011 Communication in Microkernel-Based Operating Systems Ronald Aigner Technische Universit¨atDresden July 2010 ii Abstract Communication in microkernel-based systems is much more frequent than system calls known from monolithic kernels. This can be attributed to the placement of system services into their own protection domains. Communication has to be fast to avoid unnecessary overhead. Also, communication channels in microkernel-based systems are used for more than just remote procedure calls. In distributed systems, which also have a componentized design, it is state of the art to use tools to generate stubs for the communication between components. The communication interfaces of components are described in an interface definition language (IDL). In contrast to distributed systems, components of a microkernel-based system run on the same architecture and message delivery is guaranteed. In this Thesis, I explore the different kinds of communication, which can be used in microkernel-based systems, as well as their possible representation in IDL. Specifically, I introduce the syntax to describe kernel objects in IDL. I discuss the complexity of IDL compilers and its relation to the complexity of the IDL. Furthermore, I evaluate the performance of the communication stubs generated by different IDL compilers and discuss techniques to minimize performance overhead in generated stubs. I validated these techniques by implementing the Drops IDL Compiler { Dice. Finally, this Thesis presents a mechanism to measure the frequency and performance of invocations of gen- erated communication code. I used this technique to conduct measurements in highly complex systems and introducing the least possible overhead. iii Acknowledgments First, I would like to thank my supervisor, Prof. Hermann H¨artig. He provided the environment to pursue my research ideas and to learn so much about operating systems and microkernels in particular. I want to express my gratitude for not giving up on me and supporting this work. He created a nourishing environment by shepherding so many intelligent people in his group. Furthermore, I would like to thank my reviewer, Prof. Meyer-Wegener. His infinite patience and positive feedback allowed me to look at this work from different angles and find new ideas behind the corners. Of course this Thesis would not have been possible without all the wonderful people of the Operating Systems chair at TU Dresden. Discussions with them always questioned my assumptions and made me constantly aware of flaws in my work. I would especially like to thank Lars Reuther, Martin Pohlack and Adam Lackorszynski. Last but certainly not least, I wish to thank my family. My wife and kids gave me the balance to stay on track, even if the work seemed overwhelming. And my parents: You made me who I am. iv Contents 1 Introduction1 2 Related Work5 2.1 Message Passing Interface...........................5 2.2 Remote Procedure Calls............................5 2.3 Sun RPC....................................6 2.4 Firefly......................................6 2.5 Spring......................................7 2.6 Pebble......................................7 2.7 MIG - The Mach Interface Generator.....................8 2.7.1 Specification Language.........................9 2.7.2 Usage Problems and Hints.......................9 2.7.3 Optimizations.............................. 10 2.8 Flick - Flexible Interface Compiler Kit.................... 11 2.9 Concert..................................... 14 2.10 Spec#...................................... 14 2.11 Barrelfish.................................... 15 2.12 CORBA..................................... 15 2.13 Peregrine.................................... 16 2.14 IDL Compilers for L4 µ-Kernels........................ 17 2.14.1 IDL4 ................................... 17 2.14.2 Magpie................................. 17 2.15 DCE++..................................... 18 2.16 Heidi....................................... 18 2.17 Ada....................................... 19 2.18 Summary.................................... 19 3 Forms of Communication 21 3.1 Message Passing................................ 21 3.2 Asynchronous Servers............................. 23 3.3 Shared Memory................................. 25 3.4 Streaming.................................... 27 3.5 Security..................................... 27 v vi CONTENTS 3.5.1 Access Control............................. 28 3.5.2 Shared Memory Communication................... 29 3.5.3 Robustness............................... 30 3.5.4 Denial of Service............................ 30 3.6 Summary.................................... 31 4 Portability 33 4.1 Platform Independence............................. 34 4.1.1 Platform Dependent Types and Platform Independent Attributes. 35 4.2 Optimization.................................. 36 4.2.1 Parameter reordering......................... 37 4.2.2 Copy Avoidance............................ 38 4.2.3 Memory Allocation........................... 39 4.2.4 Target Language Compiler...................... 39 4.3 Usability..................................... 40 4.4 Summary.................................... 42 5 Integration 43 5.1 IDL Extensions................................. 43 5.1.1 Indirect Parts.............................. 43 5.1.2 Aliasing Types............................. 44 5.1.3 User-Defined Types.......................... 44 5.1.4 Concurrent Data Access........................ 45 5.1.5 Operation Identifiers.......................... 45 5.1.6 Array Size................................ 46 5.2 Target Code Generation............................ 47 5.2.1 Factory Concept............................ 48 5.2.2 Message Buffer Layout......................... 49 5.2.3 Tracing................................. 51 5.2.4 Test Suite Generation......................... 51 5.3 Infrastructure Integration........................... 52 5.3.1 Automatic Service Lookup...................... 52 5.3.2 Resource Accounting.......................... 52 5.3.3 Resource Reservation......................... 53 5.4 Real-Time Communication.......................... 53 5.5 Summary.................................... 54 6 Evaluation 55 6.1 Analyze Method Invocations......................... 55 6.1.1 Scenario 1: L4Linux with L4 console................. 56 6.1.2 Scenario 2: L4Linux with L4 console and ORe network....... 61 6.1.3 Scenario 3: Netfilter.......................... 62 6.1.4 Scenario 4: L4Linux with DOpE................... 65 6.1.5 Scenario 5: Verner { video player................... 66 CONTENTS vii 6.1.6 Scenario 6: JTop { a DOpE application............... 68 6.1.7 Scenario 7: BLAC........................... 69 6.1.8 Summary................................ 70 6.2 Performance Benchmark............................ 75 6.2.1 Comparing Stub Code Generators for Fiasco............ 75 6.2.2 Comparing Stub Code Generators for Hazelnut........... 75 6.2.3 Comparing Hardware Architectures................. 77 6.2.4 Comparing Compiler Versions..................... 77 6.2.5 Comparing Stub Code Generators for Pistachio........... 79 6.2.6 Comparing Stub Code Generators for Linux sockets........ 82 6.3 Micro-benchmarks............................... 84 6.3.1 Short IPC................................ 84 6.3.2 Indirect string IPC versus direct IPC................. 85 6.3.3 Indirect string IPC versus Shared Memory.............. 88 6.4 Code Complexity................................ 89 6.4.1 Test Suite................................ 89 6.4.2 Tracing................................. 90 6.4.3 Assembly code generation....................... 90 6.4.4 Indirect String Communication.................... 91 6.4.5 Resource Reservation......................... 92 6.4.6 Default Timeout............................ 92 6.4.7 Generated Code............................ 93 6.4.8 IDL Compiler.............................. 94 6.4.9 dynrpc Code Complexity....................... 97 6.5 Summary.................................... 98 7 Conclusion and Outlook 99 7.1 Contributions.................................. 99 7.2 Outlook..................................... 100 A Performance Measurement Results 103 A.1 Performance of Stub-Code Generator for Fiasco............... 103 A.1.1 GCC version 3.4............................ 103 A.1.2 GCC version 2.95............................ 105 A.2 Performance of Stub-Code Generator for Hazelnut............. 107 A.3 Comparing Hardware Architecture...................... 109 A.4 Comparing Compiler Versions......................... 113 A.5 Comparing Stub-Code Generators for Pistachio............... 115 A.6 Comparing Stub-Code Generators for Linux sockets............ 117 viii CONTENTS List of Tables 6.1 Number of invocations for L4Linux with L4 console counted with instru- mented stubs.................................. 57 6.2 Size of IDL method calls for L4Linux with L4 console counted with in- strumented stubs................................ 58 6.3 Number of IPCs sorted by size for L4Linux with L4 console counted by kernel extension................................