THE DESIGN AND APPLICATION OF AN EXTENSIBLE OPERATING SYSTEM Leendert van Doorn VRIJE UNIVERSITEIT THE DESIGN AND APPLICATION OF AN EXTENSIBLE OPERATING SYSTEM ACADEMISCH PROEFSCHRIFT ter verkrijging van de graad van doctor aan de Vrije Universiteit te Amsterdam, op gezag van de rector magnificus prof.dr. T. Sminia, in het openbaar te verdedigen ten overstaan van de promotiecommissie van de faculteit der Exacte Wetenschappen / Wiskunde en Informatica op donderdag 8 maart 2001 om 10.45 uur in het hoofdgebouw van de universiteit, De Boelelaan 1105 door LEENDERT PETER VAN DOORN geboren te Drachten Promotor: prof.dr. A.S. Tanenbaum To Judith and Sofie Publisher: Labyrint Publication P.O. Box 662 2900 AR Capelle a/d IJssel - Holland fax +31 (0) 10 2847382 ISBN 90-72591-88-7 Copyright ! 2001 L. P. van Doorn All rights reserved. No part of this publication may be reproduced, stored in a retrieval system of any nature, or transmitted in any form or by any means, electronic, mechani- cal, now known or hereafter invented, including photocopying or recording, without prior written permission of the publisher. Advanced School for Computing and Imaging This work was carried out in the ASCI graduate school. ASCI dissertation series number 60. Parts of Chapter 2 have been published in the Proceedings of the First ASCI Workshop and in the Proceedings of the International Workshop on Object Orientation in Operat- ing Systems. Parts of Chapter 3 have been published in the Proceedings of the Fifth Hot Topics in Operating Systems (HotOS) Workshop. Parts of Chapter 5 have been published in the Proceedings of the Sixth SIGOPS Euro- pean Workshop, the Proceedings of the Third ASCI Conference, the Proceedings of the Ninth Usenix Security Symposium, and filed as an IBM patent disclosure. Contents Acknowledgments iv Samenvatting vi 1 Introduction 1 1.1 Operating Systems 2 1.2 Extensible Operating Systems 4 1.3 Issues in Operating System Research 6 1.4 Paramecium Overview 7 1.5 Thesis Contributions 10 1.6 Experimental Environment 12 1.7 Thesis Overview 14 2 Object Model 15 2.1 Local Objects 16 2.1.1 Interfaces 17 2.1.2 Objects and Classes 21 2.1.3 Object Naming 23 2.1.4 Object Compositions 27 2.2 Extensibility 30 2.3 Discussion and Comparison 31 3 Kernel Design for Extensible Systems 34 3.1 Design Issues and Choices 36 3.2 Abstractions 40 3.3 Kernel Extension Mechanisms 41 3.4 Paramecium Nucleus 45 3.4.1 Basic Concepts 46 i ii Contents 3.4.2 Protection Domains 48 3.4.3 Virtual and Physical Memory 51 3.4.4 Thread of Control 54 3.4.5 Naming and Object Invocations 67 3.4.6 Device Manager 72 3.4.7 Additional Services 74 3.5 Embedded Systems 75 3.6 Discussion and Comparison 76 4 Operating System Extensions 84 4.1 Unified Migrating Threads 85 4.1.1 Thread System Overview 85 4.1.2 Active Messages 88 4.1.3 Pop-up Thread Promotion 90 4.1.4 Thread Migration and Synchronization 93 4.2 Network Protocols 96 4.2.1 Cross Domain Shared Buffers 96 4.2.2 TCP/IP Protocol Stack 100 4.3 Active Filters 101 4.3.1 Filter Virtual Machine 105 4.3.2 Example Applications 109 4.4 Discussion and Comparison 112 5 Run Time Systems 116 5.1 Extensible Run Time System for Orca 117 5.1.1 Object-based Group Active Messages 120 5.1.2 Efficient Shared Object Invocations 123 5.1.3 Application Specific Optimizations 125 5.2 Secure Java Run Time System 128 5.2.1 Operating and Run Time System Integration 131 5.2.2 Separation of Concerns 132 5.2.3 Paramecium Integration 136 5.2.4 Secure Java Virtual Machine 137 5.2.5 Prototype Implementation 152 5.3 Discussion and Comparison 153 Contents iii 6 Experimental Verification 158 6.1 Kernel Analysis 159 6.2 Thread System Analysis 170 6.3 Secure Java Run Time System Analysis 174 6.4 Discussion and Comparison 179 7 Conclusions 181 7.1 Object Model 181 7.2 Kernel Design for Extensible Systems 183 7.3 Operating System Extensions 186 7.4 Run Time Systems 187 7.5 System Performance 189 7.6 Retrospective 189 7.7 Epilogue 190 Appendix A: Kernel Interface Definitions 192 Bibliography 196 Index 212 Curriculum Vitae 216 Acknowledgments Although my advisor, Andy Tanenbaum, thinks otherwise, I view a Ph.D. as a period of learning as much as you can in as many different subjects that are interesting. In this respect I took full advantage of my Ph.D.: I did work ranging from programming my own EEPROMs, to secure network objects, digital video on demand, and a full blown new operating system with its own TCP/IP stack, an experimental Orca runtime system, and a secure Java virtual machine. I even considered building my own network hardware but was eventually persuaded to postpone this. This advice no doubt sped up the completion of this thesis considerably. There are a large number of people who assisted and influenced me during my Ph.D. and for which I have great admiration. First of all there is my advisor, Andy Tanenbaum, who was always quick to jump on my half-baked ideas and forced me to give better explanations, and of course Sharon Perl who picked me as an intern at Digital Systems Research Center. I really felt at home at SRC, in fact so much that I came back the next summer to work with Ted Wobber on secure network objects. Two other SRC employees who deserve an honorable mention are Mike Burrows for our technical discussions and for pointing out to me that puzzles are actually fun, and Martin Abadi for his conciseness and precision; Virtues I often lack. Of course, I should not forget Rob Pike, who thought I was spending too much time at DEC SRC and that I should also visit Bell Labs. I was more than eager to take him up on that. During that summer I had ample opportunity to play with Inferno, Plan 9 and digital video. I'm forever in debt to Ken and Bonnie Thompson for their hospitality, which still extends to today. Most of my time at Bell Labs I spent with Dave Presotto and Phil Winterbottom. Phil reminded me in more than one way of Mike Burrows, albeit a much more critical version. As so many Ph.D. students, I took longer than the officially approved four years to finish my thesis. Rather than staying at the university, I was persuaded to join IBM T.J. Watson Research Center as a visiting scientist and finish my thesis there. After four months it was clear I liked the place; industrial research laboratories are surprisingly similar; and I joined IBM as a research staff member. Here I worked on my secure Java Virtual Machine, but also got distracted enough that the writing of my thesis got delayed considerably, despite the almost daily reminders by my group members and not infrequent ones from my advisor. My group members included Charles Palmer, Dave Safford, Wietse Venema, Paul Karger, Reiner Sailer and Peter Gutmann. I managed to tackle this daily nuisance by stating that inquiries about my thesis progress were in fact voluntary solicitations to proofread my thesis. Some, who iv Acknowledgments v did not get this in time, ended up proof reading my drafts. Then there were others who practically forced me to give them a copy to proofread. I guess they could not stand the suspense any longer, which I had carefully cultivated over the last two years. Without going into detail who belonged to which group, I would like to thank Charles Palmer, Jonathan Shapiro, Dan Wallach, Ronald Perez, Paul Karger, and Trent Jaeger who provided me with much useful feedback on my thesis or papers that comprise it. Besides the people mentioned above, there is the long cast of characters who contributed to my thesis in one way or another. First of all there is the old Amoeba group of which I was part and where I learned many of the traits and got my early ideas. This group, at that time, consisted of Frans Kaashoek, Ed Keizer, Gregory Sharp, Hans van Staveren, Kees Verstoep, and Philip Homburg. Philip has been my office mate at the VU for most of the time and someone with whom I could exchange hacking problems and who provided feedback on part of this thesis. Then there is the ORCA group, consisting, at that time, of Henri Bal, Koen Langendoen, Raoul Bhoedjang, Tim Ruhl, and Kees Verstoep. They did a marvelous job at the Orca run- time system which I took advantage of in my implementation. In the Dutch system, once a thesis is approved by the advisor, it is passed on to a reading committee for a final verdict. My reading committee consisted of Frans Kaashoek, Paul Karger, Sape Mullender, Charles Palmer, and Maarten van Steen. They provided much useful and insightful feedback which greatly improved this thesis. I also wish to thank the Department of Mathematics and Computer Science of the Vrije Universiteit, N.W.O., Fujitsu Micro Electronics Inc., and IBM T.J. Watson Research Center for providing support for carrying out the research and especially for the generous funding for trips, books, and equipment. Without a doubt, the hardest part of writing this thesis was the Dutch summary. Mainly because Dutch lacks the appropriate translation for many English computer terms. Fortunately, our Belgium neighbors were very helpful in this respect and put up a web site (in proper Dutch, een webstek) with lots of useful suggestions.