The Flux OS Toolkit: Reusable Components for OS Implementation Bryan Ford Kevin Van Maren Jay Lepreau Stephen Clawson Bart Robinson Jeff Turnery Department of Computer Science, University of Utah Salt Lake City, UT 84112 [email protected] http://www.cs.utah.edu/projects/flux/ Abstract 1 Introduction To an unappreciated degree, research both in operating As operating system functionality continues to expand systems and their programming languages has been severely and diversify, it is increasingly impractical for a small group hampered by the lack of cleanly reusable code providing to implement even a basic useful OS coreÐe.g., the func- mundanelow-levelOS infrastructuresuch as bootstrapcode tionality traditionally found in the Unix kernelÐentirely and device drivers. The Flux OS Toolkit solves this problem from scratch. Furthermore, in most research domains, only by providing a set of clean, well-documented components. a few speci®c areas provide fodder for interesting research These componentscan be used as basicbuildingblocks both topics. For example, from reading an OS conference pro- for operating systems and for booting language run-time ceedings, one might be given the impression that build- systems directly on the hardware. The toolkit's implementa- ing an OS ªonlyº involves writing a virtual memory sys- tion itself embodies reuse techniques by incorporating com- tem, an IPC system, a ®le system, a scheduler, some fast ponents such as device drivers, ®le systems, and network- local-area network code, and a pro®ler to produce nice bar ing code, unchanged, from other sources. We believe the kit charts. However, as any experienced OS builder knows, also makes feasiblethe production of highlyassured embed- many of the problems involved in building an OS have al- ded and operating systems: by enabling reuse of low-level ready been solved countless times, and just aren't interest- code, the high cost of detailed veri®cation of that code can ing to researchers. For example, any realistic OS, in or- be amortized over many systems for critical environments. der to be useful even for research, typically includes boot The OS toolkit is already heavily used in several different loader code, kernel startup code, variousdevice drivers, ker- OS and programming language projects, and has already nel printf and malloc code, a kernel debugger, etc. A catalyzed research and development that would otherwise research project starting a new OS completely from scratch never have been attempted. would invariably spend at least the ®rst six months simply writing such infrastructureªgrungeº before even starting on the interesting aspects of the OS. 1.1 Related work Most OS researchers have realized this problem of high Thisresearchwas supportedin partbytheDefenseAdvancedResearch Projects Agency,monitored by the Department of the Army under contract startup cost, and have resorted to cannibalizing BSD, Mach, number DABT63±94±C±0058, and Rome Laboratory, Air Force Material or other freely available OSes rather than reinventing the Command,USAF, underagreementnumberF30602±96±2±0269. The U.S. wheel. Mach used BSD, Linux [13], and vendors' device Government is authorized to reproduceand distribute reprints for Govern- drivers; SPIN [3] uses device drivers from FreeBSD; and mental purposes notwithstanding any copyright annotation hereon. y U.S. Department of Defense VINO [17] takes its device drivers, bootstrap code, and low- Copyright 1997IEEE. Personal use of this material is permitted. How- level support for virtual memory from NetBSD. ever, permission to reprint/republish this material for advertising or promo- While this approach saves time, the developer must man- tional purposesor for creatingnew collective worksfor resale or redistribu- tion to servers or lists, or to reuse any copyrightedcomponent of this work ually examine and dissect the old OS; it would save even in other works must be obtained from the IEEE. more time if the developer could simply obtain a set of 1 clearly-documented components. It would also enable re- in Fluke'satomicoperations[18]), which tookonlya month. search by those whose primary expertise is in areas other We have found this ability to prototype radical designs in a than operating systems, e.g., programming language re- ªrealº kernel tobe crucial to choosingdesignsthat are worth searchers who wish to explore the effects of higher-level lan- fully developing. guages runningdirectlyon the hardware. Thisis thepurpose of the Flux OS Toolkit. Recent research projects such as Exokernel [6], SPIN [3], 2 Toolkit design and VINO [17], focus on creating extensible systems which allow applications to modify the behavior of the core OS to suit their particular needs. However, these systems still de- ®ne a particular, ®xed set of ªcoreº functionality and a set of The Flux OS Toolkit is a framework and set of modular- policies by which the core can be used and extended. The ized librarycode, together withextensive documentation[9] OS Toolkit, in contrast, makes no attempt to be a useful OS for the construction of operating system kernels, servers, initself and doesnot de®neany particularset of ªcoreº func- and other core OS functionality. The goal is for develop- tionality, but merely provides a suite of components from ers to take the OS Kit and immediately have a starting point which real OSes can be built. for investigating ªrealº OS issues such as scheduling, VM, Other approaches involved creating an operating system IPC, ®le systems, or security. Researchers in programming built from a complex object-oriented framework, such as languages for systems software bene®t as well, as the toolkit in the Choices[5] or Taligent [15] work. Although such makes it easy to run language systems on the bare hardware. efforts have been in¯uential in other OS projects, such as The intention of this toolkit is not to ªwrite the OS for Spring,theydonotappear tohave been widelyused. Incon- youº; we certainly want to leave the OS writing to the OS trast, the OS Kit exhibits a less ambitious, but more prag- writer. The dividing line between the ªOSº and the ªOS matic, and we believe more effective, approach to software Toolkit,º as we see it, is basically the line between what OS design and re-use. Gabriel distinguished two approaches writers want to write and what they would otherwise have to software design and implementation, sardonically label- to write but don't really want to. Naturally this will vary ing them ªThe Right Thingº and ªWorse is Betterº [12]. among different OS groups and developers. If you really The former is characterized by interface perfection at the want to write yourown x86 protected-mode startup code, or cost of implementation complexity (e.g., Lisp with CLOS), have found a clever way to do it ªbetter,º you are perfectly whereas the latter sacri®ces interface elegance and com- free to do so and simply not use the corresponding code in pleteness for simplicity of implementation (e.g., Unix and our toolkit. However, the OS Kit is modular enough so that C). Gabriel makes a strong case that ªWorse is Betterº is the you can still easily use other parts of it to ®ll in other func- more sucessful approach, and we believe that the OS Kit ex- tional areas you don't want to have to deal with yourself (or empli®es this lesson. areas that you just don't have time to do ªyet.º) As such, the toolkit is designed to be usable either as a 1.2 Historical genesis of the OS Toolkit whole or in arbitrary subsets, as requirements dictate. It is useful not only for kernels but also for other OS-related pro- We followedthe cannibalization approach in our own OS grams, such as boot loaders or servers running on top of a research for some time. However, starting in 1995, that ap- microkernel. proach gradually evolved, resultingin what became theFlux While the OS Kit currently runs on x86 PCs, it is de- OS Toolkit, or ªOS Kitº. Because we were ®nding our ver- signed to be portable to other architectures, and most of the sion of Mach [11] too constraining a vehicle in which to OS Kit's exported interfaces are architecture-neutral. Most prototype new ideas, we developed a series of experimen- of the heavily architecture-speci®c aspects of the OS Kit are tal kernels to try out ideas before designing our Fluke ker- isolated in a single component, the low-level kernel support nel [10]. In doing so, we gradually modularized and for- library, whose purpose is to provide easy access to the raw malized the libraries of support code we developed, proto- privileged-mode hardware facilities without adding over- typing the OS Kit along the way. These experimental ker- head or obscuring the underlying abstractions. For exam- nels embodied radical changes to fundamental aspects of OS ple, on the x86, the kernel support library includes functions structure which would have been impossible to explore in to directly create and manipulate x86 page tables and seg- an existing operating system. One of these kernels explored ment registers. Other OS Kit components can, and often implementations of high performance kernel-mediated ca- do, provide higher-level architecture-neutral facilities built pabilities and IPC paths, and took about 2 weeks to develop on these low-level mechanisms; however, the architecture- from scratch; theotherexploredinterruptibilityof kernelop- speci®c interfaces always remain accessible in order to pro- erations at arbitrary points (®nding a more ®nal expression vide maximum ¯exibility. 2 3 Sample components boots from standard boot loaders is as easy as writingan or- dinary ªHello Worldº application in C. The toolkit currently contains ®fteen major libraries, The OS Kit also provides the necessary code to initialize ranging from uniprocessor and multiprocessor bootstrap and start multiple processors in a symmetric multiprocess- code, through memory management, to support for popular ing (SMP) system.