FLEXATo ol for Building Ecient and Flexible Systems

John B Carter Bryan Ford Mike Hibler RavindraKuramkote

Jerey Law Jay Lepreau Douglas B Orr Leigh Stol ler and Mark Swanson

University of Utah Department of Computer Science

Abstract

Mo dern op erating systems must supp ort a wide variety of services for a diverse set of users Designers

of these systems face a tradeo b etween functionality and p erformance Systems likeMachprovide a set

of general abstractions and attempt to handle every situation which can lead to p o or p erformance for

common cases Other systems such as Unix provide a small set of abstractions that can b e made very

ecient at the exp ense of functionalityWe are implementing a exible system building to ol FLEX

that allows us to supp ort a p owerful op erating systems interface eciently by constructing sp ecialized

mo dule implementations at runtime FLEX improves the p erformance of existing systems by optimizing

interpro cess communications paths and relo cating servers and clients to reduce communications overhead

These facilities improve the p erformance of Unix system calls on Mach from Furthermore FLEX

can dynamically extend the kernel in a controlled fashion which gives user programs access to privileged

data and devices not envisioned by the original op erating system implementor

Overview

Op erating systems consist of a collection of interrelated entities that provide a variety of services Beyond

this general description the sp ecics of individual op erating systems varies widely dep ending on the decisions

made by each systems designers We b elieve that there is no optimal organization b ecause the optimal

design is dep endent on a particular systems resources and the needs of its clients No single op erating

system structure can p erform optimally in a wide varietyofenvironments The most imp ortant asp ect of an

op erating system from the users p oint of view is its application programming interface denition As long as

the op erating system eciently supp orts its interface and protection requirements it is irrelevant whether

this is provided via functions found at known lo cations in the user address space eg MSDOS via traps

to the kernel eg Unix via remote pro cedure calls RPCs to server pro cesses eg much of Mach or

any combination of these options To exploit the exibility that this p ersp ectiveprovides we are developing

the FLEX system building to ol

FLEX is a coarsegrained system building service that allows systems to b e dynamically constructed

using the implementation most appropriate to a given situation In this context a system is an entity

that interacts with the op erating system plus the op erating system itself FLEX provides functionalitytoan

op erating system similar to but more p owerful than that provided to a programming system by a linker

FLEX works by reading modules manipulating them as sp ecied by mo dule manipulation les and writing

the resulting executable either to a users address space or to a le for later use The activeentityin

FLEX is the OMOS server which runs as a privileged pro cess It p erforms several functions including

generating relo cating and binding ob ject les caching relo cated ob ject les for later use managing address

spaces and interacting with the kernel to add or delete kernel routines on demand FLEX can improvean

op erating system in two dierentways i by improving the p erformance of existing system implementations

and ii by dynamically extending op erating system functionality

One of the goals of FLEX is to allow the implementations of existing systems to b e improved without

requiring their comp onent mo dules or interfaces to change FLEX uses interface denition sp ecications

to dene how individual system comp onents interact with one another This sp ecication allows FLEX to

sp ecialize instances of a system resource or communications channel to a particular situation For example

when a client program rst connects to a particular service FLEX can determine where the server is phys

ically lo cated and provide the most ecientinterpro cess communications mechanism for this clientserver

interaction If the server is on a remote machine then RPC is appropriate while if the client and server have

b een placed in the same address space a capabilityof FLEX discussed b elow then some form of pro cedure

call is preferable Similarlyif FLEX is able to determine that most of the interactions with a server are

coming from a particular client it can intervene to make those interactions less exp ensive eg by mapping

the clientinto the server address space in a controlled manner

FLEXs other asp ect is its ability to extend or mo dify the op erating system dynamically in a mostly

transparentway FLEX can mo dify the op erating system on the y by binding routines to the kernel on a p er

client basis This abilityallows users or op erating systems designers to add or mo dify kernel functionality

The means by which this can b e done safely is discussed in the following section This feature of FLEX can b e

exploited in a number of ways For example this might b e used to give an unprivileged user program direct

control over its physical memory so that it can use a paging algorithm tuned to its particular access b ehavior

or realtime needs A long term goal of this asp ect of FLEX is to allow op erating systems researchers and

vendors to distribute functional mo dications or extensions to existing op erating systems to sites without

access to the op erating system source co de

Design of FLEX

FLEXs p ower comes from OMOSs ability to manipulate mo dules in complex ways its full access to interface

denitions its ability to derive sp ecialized implementations of mo dules and their interactions and its gener

alized constraint system for managing address spaces A mo dule is an entitycontaining a number of symbols

or linkage p oints and optionally a fragment of executable co de in the machine language of the target sys

tem Most mo dules pro cessed by FLEX are normal relo catable ob ject les pro duced by a compiler but there

are also several kinds of interface denition mo dules FLEX uses a Lisplike mo dule manipulation language

develop ed as part of the Jigsaw framework to sp ecify how particular mo dules are to b e combined This

language provides a complete implementation of mo dule name manipulation muchmorepowerful than exist

ing linkers Augmented with detailed knowledge of communication mechanisms and corresp onding argument

semantics this allows FLEX to transparently sp ecialize intermo dule communication

Within a mo dule a symb ol can either b e a declaration or a reference Declarations are most often

asso ciated with co de fragments that implement a particular system service while references are most often

asso ciated with co de fragments that invoke a particular system service FLEX is able to improveupon

existing system implementations by transparently substituting a particular declaration of a symb ol with

a functionally equivalent implementation that provides the same interface and semantics The details of

the lowlevel binding mechanism are b eyond the scop e of this pap er but are describ ed elsewhere For

example if a client pro cess contains a reference to a system call that normally is implemented as an RPC to

a userlevel server pro cess whichishowmany Unix system calls are emulated in Mach it can replace

the RPC interface with an LRPClikeinterface if it recognizes that the server is always lo cal LRPC

exploits memory sharing and thread migration to reduce the overhead of an RPC but it is only p ossible

between two pro cesses on the same machine Implementing the full LRPC mechanism requires mo dications

to the Mach thread mo to supp ort thread migration Similarly if the service is currently b eing provided

by a server that has b een loaded into the kernel address space FLEX can replace the RPC with a trap that

directly invokes the desired service These examples demonstrate the p otential that FLEX has for improving

the qualityofexisting systems

In addition to b eing able to sp ecialize interfaces FLEX is able to dynamically extend the op erating system

kernel or relo cate services in a transparentway The simplest example of how this can b e accomplished arises

when FLEX detects that a desired service is not currently b eing provided as can o ccur if the server crashes

or cho oses to quit due to inactivity In this case FLEX can restart the appropriate server An incremental

improvementuponthisisthat FLEX can start a new instance of the server that is sp ecialized to the clients

needs and make the appropriate binding FLEX go es b eyond these simple mechanisms by allowing mo dules

to b e relo cated b etween address spaces to provide more ecientintermo dule interactions For example if

FLEX observes or is informed that a particular userlevel service is used frequently and the server is trusted

it can bind the service into the kernel and replace calls to that server with traps This use of FLEX blurs the

distinction b etween macrokernels and microkernels whichwe b elieve should b e the case The microkernel

organization improves co de mo dularityandprovides an easy to understand design philosophy but when a

rigid enforcement of the microkernel philosophy impacts p erformance without signicant gains it should b e

relaxed transparently to regain the p erformance b enets of a macrokernel system Enabling optimization of

more interactions FLEX can transparently load clients and trusted servers in a single address space while

protecting the servers from client accesses We view supp ort for these inserver clients and inkernel

servers as the rst step in an evolution towards a single address space system that can eciently supp ort

existing programs written for existing systems

Finallywehave exp erimented with the option of linking sp ecially authorized routines into the kernel on

a p erclient basis FLEX lets kernel level services and asso ciated trap vectors b e added allowing users to

access privileged data and devices in a controlled fashion not envisioned by the original op erating system

implementor and thus not supp orted by an existing system call This gives FLEXd programs the abilityto

read or mo dify kernel data structures eg a controlled way to reintro duce the functionalityof devkmem

the abilitytoinvoke functions not provided by the default kernel eg direct control over its physical memory

and paging algorithm and the abilitytooverride or mo dify existing kernel services eg interp osing a

message logging layer in a particular pro cess IPC routines to supp ort fault tolerance Routines that interact

with existing kernel routines such as those that mo dify existing kernel data structures or device drivers

must b e written with great care to avoid corrupting the system Our currentauthentication mechanism is

quite simple only the OMOS server can install kernellevel routines and it will install only authorized

library routines that are provided by a sup eruser However these routines can b e installed on b ehalf of any

pro cess While simple this mechanism lets us protect the integrity of the OS kernel without signicantly

reducing overall exibility

Figure illustrates how FLEX can restructure the way that a particular user program interacts with the

op erating system In this example the user pro cess is using four services a database server a le server an

authentication server and a sp ecial extension to the kernel that lets the user program control its own physical

memory Because of the frequency of interactions the user pro cess has b een loaded into the database servers

address space Thus communications b etween the user pro cess and database server o ccur via protected

pro cedure calls involving traps or directly through normal pro cedure calls and shared memory dep ending

on the desired level of protection Additionally b ecause this user pro cess needs realtime p erformance

guarantees it uses its own paging mechanism implementedasaninkernel server to control its paging

b ehavior This mechanism uses normal kernellevel services to access disk and the VM hardware but

otherwise it is indep endent of the virtual memory subsystem after it acquires the desired amountofphysical

memory Similarly the leserver is a userlevel pro cess b eing executed lo callysointeractions with it use

LRPC to reduce communications overhead Finally the system authentication server is running remotelyso

all communications with it use standard RPC FLEXs ability to restructure systems in this way can greatly

enhance b oth p erformance and exibility

Colocated USER Local Remote Server Server Database PROCESS Process Process Server lookup()

pageout() LRPC

authenticate() ConventionalRPC

TRAP read()

In-Kernel

Server Kernel

Figure Example of How FLEX Can Restructure Programs

Results and Current Status

The core of FLEX is the OMOS server OMOS has b een integrated into the Mach OSF servers exec

function so it is invoked when FLEXd programs are executed As part of the preliminary implementationof

FLEX OMOS has b een used to reduce the overhead of communications b etween mutuallytrusting clientand

server pro cesses by replacing RPC invo cations with pro cedure calls In addition OMOS was augmented to

supp ort a protected region in a users address space This supp ort allows OMOS to load routines representing

extended op erating system functionalityinto that region reconcile unresolved references found in the added

mo dules with entry p oints dened in the kernel and provide sp ecial gating routines to the user pro cess so

that it can safely access the new functionalityWork is continuing on augmenting OMOSs interface denition

supp ort to handle manyvariations of argument semantics used by dierent communicationmechanisms In

addition to its role in FLEX OMOS has b een able to improve the p erformance of a numb er of running

systems byanaverage of by optimizing the lo cality of instruction references p erformed by the systems

and thereby improving cache and paging b ehavior In addition OMOS is providing a fast and p ortable

shared library service for an line CAD system with substantial sp eedups due to its caching of

relo cated images

Wehave p erformed three exp eriments that demonstrate the value of evolving an existing op erating

system Mach and the OSF Unix server into a exibly structured system

i We transparently merged a frequently accessed and trusted userlevel server the Unix server into

the kernels protection domain while fully preserving semantics We refer to this mechanism as inkernel

servers or INKS Client calls to the server are b ound either to a simple system call interface or to a full

RPC mechanism dep ending on the servers lo cation This optimization reduces the amountofwork done

copying arguments and executing the RPC control path to handle complex and infrequently encountered

message typ es We found that RPC p erformance improves by a factor of three Unix system calls to the

server improveby and the overall p erformance of large b enchmarks improves by

ii While p erforming the INKS work we discovered that many p erformance optimizations could b e

intro duced if the Machkernel supp orted a full migratingthreads mo del Wehave implemented sucha

mechanism fully decoupling threads from tasks Our prototyp e implementation of this mechanism has a

faster call path than the most optimized message path in the existing Machkernel

iii In order to take full advantage of the p otential of FLEX protection domains must b e decoupled

from address spaces Wehave prototyp ed this on Mach and the Unix server using the HP PARISCs

negrained protection mechanisms to provide protection b etween tasks in the same address space This

mechanism allows clients to b e loaded into servers Wehave successfully executed several unmo died user

programs in the same address space as the OSF server while protecting servers from client accesses

Individual system calls demonstrate dramatic sp eedups as much as a factor of four due to the avoidance of

microkernel calls to access system call arguments Even for IO intensive programs suchasls the use of

inserver clienttechnology reduced the total running time by

Related Work

There are a numb er of systems that provide various asp ects of FLEX although none supp orts its full generality

of optimizations Like FLEX Lipto provides architectural supp ort for mo dules that is indep endentof

protection domains and thereby allows communications b etween mo dules to b e optimized using techniques

similar to those describ ed ab ove However it was not designed to work with existing system implementations

and currently do es not include supp ort for FLEXlike dynamic kernel extensions Psyche uses mo dule

interface information to construct systems comp osed of mo dules implemented by radically dierent op erating

systems and languages socalled multimodel programming It do es not recongure mo dules for p erformance

purp oses Synthesis generates b oth sp ecialized OS interfaces and actual op erating systems functions but

do es not recongure pro cesses or move functions b etween mo dules Finally the VMS op erating system

gave users the ability to extend the system dynamically by installing sp ecially authorized trap handlers VMS

supp orted this capabilityby asso ciating a p erpro cess dispatcher routine with each protected shareable image

that was installed and querying the appropriate dispatchers when an unrecognized trap was encountered

FLEX allows references to added routines to b e installed directly in a single p erpro cess trap table b ecause

all such installations are p erformed by OMOS which will improve the p erformance of this mechanism

Conclusions

FLEX can improve an op erating system in two dierentways i by improving the p erformance of existing

system implementations and ii by dynamically extending op erating system functionality The core of FLEX

is the OMOS server FLEXs p ower comes from OMOSs ability to manipulate mo dules in complex ways

its full access to interface denitions its ability to derive sp ecialized implementations of mo dules and their

interactions and its generalized constraint system for managing address spaces A ma jor source of overhead

in mo dern op erating systems is the time sp ent executing generalpurp ose communications abstractions

FLEX allows systems to b e dynamically constructed using the implementation most appropriate to a given

situation Preliminary results demonstrate the value of evolving an existing op erating system into a exibly

structured system By restructuring existing systems FLEX has b een able to improve the p erformance of

Unix system calls on Mach from In addition FLEX allows programs to extend the functionality

of the op erating system which gives user programs controlled access to privileged data and devices in ways

not anticipated by the original op erating system implementor These results indicate that a dynamic system

restructuring to ol like FLEX can signicantly improve the p erformance of existing systems and at the same

time act as a cornerstone for future system development

References

M Accetta R Baron W BoloskyDGolubRRashidATevanian and M Young Mach A new

kernel foundation for Unix development In Proceedings of the Summer Usenix Conference July

BN Bershad TE Anderson ED Lazowska and HM LevyLightweight remote pro cedure call

ACM Transactions on Computer Systems February

G Bracha The Programming Language Jigsaw Mixins Modularity and Multiple InheritancePhD

thesis University of Utah March

JB Carter A Cox D Johnson and W Zwaenep o el Distributed op erating systems based on a

protected global virtual address space In Third Workshop on Workstation Operating SystemsMay

P Druschel LL Peterson and N Hutchinson Beyond microkernel design Decoupling mo dularity

and protection in Lipto In Proceedings of the th International Conference on Distributed Computing

Systems pages June

B Ford M Hibler and J Lepreau Notes on thread mo dels in MachTechnical Rep ort UUCS

University of Utah Computer Science Department April

B Ford J Lepreau and D Orr Remote pro cedure call sp ecialization with the OMOS ob ject server

Technical rep ort University of Utah June

LJ Kenah RE Goldenb erg and SF Bate VAXVMS Internals and Data Structures Digital Press

Bedford Massachusetts

J Lepreau M Hibler B Ford and J Law Inkernel servers on Mach Implementation and

p erformance In Proceedings of the Third Usenix Mach Symposium pages April

H Massalin and C Pu Threads and inputoutput in the Synthesis kernel In Proceedings of the th

ACM Symposium on Operating Systems Principles pages December

DB Orr J Bonn J Lepreau and R Mecklenburg Fast and exible shared libraries In Proceedings

of the Summer USENIX Conference Cincinnati OH Summer To app ear

DB Orr and R Mecklenburg OMOS an ob ject server for program execution In Proceedings

of the Second International Workshop on Object Orientation in Operating Systems pages

September

DB Orr RW Mecklenburg PJ Ho ogenb o om and J Lepreau Dynamic program monitoring and

transformation using the OMOS ob ject server In Proceedings of the TwentySixth Annual Hawaii

International Conference on System Sciences pages January

ML Scott TJ LeBlanc and BD Marsh Multimo del parallel programming in Psyche In Pro

ceedings of the Conference on the Principles and Practice of Paral lel Programming pages

March