The following paper was originally published in the Proceedings of the 7th USENIX Security Symposium San Antonio, Texas, January 26-29, 1998 Operating System Protection for Fine-Grained Programs Trent Jaeger, Jochen Liedtke, and Nayeem Islam IBM T.J. Watson Research Center For more information about USENIX Association contact: 1. Phone: 510 528-8649 2. FAX: 510 548-5738 3. Email: [email protected] 4. WWW URL:http://www.usenix.org/ Op erating System Protection for Fine-Grained Programs Trent Jaeger Jo chen Liedtke Nayeem Islam IBM T. J. Watson Research Center Hawthorne, NY 10532 Emails:fjaegertjjo [email protected] m.comg Abstract numb er of protection domain crossings must b e han- dled securely i.e., correctly with resp ect to the secu- We present an op erating system-level security mo del rity requirements to prevent attacks and eciently for controlling ne-grained programs, suchasdown- to minimize p erformance degradation. In this pap er, loaded executable content, and compare this security we show that a security mo del implemented on a fast mo del's implementation to that of language-based and exible IPC mechanism can enforce security re- security mo dels. Language-based security has well- quirements that language-based systems cannot with known limitations, such as the lack of complete me- little p erformance impact. diation e.g., for compiled programs or race condi- Several op erating systems use hardware-based tion attacks and faulty self-protection e ective secu- protection to prevent pro cesses from inadvertently rity is unproven. Op erating system-level mo dels are and/or maliciously mo difying one another. Each capable of complete mediation and self-protection, pro cess has an address space that de nes a set of but some researchers argue that op erating system- memory segments and the pro cess's access rights to level security mo dels are unlikely to supplant such those segments. A pro cess can only access memory language-based mo dels b ecause they lack p ortability in its own address space. In addition, the op erating and p erformance. In this pap er, we detail an op er- system has a security mo del that asso ciates pro cesses ating system-level security mo del built on the Lava with their access rights to system resources. Using Nucleus, a minimal, fast -kernel op erating system. address spaces of suitable granularity and pro cess ac- We showhow it can enforce security requirements cess rights to controlled resources, an op erating sys- for ne-grained programs and show that its p erfor- tem can control a pro cess's op erations as desired. mance overhead with the additional security can However, op erating system security mo dels have b e virtually negligible when compared to language- b een deemed to lack the p erformance and exibility based mo dels. Given the sucient p erformance and necessary to control ne-grained programs. While security, the p ortability issue should b ecome mo ot some systems have b een built that eciently con- b ecause other vendors will have to meet the higher trol pro cesses in dynamically-de ned protection do- security and p erformance exp ectations of their cus- mains [3,8, 14], these systems have b een applied tomers. only to more traditional applications e.g., PostScript interpreters. In an application comp osed of ne- grained programs, programs with di erent protection 1 Intro duction domains interact often p erhaps as much as on each metho d invo cation. In a recent pap er, Wallach et We demonstrate how op erating system protection can al. [25] discard address space-based protection from b e used to control ne-grained programs exibly and consideration for applications with ne-grained pro- eciently. Op erating systems use hardware-based grams by noting that IPC b etween two COM ob jects protection to isolate pro cesses from one another. on Windows NT takes 1000 times longer than a pro- However, the way that current op erating systems cedure call 230 sto0.2s. We claim that this implement this protection has caused researchers to discrepancy can b e virtually eliminated while gaining deem them to o slow and in exible for controlling ne- security and maintaining exibility. grained programs. Fine-grained programs have dif- ferent protection domains and mayinteract often in Wehave develop ed a prototyp e implementation of the course of a computation. The e ect of a large a exible security mo del for controlling downloaded content. This mo del is implemented on the Lava Nu- individual content. The securitykernel must b e able cleus. The Lava Nucleus provides address spaces, to mediate all controlled op erations. To enforce least threads, fast IPC, exible paging, and IPC inter- privilege on content p ermissions may b e based on ap- ception that enable ecient and exible control of plication state and evolve as application state evolves. pro cesses. In this pap er, we primarily concentrate on The securitykernel must b e able to control this evolu- the e ectiveness of the Lavanucleus for implement- tion within reasonable limits. Lastly, the kernel must ing a exible security mo del and its resultant p er- b e able to protect itself from mo di cations that may formance. We show that fast IPC and IPC intercep- result in tamp ering with its b ehavior. tion enable the implementationof dynamical ly autho- A question that has re-app eared recently is whether rized IPC that can b e p erformed in as little as 9.5 s. language-based or op erating system-based protection We b elieve further ro om for optimziation is p ossible, is b etter suited for e ectively and eciently control- given an ideal estimate for dynamically authorized ling such ne-grained programs. Or otherwise stated: IPC is ab out 4 s. Also, exible page mapping in the to what extent can op erating system protection e- Lava Nucleus enables ob jects of size greater than the ciently provide e ective security and to what extent hardware page size to b e shared among pro cesses, so can language protection e ectively provide ecient coarse-grained sharing of memory b etween pro cesses security? As describ ed b elow, op erating system pro- is p ossible. tection has several advantages over language protec- The structure of this pap er is as follows. In Sec- tion from a security p ersp ective, but the cost of do- tion 2, we compare language-based and op erating main crossings make it questionable whether ecient system-based security mo dels. In Section 3, we de- op erating system protection for ne-grained pro cesses scrib e an op erating system security mo del for down- is p ossible. On the other hand, language protection loaded executable content. In Section 4, we describ e can b e implemented eciently, but some key security the implementation of this mo del on the Lava Nu- safeguards are weakened such that e ective security cleus. In Section 5, we examine the p erformance of may b e lost. the prototyp e implementation and compare its p er- Traditionally, op erating systems have enforced sys- formance to language-based mo dels for ne-grained tem security requirements b ecause hardware-based programs of the sort discussed byWallach et al. In protection provides signi cant advantages in the key Section 6, we conclude and present future work. areas of economy of mechanism, fail-safe defaults, and complete mediation [23]. The op erating system's TCB can protect pro cesses by restricting them to 2 Language vs. O/S Protection their own address spaces which can b e enforced by a simple mechanism at least compared to a com- The basic problem is to implement a securitykernel piler. Since only the program requested is placed in that e ectively and eciently enforces the security the address space, other, indep endent programs are requirements of downloaded executable content. The not a ected by its failure assuming the op erating basic requirements of any security infrastructure are 1 system adequately protects itself from such failures. that it can adapted from [2]: Also, since the op erating system can intercept any in- terpro cess communication IPC b etween pro cesses, assign p ermissions dynamically to individual controlled op erations by a program including com- content, piled ones can b e completely mediated by the op er- mediate all controlled op erations using such p er- ating system. missions i.e., an op eration that a pro cess is not Language-based protection has gained favor in re- unconditionally trusted to invoke, centyears, however. We attribute this p opularity to three factors: 1 improvements in the develop- manage the evolution of p ermissions as content ment of \safe" languages; 2 the p erception that is executed, programs will b ecome increasingly ne-grained, and protect itself from tamp ering ne-grained domains are prohibitively exp ensiveto enforce; and 3 lack of exibility in op erating system For a system that uses dynamically downloaded ex- security mo dels. \Typ e-safe" languages are strongly- ecutable content, p ermissions must b e assignable to typ ed i.e., all data is typ ed and casting is restricted 1 There is an additional requirement that any \securityker- or prohibited and do not p ermit direct addressing nel" b e simple enough that indep endentevaluators can assess of the system's memory i.e., no p ointers. There- whether it will op erate prop erly. While we will not prove such fore, all data is accessed according to its interface, so a feature ab out our system here we do attempt to keep the security mo del as simple as is feasible.