ISOLATION, RESOURCE MANAGEMENT AND SHARING IN THE KAFFEOS JAVA RUNTIME SYSTEM by Godmar Back A dissertation submitted to the faculty of The University of Utah in partial fulfillment of the requirements for the degree of Doctor of Philosophy in Computer Science School of Computing The University of Utah May 2002 Copyright °c Godmar Back 2002 All Rights Reserved THE UNIVERSITY OF UTAH GRADUATE SCHOOL SUPERVISORY COMMITTEE APPROVAL of a dissertation submitted by Godmar Back This dissertation has been read by each member of the following supervisory committee and by majority vote has been found to be satisfactory. Chair: Wilson C. Hsieh John B. Carter Elmootazbellah N. Elnozahy Jay Lepreau Gary Lindstrom THE UNIVERSITY OF UTAH GRADUATE SCHOOL FINAL READING APPROVAL To the Graduate Council of the University of Utah: I have read the dissertation of Godmar Back in its final form and have found that (1) its format, citations, and bibliographic style are consistent and acceptable; (2) its illustrative materials including figures, tables, and charts are in place; and (3) the final manuscript is satisfactory to the Supervisory Committee and is ready for submission to The Graduate School. Date Wilson C. Hsieh Chair, Supervisory Committee Approved for the Major Department Thomas C. Henderson Chair/Dean Approved for the Graduate Council David S. Chapman Dean of The Graduate School ABSTRACT Single-language runtime systems, in the form of Java virtual machines, are widely deployed platforms for executing untrusted mobile code. These runtimes provide some of the features that operating systems provide: interapplication mem- ory protection and basic system services. They do not, however, provide the ability to isolate applications from each other. Neither do they provide the ability to limit the resource consumption of applications. Consequently, the performance of current systems degrades severely in the presence of malicious or buggy code that exhibits ill-behaved resource usage. In this dissertation, we show that Java runtime systems can be extended to provide a process model, and that such a process model can provide robust and efficient support for untrusted applications. Processes are an operating system abstraction in which each process executes as if it were run in its own virtual machine. We have designed and prototyped KaffeOS, which is a Java runtime system that provides support for processes. KaffeOS isolates processes and manages the physical resources available to them, in particular: CPU and memory. Each process is given its own heap, which can be separately garbage collected. Unlike existing Java virtual machines, KaffeOS can safely terminate processes without adversely affecting the integrity of the system, and it can fully reclaim a terminated process’s resources. The novel aspects of the KaffeOS architecture include the application of a user/kernel boundary as a structuring principle to runtime systems, the employment of garbage collection techniques for resource management and isolation, and a model for direct sharing of objects between untrusted applications. The difficulty in designing KaffeOS lay in balancing the goals of isolation and resource management against the goal of allowing direct sharing of objects. We built a prototype of our design and ran the SPEC JVM 98 benchmarks to evaluate its performance for well-behaved applications. We found that for those applications, our KaffeOS prototype is no more than 8% slower than the freely available JVM on which it is based, which is an acceptable penalty for the safety that it provides. At the same time, our KaffeOS prototype can support more applications than a hardware-based approach on the same platform. We demonstrate that in the presence of malicious or buggy code that engages in a denial-of-service attack directed against resources, KaffeOS-based systems can contain the attack, remove resources from the attacked applications, and continue to provide robust service to other clients. v CONTENTS ABSTRACT :::::::::::::::::::::::::::::::::::::::::::::::: iv LIST OF TABLES ::::::::::::::::::::::::::::::::::::::::::: viii LIST OF FIGURES :::::::::::::::::::::::::::::::::::::::::: ix ACKNOWLEDGMENTS ::::::::::::::::::::::::::::::::::::: xi CHAPTERS 1. INTRODUCTION ::::::::::::::::::::::::::::::::::::::: 1 1.1 Motivation . 1 1.2 KaffeOS . 7 1.3 Roadmap . 10 2. THE JAVA PROGRAMMING LANGUAGE ::::::::::::::: 11 2.1 Language Overview . 11 2.2 The Java Virtual Machine . 16 3. DESIGN OF KAFFEOS :::::::::::::::::::::::::::::::::: 22 3.1 Protection and Isolation . 22 3.2 Memory Management in KaffeOS . 30 3.3 Interprocess Communication . 39 3.4 Hierarchical Resource Management . 45 3.5 Summary . 50 4. IMPLEMENTATION :::::::::::::::::::::::::::::::::::: 53 4.1 Kernel . 53 4.2 Namespace and Class Management . 56 4.3 Memory Management . 63 4.4 Summary . 70 5. EVALUATION :::::::::::::::::::::::::::::::::::::::::: 73 5.1 Write Barrier Overhead . 74 5.2 Overhead for Thread Stack Scanning . 86 5.3 Denial of Service Scenarios . 89 5.4 Sharing Programming Model . 101 5.5 Summary . 114 6. RELATED WORK ::::::::::::::::::::::::::::::::::::::: 117 6.1 Java-based Systems . 117 6.2 Garbage Collection and Resource Management . 126 6.3 Operating Systems . 130 7. CONCLUSION :::::::::::::::::::::::::::::::::::::::::: 135 7.1 Future Work . 135 7.2 JanosVM Application . 138 7.3 Summary . 139 APPENDICES A. ON SAFE TERMINATION ::::::::::::::::::::::::::::::: 143 B. SERVLET MICROENGINE :::::::::::::::::::::::::::::: 146 REFERENCES :::::::::::::::::::::::::::::::::::::::::::::: 156 vii LIST OF TABLES 4.1 Number of shared runtime classes for SPEC JVM98 benchmarks. 62 5.1 Best-case overhead per write barrier by type. 76 5.2 Number of write barriers executed by SPEC JVM98 benchmarks. 79 5.3 Type of write barriers executed SPEC JVM98 benchmarks. 80 5.4 Measured vs. best-case overhead. 84 5.5 Cycles spent on walking entry items for SPEC JVM98 Benchmarks . 87 5.6 Expected results for CpuHog and Garbagehog scenarios. 98 5.7 Affected methods of java.util.Vector if instance is located on a shared heap. 104 5.8 Affected methods of shared.util.Hashmap if instance is located on a shared heap. 108 5.9 Affected methods of shared.util.LinkedList if instance is located on a shared heap. 110 LIST OF FIGURES 1.1 Single JVM model. 5 1.2 Multiple JVM model. 6 1.3 KaffeOS model. 7 2.1 Java compilation and execution. 17 2.2 Example of Java bytecode. 18 3.1 User/kernel boundary in KaffeOS. 28 3.2 Valid references for user and kernel heaps. 33 3.3 Use of entry and exit items. 35 3.4 Terminating a process. 37 3.5 Valid cross-references for shared heaps. 42 3.6 Hierarchical memory management in KaffeOS. 47 3.7 Hierarchical CPU management in KaffeOS. 50 4.1 Protecting kernel-internal locks. 55 4.2 Class loader delegation in KaffeOS. 58 4.3 Transforming library code. 61 4.4 Internal and external fragmentation in KaffeOS. 66 4.5 Worst-case scenario for remote thread stacks in KaffeOS. 68 4.6 Pseudocode for write barrier. 70 4.7 Pseudocode for mark phase. 71 5.1 KaffeOS performance in SPECjvm98 benchmarks. 81 5.2 Time spent in GC for SPECjvm98 benchmarks. 83 5.3 Time spent scanning remote thread stacks. 88 5.4 Configurations for MemHog. 90 5.5 Throughput for MemHog servlet. 93 5.6 MemHog stress test. 95 5.7 Throughput for CpuHog scenario. 99 5.8 Throughput for GarbageHog scenario. 101 5.9 Use of java.util.Vector on a shared heap. 105 5.10 Implementation of shared.util.Hashmap. 107 5.11 Use of shared.util.HashMap on a shared heap. 109 5.12 Implementation of shared.util.LinkedList. 111 5.13 Use of shared.util.LinkedList on a shared heap. 112 A.1 Packet dispatcher example. 143 A.2 Naive approach to guaranteeing queue consistency using cleanup han- dlers. 144 A.3 Packet dispatch using deferred termination. 145 x ACKNOWLEDGMENTS First and foremost, I need to thank my advisor, Wilson Hsieh, for his immea- surable help and support not only in performing the research described in this dissertation but also in finding the motivation and strength to complete it. I would also like to thank my committee members for their comments on drafts of this dissertation and their help on papers we published earlier. Jay Lepreau deserves credit for giving me the room and support to finish this dissertation. Tim Wilkinson built the Kaffe system that I used as a foundation for the KaffeOS system described in this work. Patrick Tullmann contributed to many discussions about Java operating systems; he also proofread several chapters of this dissertation. Jason Baker contributed to KaffeOS’s implementation; he also proofread Chapter 4. Tim Stack helped the implementation by updating it with a more recent version of Kaffe. I thank Eric Eide for his help with the much-hated LATEX type setting system. Thanks go to the other members of the Flux research group from whom I have learned a great deal during my time as a graduate student in Utah and to the members of the operating systems research community who helped me by engaging in discussions at meetings and conferences and reviewing earlier papers. Finally, last but not least, I would like to thank my fiance´eAnnette for her support and patience. CHAPTER 1 INTRODUCTION The need to support the safe execution of untrusted programs in runtime sys- tems for type-safe languages has become clear. Language runtimes are being used in many environments to execute untrusted code that may violate a system’s safety or security. Current runtime systems implement memory protection in software through the enforcement of type safety [9]. In addition, these systems provide secure system services through a number of mechanisms, including language-based access modifiers and a security model for granting access privileges to system services. They do not, however, sufficiently isolate untrusted code and are unable to control the computational resources used by such code. This dissertation presents KaffeOS, a design for a Java virtual machine (JVM) that allows for the robust execution of untrusted code. Like a hardware-based operating system that supports and manages multiple processes running on one physical machine, KaffeOS provides resource control and isolation to its processes within one virtual.