The Design and Implementation of a Java Virtual Machine on a Cluster of Workstations by Carlos Daniel Cavanna A thesis submitted in conformity with the requirements for the degree of Master of Applied Science Edward S. Rogers Sr. Department of Electrical and Computer Engineering University of Toronto 2003 c Copyright by Carlos Daniel Cavanna 2003 ABSTRACT The Design and Implementation of a Java Virtual Machine on a Cluster of Workstations by Carlos Daniel Cavanna Master of Applied Science Edward S. Rogers Sr. Department of Electrical and Computer Engineering University of Toronto 2003 We present the design, implementation, and evaluation of a Java Virtual Machine (JVM) on a cluster of workstations, which supports shared memory in software. More specifically, we first extend the Jupiter JVM infrastructure to support multithreading on Symmetric Multiprocessors (SMPs). We then extend this multithreaded Jupiter to deal with memory consistency, private memory allocation, and the limitations on the use of some resources on a 16-processor cluster of PCs interconnected by a Myrinet network; which supports Shared Virtual Memory and a pthreads interface. Our experimental evaluation is performed using standard benchmarks. On a 4-processor SMP, it indicates that the performance of the multithreaded version of Jupiter is midway between a na¨ıve and a state-of-the-art implementations of a Java interpreter. Our evaluation on the cluster demonstrates that good speedup is obtained. iii For Mariela, my beloved wife; and my dear parents. v ACKNOWLEDGEMENTS First, I would like to thank my supervisor, Professor Tarek Abdelrahman, for his constant guidance and encouragement during the development of this work. His high standards, attention to detail and confidence placed this work among the most interesting and challenging experiences I have ever had. I would also like to thank Patrick Doyle, for the long and productive discussions about Jupiter, his support during the initial stages of this project and for providing such a good system to work with. And Peter Jamieson, for the long hours we spent solving development issues in the SVM system when enabling Jupiter to run on the cluster, and debugging the newly released Linux version of CableS. I must extend my appreciation to Professor Angelos Bilas for providing the resources to make this project advance, and Rosalia Christodoulopoulou and Reza Azimi for continuously improving VMMC. I am also thankful to all my friends, who were always present when I needed them. In particular to Diego Novillo, who constantly provided accurate suggestions throughout this work, and to Professors Gerardo Castillo and Andrea Lanaro, who I have always considered very good friends, for motivating me so strongly to follow this path. Last, but not least, I want to express my deepest gratitude to my wife for her understanding, patience and love; and my family, without whose support, endless love, and lifelong encouragement I would not have accomplished this achievement. vii TABLE OF CONTENTS DEDICATION ........................................ v ACKNOWLEDGEMENTS ................................ vii LIST OF TABLES ..................................... xiii LIST OF FIGURES .................................... xv CHAPTERS 1. Introduction .................................... 1 1.1 Objectives and Contribution . ....................... 2 1.2ThesisOrganization............................. 4 2. Background and Related Work ........................ 5 2.1TheJavaModel............................... 5 2.2TheJupiterJVM.............................. 6 2.3MemoryConsistency............................ 6 2.4Clusters.................................... 9 2.5SharedVirtualMemoryClusters...................... 9 2.5.1 The Myrinet Cluster, VMMC, GeNIMA and CableS . 10 2.5.2 ProgrammingusingCableS................... 12 2.6RelatedWork................................ 14 2.6.1 JavaParty............................. 14 2.6.2 JESSICA/JESSICA2....................... 15 2.6.3 Java/DSM............................. 16 2.6.4 cJVM............................... 17 2.6.5 Kaffemik.............................. 17 2.6.6 Hyperion.............................. 18 ix 3. Multithreading Extensions ........................... 19 3.1MultithreadinginJVMs.......................... 19 3.1.1 ThreadHandling......................... 19 3.1.2 Synchronization.......................... 21 3.1.3 Wait-SetsandNotification.................... 21 3.2MultithreadinginJupiter.......................... 22 3.2.1 ThreadingModules........................ 25 3.2.2 ThinThread ............................ 27 3.2.3 Thread and Monitor Modules.................. 29 3.2.4 Monitor Support for Recursive Calls . ........... 31 3.2.5 Monitor and Spurious Wake-ups . ............... 31 3.2.6 ThreadJoining.......................... 34 3.2.7 WaitingonNon-DaemonThreads................ 34 3.3InitializationRoutines............................ 36 3.4MutableDataandSynchronization.................... 37 3.5MultithreadedQuickOpcodes....................... 39 3.5.1 Quick Opcodes Overview . ................... 39 3.5.2 MultithreadedQuickOpcodeReplacement........... 42 4. Shared Virtual Memory Extensions ..................... 49 4.1DesignIssues................................. 49 4.2MemoryConsistency............................ 50 4.2.1 LockingonJavaClassCreation................. 51 4.2.2 Locking on Java Object Access . ............... 52 4.2.3 LockingonJavaObjectCreation................ 53 4.2.4 VolatileFields........................... 53 4.2.5 ArgumentPassingatSystemThreadCreation......... 54 4.2.6 MemoryConsistencyatJavaThreadCreation......... 57 4.2.7 MemoryConsistencyatJavaThreadTermination....... 60 4.3PrivateMemoryAllocation......................... 60 4.3.1 Private and Shared Memory Sources . ........... 61 4.3.2 ThreadStack........................... 63 4.3.3 JNI(JavaNativeInterface)................... 64 4.3.4 ThreadHandling......................... 65 4.3.5 OpcodeInterpretation...................... 66 4.3.6 ClassParsing........................... 66 4.4 Limitation of Resources ........................... 66 4.5BarrierImplementation........................... 68 4.6GarbageCollection............................. 69 4.7TheJavaMemoryModel.......................... 70 x 5. Experimental Evaluation ............................ 71 5.1TheBenchmarks............................... 71 5.2TheExperimentPlatforms......................... 74 5.2.1 MeasurementMethodology................... 75 5.3SMPEvaluation............................... 76 5.3.1 Single-Threaded Jupiter vs. Multithreaded Jupiter . 77 5.3.2 Multithreaded Jupiter vs. Reference JVMs ........... 78 5.3.3 Speedup.............................. 79 5.4ClusterEvaluation.............................. 81 5.4.1 EvaluationRemarks....................... 83 5.4.2 OneProcessorperNode..................... 84 5.4.3 TwoProcessorsperNode.................... 85 5.4.4 PerformanceImpactoftheSVMSystem............ 86 6. Conclusions ..................................... 99 6.1SummaryandConclusions......................... 99 6.2 Future Work . ............................... 101 APPENDICES A. Benchmark Details ................................ 107 A.1SPECjvm98................................. 107 A.2JavaGrande................................. 108 B. Stack Design .................................... 113 B.1 Java Stack Overview . ........................... 113 B.2Variable-SizeStackSegment........................ 114 B.3ExperimentalEvaluation.......................... 116 BIBLIOGRAPHY ......................................119 xi LIST OF TABLES Table 5.1 SPECjvm98BenchmarkApplications....................... 72 5.2 JavaGrandeBenchmarkApplications...................... 72 5.3 SMPExperimentPlatform............................. 74 5.4 ClusterExperimentPlatform........................... 75 5.5 JVMs........................................ 77 5.6 Execution Times for the Multithreaded Jupiter with Respect to the Single- ThreadedJupiterUsingtheSPECjvm98Benchmarks.............. 77 5.7 Execution Times for the Multithreaded Jupiter with Respect to the Reference JVMsUsingtheSPECjvm98Benchmarks.................... 79 5.8 Execution Times for the Multithreaded Jupiter with Respect to the Reference JVMsUsingtheJavaGrandeBenchmarks(SizeA)............... 80 5.9 Stress Tests for the Multithreaded Jupiter with Respect to the Reference JVMs UsingtheJavaGrandeBenchmarks........................ 81 5.10 Added Execution Times for the Multithreaded Jupiter and the Reference JVMsUsingtheJavaGrandeBenchmarks(SizeA)............... 82 5.11 Average Speedup for the Multithreaded Jupiter and the Reference JVMs Using theJavaGrandeBenchmarks(SizeA)...................... 82 5.12 Execution and Benchmark Times for the Cluster-Enabled Jupiter on One Pro- cessorperNodeUsingtheJavaGrandeBenchmarks(SizeA)......... 87 5.13 Execution and Benchmark Times for the Cluster-Enabled Jupiter on One Pro- cessorperNodeUsingtheJavaGrandeBenchmarks(SizeB)......... 88 xiii 5.14 Average Speedup for the Cluster-Enabled Jupiter on One Processor per Node UsingtheJavaGrandeBenchmarks(SizeA).................. 89 5.15 Average Speedup for the Cluster-Enabled Jupiter on One Processor per Node UsingtheJavaGrandeBenchmarks(SizeB)................... 89 5.16 Execution and Benchmark Times for the Cluster-Enabled Jupiter on Two Pro- cessorsperNodeUsingtheJavaGrandeBenchmarks(SizeA)......... 92 5.17 Execution and Benchmark Times for the Cluster-Enabled Jupiter on Two Pro- cessorsperNodeUsingtheJavaGrandeBenchmarks(SizeB)......... 93 5.18 Average Speedup