f f f f f f f f f f f f f f f f f f f f Verified Java Bytecode Verification Gerwin Klein Lehrstuhl f¨urSoftware & Systems Engineering Institut f¨ur Informatik Technische Universit¨atM¨unchen Lehrstuhl f¨urSoftware & Systems Engineering Institut f¨urInformatik Technische Universit¨atM¨unchen Verified Java Bytecode Verification Gerwin Klein Vollst¨andiger Abdruck der von der Fakult¨atf¨urInformatik der Technischen Universit¨at M¨unchen zur Erlangung des akademischen Grades eines Doktors der Naturwissenschaften (Dr. rer. nat.) genehmigten Dissertation. Vorsitzender: Univ.-Prof. Dr. Alois Knoll Pr¨ufer der Dissertation: 1. Univ.-Prof. Tobias Nipkow, Ph.D. 2. Univ.-Prof. David Basin, Ph.D. Eidgen¨ossische Technische Hochschule Z¨urich/Schweiz Die Dissertation wurde am 28. November 2002 bei der Technischen Universit¨atM¨unchen eingereicht und durch die Fakult¨atf¨ur Informatik am 7. Februar 2003 angenommen. Kurzfassung Der Bytecode Verifier ist ein essentieller Bestandteil der Sicherheitsarchiktektur der Pro- grammierplattform Java. Die Dissertation pr¨asentiert eine formale, ausfuhrbare¨ Spezi- fikation des Bytecode Verifiers sowie den Beweis, dass dieser korrekt ist. Die Forma- lisierung im Theorembeweiser Isabelle besteht aus einem abstrakten Framework fur¨ Bytecode-Verifikation, das mit zunehmend ausdrucksstarken Typsystemen instantiiert wird. Diese decken s¨amtliche interessanten Eigenschaften der Java-Plattform ab: Klassen, Objekte, virtuelle Methoden, Vererbung, Ausnahmebehandlung, Konstruktoren, Objekt- Initialisierung, Subroutinen und Felder. Die Formalisierung liefert zwei ausfuhrbare¨ veri- fizierte Bytecode Verifier: den iterativen Standard-Algorithmus sowie einen Lightweight Bytecode Verifier fur¨ Ger¨ate mit eingeschr¨ankten Ressourcen. I II Abstract The bytecode verifier is an important part of Java’s security architecture. This thesis presents a fully formal, executable, and machine checked specification of a representative subset of the Java Virtual Machine and its bytecode verifier together with a proof that the bytecode verifier is safe. The specification consists of an abstract framework for bytecode verification which is instantiated step by step with increasingly expressive type systems covering all of the interesting and complex properties of Java bytecode verification: classes, objects, inheri- tance, virtual methods, exception handling, constructors, object initialization, bytecode subroutines, and arrays. The instantiation yields two executable verified bytecode verifiers: the iterative data flow algorithm of the standard Java platform and also a lightweight bytecode verifier for resource-constrained devices such as smart cards. All specifications and proofs have been carried out in the interactive theorem prover Isa- belle/HOL. Large parts of the proofs are written in the human-readable proof language Isabelle/Isar making it possible to understand and reproduce the reasoning indepen- dently of the theorem prover. All formal proofs in this thesis are machine checked and generated directly from Isabelle sources. III IV Acknowledgements I thank Tobias Nipkow for supervising this thesis. His support and advice have been invaluable. I thank Gilles Barthe and David Basin for acting as referees. I am indebted to Norbert Schirmer, Sebastian Skalberg, and Martin Strecker for reading and commenting on drafts of this thesis. I thank my wife, Elisabeth Meister, not only for proof reading. The remaining errors are mine, and they exist because I have ignored good advice. I also thank the other members of the Isabelle team in Munich, current and former, for an excellent, friendly, and relaxed working atmosphere: Clemens Ballarin, Gertrud Bauer, Stefan Berghofer, David von Oheimb, Farhad Mehta, Leonor Prensa-Nieto, Cornelia Pusch, Markus Wenzel, and Martin Wildmoser. V VI Contents Contents 1 Introduction 1 1.1 Motivation ................................... 1 1.2 Contributions .................................. 3 1.3 The Java Virtual Machine ........................... 4 1.4 The Bytecode Verifier ............................. 5 1.5 Related Work .................................. 7 1.6 Isabelle ..................................... 10 1.7 Overview .................................... 14 2 An Abstract Framework 15 2.1 Introduction ................................... 15 2.2 Semilattices ................................... 16 2.2.1 Partial Orders ............................. 16 2.2.2 Semilattices ............................... 16 2.2.3 The Error Type and Err-semilattices ................. 17 2.2.4 The Option Type ............................ 18 2.2.5 Products ................................ 19 2.2.6 Lists of Fixed Length ......................... 19 2.2.7 Sets ................................... 20 2.3 Stability ..................................... 21 2.3.1 Welltypings ............................... 21 2.3.2 Constraints on the Transfer Function ................. 22 2.3.3 Refining the Transfer Function .................... 24 2.4 Kildall’s Algorithm ............................... 25 2.5 Lightweight Bytecode Verification ....................... 27 2.5.1 Introduction .............................. 27 2.5.2 The Algorithm ............................. 30 2.5.3 Soundness ................................ 32 2.5.4 Completeness .............................. 35 2.5.5 Conclusion ............................... 39 VII Contents 3 Instantiating the Framework 41 3.1 Introduction ................................... 41 3.2 The µJava Virtual Machine .......................... 43 3.2.1 Structure ................................ 43 3.2.2 State Space ............................... 48 3.2.3 Operational Semantics ......................... 49 3.2.4 A Defensive VM ............................ 53 3.3 The Bytecode Verifier ............................. 57 3.3.1 The Semilattice ............................. 57 3.3.2 Applicability and Effect ........................ 59 3.3.3 Welltypings ............................... 63 3.3.4 Executable Bytecode Verifiers ..................... 65 3.4 Type Safety ................................... 66 3.4.1 The Theorem .............................. 67 3.4.2 Conformance .............................. 68 3.4.3 Invariance Proof ............................ 70 3.5 Conclusion ................................... 75 4 Object Initialization 77 4.1 Introduction ................................... 77 4.2 Types ...................................... 80 4.3 Operational Semantics ............................. 81 4.3.1 Structure and State Space of the VM ................ 81 4.3.2 Aggressive Machine .......................... 86 4.3.3 Defensive Machine ........................... 91 4.4 Bytecode Verification .............................. 94 4.4.1 The Semilattice ............................. 94 4.4.2 Applicability and Effect ........................ 95 4.4.3 Executable Bytecode Verifiers ..................... 100 4.5 Type Safety ................................... 102 4.5.1 The Theorem .............................. 103 4.5.2 Conformance .............................. 104 4.6 Conclusion ................................... 109 5 Subroutines 111 5.1 Introduction ................................... 111 5.2 Types and Values ................................ 115 5.3 Operational Semantics ............................. 117 5.4 The Bytecode Verifier ............................. 119 VIII Contents 5.4.1 The Semilattice ............................. 119 5.4.2 Applicability and Effect ........................ 120 5.4.3 Executable Bytecode Verifiers ..................... 126 5.5 Type Safety ................................... 128 5.6 Conclusion ................................... 130 6 Arrays 131 6.1 Introduction ................................... 131 6.2 Types and Wellformed Programs ....................... 134 6.3 Operational Semantics ............................. 136 6.3.1 State Space ............................... 136 6.3.2 Aggressive Machine .......................... 137 6.3.3 Defensive Machine ........................... 140 6.4 Bytecode Verification .............................. 142 6.4.1 The Semilattice ............................. 142 6.4.2 Applicability and Effect ........................ 143 6.4.3 Executable Bytecode Verifiers ..................... 147 6.5 Type Safety ................................... 149 6.6 Conclusion ................................... 151 7 Conclusion 153 7.1 Summary .................................... 153 7.2 Further Work .................................. 155 IX Contents X 1 Introduction 1.1 Motivation The bytecode verifier is an important, integral part of Java’s security architecture. It has become popular to download and execute untrusted web applets and JavaCard applets often even without the user’s approval or intervention. Contrary to the approach of a certain widely used operating platform, Java provides a scheme to execute untrusted code safely: the sandbox. The sandbox is an insulation layer that implements a control policy preventing unsafe access to hardware and operating system resources. It relies on three security measures: • Java is not compiled into machine code, but rather into an intermediate format for a virtual machine [51]. • Hardware access is mediated by a set of API classes [30] that implement a suitable access control policy. • Before execution, the bytecode is statically verified [51] to ensure that it does not bypass the two security measures above. The last of the three components is the bytecode verifier (BV). It is a central component of the architecture: any bug in the BV causing an unsafe applet