Bachelor Degree Project Evaluation of GraalVM Performance for Java Programs Author: Robin Larsson Supervisor: Sabri Pllana Semester: VT 2020 Subject: Computer Science Abstract GraalVM is a new Java Development Kit developed by Oracle that promises improved performance when running Java programs. In this paper we compare the performance of both the community edition and the enterprise edition of GraalVM to OpenJDK and OracleJDK, using both Java 8 and Java 11. We use the DaCapo bench- mark suite to test these JDKs. The results are analysed using statistically rigorous methodology. We find that the performance of the different JDKs vary significantly depending on the benchmark, making it difficult to draw any definitive conclusions. In the end we can see that GraalVM Enterprise Edition using Java 8 will outperform the other JDKs tested in the majority of cases. Keywords: Java, GraalVM, performance, JDK, JVM Preface I would like to extend my thanks to my supervisor Sabri Pllana for all his advice and support during this project. Contents List of Figures List of Tables 1 Introduction1 1.1 Background.................................1 1.2 Related work................................1 1.3 Problem formulation............................2 1.4 Motivation..................................2 1.5 Objectives..................................2 1.6 Scope....................................3 1.7 Target group.................................4 1.8 About the JVM...............................4 1.8.1 Garbage collection.........................4 1.8.2 The Java HotSpot virtual machine.................4 1.8.3 HotSpot and GraalVM.......................4 1.9 Outline...................................5 2 Method6 2.1 Performance comparison of system implementations using benchmarks.6 2.1.1 Calculations............................6 2.2 The DaCapo benchmark suite........................7 2.3 Benchmark execution............................8 2.4 Hardware..................................8 2.5 Artifact versions...............................8 2.6 Ethical considerations and data management................9 3 Experimental Evaluation 10 3.1 JDK benchmarks.............................. 10 3.2 JDK comparisons.............................. 25 4 Conclusions and Future Work 28 4.1 Conclusions................................. 28 4.2 Future work................................. 28 References 29 List of Figures 3.1 avrora normalized execution time...................... 11 3.2 avrora JDK comparison heatmap...................... 11 3.3 fop normalized execution time....................... 12 3.4 fop JDK comparison heatmap....................... 13 3.5 h2 normalized execution time........................ 14 3.6 h2 JDK comparison heatmap........................ 14 3.7 jython normalized execution time...................... 15 3.8 jython JDK comparison heatmap...................... 16 3.9 luindex normalized execution time..................... 17 3.10 luindex JDK comparison heatmap..................... 17 3.11 lusearch-fix normalized execution time................... 18 3.12 lusearch-fix JDK comparison heatmap................... 19 3.13 pmd normalized execution time....................... 20 3.14 pmd JDK comparison heatmap....................... 20 3.15 sunflow normalized execution time..................... 21 3.16 sunflow JDK comparison heatmap..................... 22 3.17 tradebeans normalized execution time................... 23 3.18 tradebeans JDK comparison heatmap.................... 23 3.19 xalan normalized execution time...................... 24 3.20 xalan JDK comparison heatmap...................... 25 3.21 Total JDK comparison heatmap for all studied benchmarks........ 26 List of Tables 1.1 Objectives..................................3 2.2 Commonly used z∗-values [21].......................7 2.3 DaCapo benchmarks used [22].......................8 2.4 Artifact versions...............................9 3.5 Comparisons of select JDKs........................ 27 Glossary This section aims to explain some terms used throughout this report. • JVM: A Java Virtual Machine allows a computer to read Java bytecode, effectively translating the bytecode into native code that the computer can understand. • Java bytecode: Java code first compiles to Java bytecode. No matter what com- puter is used to do the compiling the resulting bytecode will be the same. This allows any Java code to be run on any machine as long as it supports a JVM. • JVM language: A JVM language is any language that can run on a JVM, for example Java, Kotlin and Scala. • JIT compiler: A Just-In-Time compiler will compile code to native code during runtime, as opposed to before the program is ran. • JDK: Java Development Kit, an implementation of the Java specification. It is used to develop Java applications and it usually comes bundled with a runtime environ- ment. • Benchmark: A program used to measure the performance of a runtime environ- ment. 1 Introduction GraalVM [1] is a project by Oracle which aims to produce a new polyglot JVM (Java Virtual Machine) and a new JIT (just-in-time) compiler for JVM languages. It promises performance increases when running Java code, as well as allowing the use of many differ- ent languages in the same program, including Java, JavaScript, and Python. Furthermore, it allows any code that runs on GraalVM to be compiled directly to native machine code, lowering startup time and memory usage [2]. This project aims at comparing the performance of Java programs ran using GraalVM to programs ran using OpenJDK [3] or OracleJDK [4]. Note: What is referred to as OracleJDK in this report is officially known as Java SE (Java Platform, Standard Edition). We call it OracleJDK to make it clear that this is Oracles commercial JDK. 1.1 Background Java has historically been criticized as a slow language, especially when compared to languages that compiles directly to native code, like C/C++ [5]. When Java was first re- leased it was a purely interpreted language. Later editions included the HotSpot JVM, which complies often-used methods to native code at runtime, providing significant per- formance increases [6]. The HotSpot JVM includes two different JIT-compilers, the client compiler (or C1) and the server compiler (or C2). C1 provides fast compiling but low op- timization, while C2 provides good optimization but slow compilation. Originally users had to choose which compiler to use, but since Java 7 a technology called tiered compi- lation has been included which enables HotSpot to make use of both compilers simulta- neously [7]. Thanks to the improvements to the JVM and the JIT compilers, Java today is a much faster language than it was at release, being equal to or even beating C/C++ in some benchmarks [8]. GraalVM is a new JDK developed by Oracle. Major features include improved per- formance, the ability to pre-compile Java into native code, and the ability to run non-JVM languages inside the JVM. The first production ready release was made in 2019 with the release of GraalVM 19.0 [9]. 1.2 Related work This report makes heavy use of the DaCapo benchmark suite, which was first released in 2006 along with a scientific article [10]. DaCapo was created in an effort to provide high quality benchmarks for Java. Criticism was directed at other Java benchmarks available at the time, which failed to take into account the complex runtime environment of Java programs. The efforts of the DaCapo team resulted in a series of benchmarks as well as a bench- mark harness which is used to run the benchmarks and report the results. These bench- marks are based on a number of real applications, carefully selected to cover a broad range of use-cases. Attention was given to creating benchmarks with complex code and object lifetimes, so as to better be able to measure the Java environment made complex due to the JVM. A. Georges et al analyzed prevalent methodologies of other Java performance evalua- tion studies in a research paper published in 2007 [11]. The difficult of correctly analyzing Java benchmark results is brought up and a more statistically rigorous methodology is pre- 1 sented. This report makes use of the methodology advocated by A. Georges et al when producing and analyzing the results. 1.3 Problem formulation The goal of this project is to make a performance analysis of GraalVM in the context of running Java code in order to see if it can provide better performance than other popular JDKs. The performance analysis will be based on a number of different benchmark pro- grams, designed to imitate different types of software. The GraalVM JDK is available in two editions, the open source community edition and the commercial enterprise edition. Both of these editions are available based on either Java 8 or Java 11. All of these differ- ent editions will be tested and compared to the reference Java implementation provided by the OpenJDK [3] project as well as Oracles commercial JDKs [4]. 1.4 Motivation As of 2019 Java is a very popular language [12][13], and moreover it is often used for server applications [14]. The cost of providing an internet service is directly linked to how efficient the software providing the service is; a hypothetical software A that can serve twice as many users as a hypothetical software B on the same kind of hardware requires half the number of physical server machines to provide the same service, resulting in much lower operations costs. Improved performance of an application can translate directly to decreased expenses for companies providing internet services. It could also end up