Relative Factors in Performance Analysis of Java Virtual Machines Dayong Gu Clark Verbrugge Etienne M. Gagnon School of Computer Science D´epartement d’informatique McGill University Universit´eduQu´ebeca ` Montr´eal Montr´eal, Qu´ebec, Canada Montr´eal, Qu´ebec, Canada {dgu1, clump}@cs.mcgill.ca [email protected] Abstract perturbing performance measurements. This resulting lack of guid- Many new Java runtime optimizations report relatively small, ance often means that optimization researchers do not always give single-digit performance improvements. On modern virtual and a sufficiently wide or deep consideration to potential confounding actual hardware, however, the performance impact of an optimiza- factors from the underlying systems. tion can be influenced by a variety of factors in the underlying In this paper we focus on two main concerns. First, an expo- systems. Using a case study of a new garbage collection optimiza- sition of various external, and benchmark specific factors that can tion in two different Java virtual machines, we show the relative influence a VM or source-level optimization, and second a quanti- effects of issues that must be taken into consideration when claim- tative evaluation and comparison of the different influences. Using ing an improvement. We examine the specific and overall perfor- an in-depth case study of a simple garbage collection optimization mance changes due to our optimization and show how unintended we are able to show that a combination of instruction cache changes side-effects can contribute to, and distort the final assessment. Our due to trivial code modifications, and subtle, consequent data lay- experience shows that VM and hardware concerns can generate out and usage differences can induce almost a 10% whole program variances of up to 9.5% in whole program execution time. Consid- performance variation. We are able to show significant variations eration of these confounding effects is critical to a good, objective in both interpreter and JIT environments. As the largest contribu- understanding of Java performance and optimization. tors to variance are not unique to our case study, other optimiza- tions achieving single-digit performance improvements (or degra- Categories and Subject Descriptors C.4 [Performance of Sys- dations) may thus be affected by the same issues. tems]: Measurement Techniques, Design Studies Previous studies on the complexity of measuring performance in modern VMs have argued for the importance of a holistic view General Terms Experimentation, Measurement, Performance of program performance [19], or have pointed out some of the fac- Keywords performance analysis, Java, garbage collection, hard- tors that can directly affect and distort the measurement of specific ware counters, caches optimizations, such as garbage collection [4]. In this paper we ex- tend these concerns showing the wide range of issues that must be addressed to ensure a well-informed interpretation of performance 1. Introduction change due to optimization, and the surprisingly large potential im- Compiler and runtime optimizations are typically deemed success- pact of low-level concerns on high level performance. ful if a measurable, reasonably stable performance improvement Specific contributions of our work include: can be shown over a selection of benchmarks, even if the effect is • We give experimental results from both an interpreter and a JIT relatively small or not uniformly positive. In the case of Java virtual environment showing that side-effects of lower level execution machine (VM) or software level optimizations, low-level hardware can account for almost 10% of measured performance. and VM effects are often presumed amortized through the com- • plexity of interaction, or by considering average case behaviour. We provide guidance on which factors are most critical by giving Many optimizations, however, result in small, single-digit per- a quantitative and comparative analysis of a variety of potential formance changes, and in these cases uncontrolled factors aris- confounding factors in assessing JVM performance. ing from lower levels of execution may account for a significant • Using both VM and low-level hardware counter information, we amount of the performance difference, greatly distorting the per- provide an analysis and unique characterization of the SPECjvm98 ceived effect of an optimization. Unfortunately, not only is the [28] and the DaCapo [23] benchmarks with respect to their in- breadth of potential influences not always obvious, but it is also struction and data cache sensitivity. not always clear which factors are most important with respect to • Our optimization case study is also a new garbage collection optimization that can improve GC performance in both JIT (Jikes RVM) and interpreter (SableVM) environments. Permission to make digital or hard copies of all or part of this work for personal or The remainder of this paper is organized as follows. In Section classroom use is granted without fee provided that copies are not made or distributed 2, we discuss related work on GC and Java performance measure- for profit or commercial advantage and that copies bear this notice and the full citation ment and analysis. Section 3 then describes our garbage collection on the first page. To copy otherwise, to republish, to post on servers or to redistribute optimization, and Section 4 gives initial data on its performance. In to lists, requires prior specific permission and/or a fee. Section 5, we refine our investigation and give more detail on the VEE’06 June 14–16, 2006, Ottawa, Ontario, Canada. Copyright c 2006 ACM 1-59593-332-6/06/0006. $5.00. 111 impact of various performance influences. Section 6 provides di- Our optimization case study is based on a simple and general rections for future work in this area and presents our conclusions. improvement to tracing garbage collectors. Tracing GCs are found in most Java virtual machines. 2. Related work Starting from a set of root references (static variables, stack ref- erences), a tracing GC visits each reachable object seeking refer- Measuring the performance and understanding the behaviour of ences to other reachable objects. Once the live set is determined Java programs is challenging. Many factors, from program charac- the memory storage of non-reachable objects is reclaimed. Gagnon teristics, VM techniques and implementations, and OS strategies, and Hendren proposed a bi-directional object layout [14] aiming to hardware platform performance, can affect the final measured to improve the performance of GC tracing, and here we present a performance. B. Dufour et al. provides a set of concise and pre- reference section tracing strategy that attempts to validate and im- cisely defined dynamic metrics for Java programs [10]. At a lower prove that work. level, L. Eeckhout et al. [11] analyze the interaction between Java virtual machine and microarchitectural platform factors by using 3.1 Bi-directional layout and reference section scanning principal component analysis to reduce the data dimensionality. M. Hauswirth et al. suggest an approach named vertical profiling [19] Bi-directional layout is an alternative way of physically represent- to understand how Java programs interact with the underlying ab- ing objects in memory. Traditionally, all the fields of an object are straction levels, from application, virtual machine, and OS, to hard- located after the object header. The left graph in Figure 1 shows the ware. To understand the behaviour of a particular program execu- traditional layout of an object of type C extending type B extend- tion, they first obtain profiling data from different levels, then visu- ing type A. The right graph in Figure 1 shows the bi-directional alize the data and discover the correlations between the anomalous layout of the same object. The basic idea of bi-directional layout performance and the profiling data visually or using statistical met- is to relocate reference fields before the object header and group rics. Our examination here is in the same spirit of a multi-levelled them together in a contiguous section; we denote these sections as view of performance, though focusing on the variance due to opti- reference sections. The main advantage of the bi-directional layout mizations rather than for program development. is the simplicity of locating all references in an object during GC. Our investigations are motivated through a case study of a References are contiguous, and only a single count of reference garbage collection (GC) optimization. GC has been a target of section size must be stored (usually in the object header). There is optimization for decades, and from a variety of directions. Ungar’s no need to access a table of offsets in the object’s type information generational scavenging [32] technique and more recent works on block to identify references, as must be done with the traditional Age-based GC [30] Older-first GC [29] and Beltway GC [5], for layout. instance, all aim to improve performance by adjusting collection time according to object lifetimes. Alternatively, live objects can be aggregated into regions in the heap based on a selection of object attributes. This either aims to improve data locality in the program [20, 17], or to reduce the memory access overhead of the collector [24]. Optimizations on data prefetching and lazy sweeping [8, 6] aim to improve data cache performance. Our approach tries to reduce the GC workload, although the implementation design is also helpful in reducing data cache misses. Some other works specifically study GC performance. S. Black- burn et al. [4] discuss performance myths of canonical GC algo- rithms on widely used Java benchmarks. They compare the perfor- mance of classic GC and memory allocation algorithms in differ- ent configurations and environments. The impact of special imple- mentation factors, such as “write barriers” and the size of nursery space of generational collectors, on mutator and GC performance are carefully studied. In this paper we extend their results to a fur- ther range of factors and influences, particularly unintended cache effects.