Design and Analysis of a Scala Benchmark Suite for the Java Virtual Machine Entwurf und Analyse einer Scala Benchmark Suite für die Java Virtual Machine Zur Erlangung des akademischen Grades Doktor-Ingenieur (Dr.-Ing.) genehmigte Dissertation von Diplom-Mathematiker Andreas Sewe aus Twistringen, Deutschland April 2013 — Darmstadt — D 17 Fachbereich Informatik Fachgebiet Softwaretechnik Design and Analysis of a Scala Benchmark Suite for the Java Virtual Machine Entwurf und Analyse einer Scala Benchmark Suite für die Java Virtual Machine Genehmigte Dissertation von Diplom-Mathematiker Andreas Sewe aus Twistrin- gen, Deutschland 1. Gutachten: Prof. Dr.-Ing. Ermira Mezini 2. Gutachten: Prof. Richard E. Jones Tag der Einreichung: 17. August 2012 Tag der Prüfung: 29. Oktober 2012 Darmstadt — D 17 For Bettina Academic Résumé November 2007 – October 2012 Doctoral studies at the chair of Prof. Dr.-Ing. Er- mira Mezini, Fachgebiet Softwaretechnik, Fachbereich Informatik, Techni- sche Universität Darmstadt October 2001 – October 2007 Studies in mathematics with a special focus on com- puter science (Mathematik mit Schwerpunkt Informatik) at Technische Uni- versität Darmstadt, finishing with a degree of Diplom-Mathematiker (Dipl.- Math.) iii Acknowledgements First and foremost, I would like to thank Mira Mezini, my thesis supervisor, for pro- viding me with the opportunity and freedom to pursue my research, as condensed into the thesis you now hold in your hands. Her experience and her insights did much to improve my research as did her invaluable ability to ask the right questions at the right time. I would also like to thank Richard Jones for taking the time to act as secondary reviewer of this thesis. Both their efforts are greatly appreciated. Time and again, I am astonished by the number of collaborators and co-authors whom I have worked with during these past five years: Mehmet Ak¸sit,Sami Al- souri, Danilo Ansaloni, Remko Bijker, Walter Binder, Christoph Bockisch, Eric Bodden, Anis Charfi, Michael Eichberg, Samuel Z. Guyer, Kardelen Hatun, Mo- hamed Jmaiel, Jannik Jochem, Slim Kallel, Stefan Katzenbeisser, Lukáš Marek, Mira Mezini, Ralf Mitschke, Philippe Moret, Hela Oueslati, Nathan Ricci, Aibek Sarimbekov, Martin Schoeberl, Jan Sinschek, Éric Tanter, Petr Tuma,˚ Zhengwei Qi, Alex Villazón, Dingwen Yuan, Martin Zandberg, and Yudi Zheng. Others whom I have worked—albeit not written a paper—with are several talented and enthusiastic students: Erik Brangs, Pascal Flach, Felix Kerger, and Alexander Nickol; their work influenced and improved mine in various ways. Still others have helped me realize the vision of a Scala benchmark suite for the Java Virtual Machine by graciously providing help with the suite’s various pro- grams, the programs’ input data, or both: Joa Ebert, Patrik Nordwall, Daniel Ram- age, Bill Venners, Tim Vieira, Eugene Yokota, and the late Sam Roweis. I am also deeply grateful to the DaCapo group for providing such an excellent foundation on which to build my Scala benchmark suite. Of all the aforementioned, there are few to which I want to express my thanks in particular: Christoph Bockisch and everyone at the Dynamic Analysis group of the University of Lugano. Together with Michael Haupt, my colleague Christoph exposed me to the wonderful world of virtual machines in general and Jikes RVM in particular. He also supported me not only during my Diplom thesis and my first year at the Software Technology group, but also after he left us for the Netherlands, to assume a position as assistant professor at the Universeit Twente. Aibek, Danilo, and Walter from the University of Lugano helped my tremendously in developing and refining many of the tools needed for the numerous experiments conducted as part of this thesis. Moreover, they made me feel very welcome on my visit to what may be the most beautiful part of Switzerland. v Here at the Software Technology group, further thanks go to my fellow PhD can- didates and post-docs for inspiring discussions, over lunch and beyond: Christoph Bockisch, Eric Bodden, Marcel Bruch, Tom Dinkelaker, Michael Eichberg, Vaidas Gasiunas, Ben Hermann, Sven Kloppenburg, Roman Knöll, Johannes Lerch, Ralf Mitschke, Martin Monperrus, Sebastian Proksch, Guido Salvaneschi, Lucas Sa- tabin, Thorsten Schäfer, Jan Sinschk, and Jurgen van Ham. The quality of my quantitative evaluations in particular benefited from dozens of discussions Ihad with Marcel. I would also like to thank Gudrun Harris for her unfailing support in dealing with all things administrative and for just making the Software Technology Group a very pleasant place to work at. And for her excellent baking, of course. My thanks also go to both the participants of and the reviewers for the Work- on-Progress session at the 2010 International Conference on the Principles and Practice of Programming in Java, held in Vienna; their suggestions and encourage- ment helped to turn a mere position paper into the thesis you now hold in your hands. My parents Renate and Kai-Udo Sewe provided valuable support throughout all my life. Finally, I am indebted to Bettina Birkmeier for her encouragement and patience—not to mention countless hours of proof-reading. Funding Parts of my work have been funded by AOSD-Europe, the European Network of Excellence on Aspect-Oriented Software Development.1 Other parts of the work have been funded by CASED, the Center for Advanced Security Research Darm- stadt,2 through LOEWE, the “Landes-Offensive zur Entwicklung Wissenschaftlich- ökonomischer Exzellenz.” 1 See http://www.aosd-europe.org/. 2 See http://www.cased.de/. vi Acknowledgements Abstract In the last decade, virtual machines (VMs) for high-level languages have become pervasive, as they promise both portability and high performance. However, these virtual machines were often designed to support just a single language well. The design of the Java Virtual Machine (JVM), for example, is heavily influenced by the Java programming language. Despite its current bias towards Java, in recent years the JVM in particular has been targeted by numerous new languages: Scala, Groovy, Clojure, and others. This trend has not been reflected in JVM research, though; all major benchmark suites for the JVM are still firmly focused on the Java language rather than onthe language ecosystem as a whole. This state of affairs threatens to perpetuate the bias towards Java, as JVM implementers strive to “make the common case fast.” But what is common for Java may be uncommon for other, popular languages. One of these other languages is Scala, a language with both object-oriented and functional features, whose popularity has grown tremendously since its first public appearance in 2003. What characteristics Scala programs have or have not in common with Java programs has been an open question, though. One contribution of this thesis is therefore the design of a Scala benchmark suite that is on par with modern, widely- accepted Java benchmark suites. Another contribution is the subsequent analysis of this suite and an in-depth, VM-independent comparison with the DaCapo 9.12 benchmark suite, the premier suite used in JVM research. The analysis shows that Scala programs exhibit not only a distinctive instruction mix but also object demographics close to those of the Scala language’s functional ancestors. This thesis furthermore shows that these differences can have a marked effect on the performance of Scala programs on modern high-performance JVMs. While JVMs exhibit remarkably similar performance on Java programs, the performance of Scala programs varies considerably, with the fastest JVM being more than three times faster than the slowest. vii Zusammenfassung Aufgrund ihres Versprechens von Portabilität und Geschwindigkeit haben sich vir- tuelle Maschinen (VMs) für höhere Programmiersprachen in der letzten Dekade auf breiter Front durchgesetzt. Häufig ist ihr Design jedoch nur darauf ausgelegt, eine einzige Sprache gut zu unterstützen. So wurde das Design der Java Virtual Machine (JVM) zum Beispiel stark durch das Design der Programmiersprache Java beeinflusst. Trotz ihrer aktuellen Ausrichtung auf Java hat sich insbesondere die JVM als Plattform für eine Vielzahl von neuer Programmiersprachen etabliert, darunter Sca- la, Groovy und Clojure. Dieser Entwicklung wurde in der Forschung zu JVMs bisher jedoch wenig Rechnung getragen; alle großen Benchmark Suites für die JVM sind immer noch stark auf Java als Sprache anstatt auf die Plattform als Ganzes fo- kussiert. Dieser Zustand droht, die systematische Bevorzugung von Java auf lange Zeit festzuschreiben, da die JVM-Entwickler ihre virtuellen Maschinen für die häu- figsten Anwendungsfälle optimieren. Was aber häufig für Java ist, muss keinesfalls häufig für andere populäre Sprachen sein. Eine dieser Sprachen ist Scala, welche sowohl funktionale als auch objekt-orientierte Konzepte unterstützt und seit ihrer Veröffentlichung im Jahre 2003 stetig in der Entwicklergunst gestiegen ist. Welche Charakteristika Scala-Programme mit Java-Programmen gemein haben ist allerdings eine weitgehend ungeklärte Frage. Ein Beitrag dieser Dissertation ist daher das Erstellen einer Benchmark Suite für die Programmiersprache Scala, die mit modernen, etablierten Benchmark Suites für Java konkurrieren kann. Ein weiterer Beitrag ist eine umfassende Analyse der in der Suite enthaltenen Bench- marks und ein VM-unabhängiger Vergleich mit den Benchmarks der DaCapo 9.12 Benchmark Suite, die bisher bevorzugt in der Forschung zu JVMs eingesetzt wird. Diese Analyse