DEGREE PROJECT IN COMPUTER SCIENCE AND ENGINEERING, SECOND CYCLE, 30 CREDITS STOCKHOLM, SWEDEN 2020 A comparison of compiler strategies for serverless functions written in Kotlin KIM BJÖRK KTH ROYAL INSTITUTE OF TECHNOLOGY SCHOOL OF ELECTRICAL ENGINEERING AND COMPUTER SCIENCE A comparison of compiler strategies for serverless functions written in Kotlin - En j¨amf¨orelse av kompilatorstrategier f¨or serverless-funktioner skrivna i Kotlin Kim Bj¨ork - [email protected] Royal Intitute of Technology Stockholm, Sweden Supervisor: Cyrille Artho - [email protected] Examiner: Pontus Johnson - [email protected] January 2020 Abstract Hosting options for software have become more modifiable with time, from re- quiring on-premises hardware to now being able to tailor a flexible hosting solu- tion in a public cloud. One of the latest hosting solution option is the serverless architecture, entailing running software only when invoked. Public cloud providers such as Amazon, Google and IBM provide serverless solutions, yet none of them provide an official support for the popular language Kotlin. This may be one of the reasons why the performance of Kotlin in a serverless environment is, to our knowledge, relatively undocumented. This the- sis investigates the performance of serverless functions written in Kotlin when run with different compiler strategies, with the purpose of contributing knowl- edge within this subject. One Just-In-Time compiler, the Hotspot Java Virtual Machine (JVM), is set against an Ahead-Of-Time compiler, GraalVM. A benchmark suite was constructed and two serverless functions were created for each benchmark, one run with the JVM and one run as a native image, created by GraalVM. The benchmark tests are divided in two categories. One consisting of cold starts, an occurrence that arises the first time a serverless function is invoked or has not been invoked for a longer period of time, causing the need for certain start-up actions. The other category is warm starts, a run when the function has recently been invoked and the cold starts start-up actions are not needed. The result showed a faster total runtimes and less memory requirements for GraalVM-enabled functions during cold starts. During warm starts the GraalVM-enabled functions still required less memory but the JVM functions showed large improvements over time, making the total runtimes more similar to their GraalVM-enabled counterparts. Sammanfattning M¨ojligheterna att hysa (engelska: host) mjukvara har blivit fler och mer modi- fierbara, fr˚anatt beh¨ova ¨aga all h˚ardvara sj¨alv till att man nu kan skr¨addarsy en flexibel l¨osning i molnet. Serverless ¨ar en av de senaste l¨osningarna. Olika leverant¨orer av publika molntj¨anster s˚asom Amazon, Google och IBM tillhandah˚allerserverless-l¨osningar. Dock har ingen av dessa leverant¨orer ett officiellt st¨od f¨or det popul¨ara programmeringsspr˚aket Kotlin. Detta kan vara en av anledningarna till att spr˚akets prestanda i en serverless-milj¨o ¨ar, s˚avitt vi vet, relativt ok¨and. Denna rapport har som syfte att bidra med kunskap inom just detta omr˚ade. Tv˚aolika kompilatorstrategier kommer att j¨amf¨oras, en JIT (Just-In-Time) -kompilator och en AOT (Ahead-Of-Time) -kompilator. Den JIT-kompilator som anv¨ands ¨ar Hotspot Java Virtual Machine (JVM). Den AOT-kompilator som anv¨ands ¨ar GraalVM. F¨or detta arbete har en benchmark svit skapats och f¨or varje test i denna svit skapades tv˚aserverless-funktioner. En som kompileras med JVM och en som k¨ors som en f¨ardig bin¨ar skapad av GraalVM. Testerna har delats upp i tv˚akategorier. En d¨ar alla tester genomg˚attkallstarter, n˚agotsom sker d˚adet ¨ar f¨orsta g˚angenfunktionen kallas eller d˚adet har g˚atten l¨angre tid sedan funktionen kallades senast. Den andra kategorien ¨ar d˚atestet inte beh¨over g˚a igenom en kallstart, d˚ahar funktionen blivit kallad nyligen. K¨orningen kan d˚a undvika att genomg˚avissa steg som kr¨avs vid en kallstart. Resultatet visade att de tester som genomf¨ordes inom kategorin kallstarter visade p˚aatt k¨ortiden var snabbare och att minnesanv¨andningen var mindre f¨or de funktioner som kompilerats av GraalVM. I den andra kategorin, d˚atesterna inte beh¨ovde genomg˚aen kallstart, kr¨avde GraalVM-funktionerna fortfarande mindre minne men JVM-funktionerna visade p˚aen stor f¨orb¨attring n¨ar det kom till exekveringstid. De totala k¨ortiderna av de tv˚aolika kompilatorstrategierna var d˚amer lika. Contents 1 Introduction 3 1.1 Problem and Research Question . .4 1.2 Contributions and Scope . .5 1.3 Ethics and Sustainability . .5 1.4 Outline . .6 2 Background 7 2.1 Serverless . .7 2.1.1 The Attributes of Serverless . .7 2.1.2 Use Cases for Serverless Functions . .9 2.2 Kotlin . 11 2.3 Types of Compilers . 11 2.3.1 Ahead-of-Time Compiler (AOT) . 12 2.3.2 Just-In-Time Compiler (JIT) . 12 2.4 The JVM Compiler . 12 2.5 The GraalVM Compilation Infrastructure . 14 2.6 Performing Benchmark Tests . 15 2.7 Related Work . 17 2.7.1 Solutions Silimar to Serverless . 17 2.7.2 GraalVM at Twitter . 18 2.7.3 Benchmark Environment and the Cloud . 18 2.8 Summary . 19 3 Method 20 3.1 Metrics . 20 3.1.1 Latency . 21 3.1.2 Response time . 22 3.1.3 Memory consumption . 22 3.1.4 Execution time . 22 3.2 Benchmarks . 22 3.2.1 Real benchmarks . 23 3.2.2 Complementary benchmarks . 24 3.3 Environment and Setup . 25 3.4 Sampling Strategy and Calculations . 26 1 3.5 Summary . 27 4 Result 28 4.1 Static metrics . 28 4.2 Latency . 29 4.3 Application Runtime . 32 4.4 Response Time . 33 4.5 Memory Consumption . 34 5 Discussion 36 5.1 Latency . 36 5.1.1 Cold start . 36 5.1.2 Warm start . 37 5.2 Application Runtime . 38 5.2.1 Cold start . 39 5.2.2 Warm start . 40 5.3 Response Time . 46 5.3.1 Cold start . 46 5.3.2 Warm start . 48 5.4 Memory Consumption . 50 5.4.1 Cold start . 51 5.4.2 Warm start . 52 5.5 Threats to validity . 53 6 Conclusion 55 6.1 Performance . 55 6.1.1 Latency . 55 6.1.2 Application Runtime . 56 6.1.3 Response Time . 57 6.1.4 Memory Consumtion . 57 6.2 Future work . 57 2 Chapter 1 Introduction Companies are constantly looking to digitize and are conceiving new use cases they want to explore every day. This is preferably done in an agile and mod- ular way. The key factors to making this possible are a reasonable cost, fast realization time and flexibility. Hosting is an area that has followed this trend. As a response, to this need for a more agile way of working, companies have moved from bare metal on- premises hosting to cloud hosting. In a cloud adaptation survey by IDG done in 2018, 73 % of companies stated that they already had adopted cloud technology and 17 % said they intended to do so within a year [1]. Another survey predicts that 83 % of enterprise workload will be in the cloud [2]. By using cloud computing, companies can allocate just the amount of com- putation power they need to host their solutions. A cloud solution can also easily be scaled up or down when the need changes. Cloud computing also makes it possible for small-scale solutions to be hosted with great flexibility and be economically defensible. The next step in this development toward more agile and modular hosting options could be claimed to be the serverless architecture. A serverless archi- tecture lets customers run code without having to buy, rent or provision servers or virtual machines. In fact a serverless architecture also relieves a client of everything that is connected to servers and more traditional hosting, such as maintenance, monitoring and everything infrastructure related. All that the clients need to concern themselves with is the actual code. These attributes enables a more fine-grained billing method, where clients gets charge solely on the resources used. These resources are the time it takes, as well as the memory needed, to execute the serverless function. The vendors providing the serverless solution, such as Amazon (AWS Lambda [3]) and Google (Google Cloud Func- tions [4]), also provide automatic scaling for their serverless solutions enabling a steady high availability. These are presumably among the top reasons why serverless is rapidly increasing in usage. According to Serverless the usage of serverless functions has almost doubled among their respondents from 45 % in 2017 to 82 % in 2018 [5]. Notable is also that 53.2 % stated serverless technology 3 is critical for their job. During the growth of serverless architecture, cloud providers have added support for more languages. AWS for example have gone from only supporting Node.js to now also supporting Python, Ruby, Java, Go and C# [6]. But one language that is still lacking official support, from any cloud provider that offers a serverless solution, is Kotlin. Kotlin is a programming language developed by JetBrains and was first re- leased in February 2016. Kotlin is mostly run on JVM but can also be compiled into JavaScript or native code (utilizing LLVM) [7]. Despite being a newer lan- guage it has already gained a lot of traction by being adopted by large companies and is currently used in production by Pinterest [8] and Uber [9] among others. Kotlin is also as of 7th May 2019 Google's preferred language for Android app development [10] and has been among the top in \most loved" languages ac- cording to Stackoverflow developer survey reports the last years [11, 12].