CC-BY 4.0. This is the author’s preprint version of the camera-ready article. A full version of this paper is published in the proceedings of 14th ACM International Conference on Distributed and Event-Based Systems (DEBS 2020). FaaSdom: A Benchmark Suite for Serverless Computing Pascal Maissen Pascal Felber Department of Computer Science Department of Computer Science University of Fribourg, Switzerland University of Neuchâtel, Switzerland [email protected] [email protected] Peter Kropf Valerio Schiavoni Department of Computer Science Department of Computer Science University of Neuchâtel, Switzerland University of Neuchâtel, Switzerland [email protected] [email protected] ABSTRACT Edge Master Serverless computing has become a major trend among cloud User Worker interface function main() { code providers. With serverless computing, developers fully dele- return { payload:'Hello!' code gate the task of managing the servers, dynamically allocating API }; code } the required resources, as well as handling availability and gateway … Dispatcher Event queue Worker Worker fault-tolerance matters to the cloud provider. In doing so, Cloud event … sources code code code code developers can solely focus on the application logic of their software, which is then deployed and completely managed Figure 1: Typical FaaS architecture. The main aspects in the cloud. to benchmark are: performance of a worker (i.e., execu- Despite its increasing popularity, not much is known re- tion speed of a function) and quality of the auto-scaling garding the actual system performance achievable on the mechanism (e.g., allocation of new VMs, how to handle currently available serverless platforms. Specifically, it is flash-crowds [36]). cumbersome to benchmark such systems in a language- or runtime-independent manner. Instead, one must resort to a 1 INTRODUCTION full application deployment, to later take informed decisions on the most convenient solution along several dimensions, The serverless computing paradigm is an emerging approach including performance and economic costs. for developing cloud-based applications [12, 13, 57]. IBM [34] FaaSdom is a modular architecture and proof-of-concept defines it as “an approach to computing that offloads responsi- implementation of a benchmark suite for serverless comput- bility for common infrastructure management tasks (scaling, ing platforms. It currently supports the current mainstream scheduling, patching, provisioning, etc.) to cloud providers serverless cloud providers (i.e., AWS, Azure, Google, IBM), and tools, allowing engineers to focus their time and effort a large set of benchmark tests and a variety of implementa- on the business logic specific to their applications or process”. tion languages. The suite fully automatizes the deployment, Serverless computing requires less expertise than other self- execution and clean-up of such tests, providing insights (in- managed approaches. Users do not manage directly the in- cluding historical) on the performance observed by serverless frastructure and runtime of the system, but instead dele- applications. FaaSdom also integrates a model to estimate gate its operations to the cloud provider. Additionally, cloud budget costs for deployments across the supported providers. providers can deploy finer-grain billing policiese ( .g., on a FaaSdom is open-source and available at https://github.com/ per service-call basis) for any of the offered services [11, 50], bschitter/benchmark-suite-serverless-computing. generally leading to reduced costs for developers. One can distinguish between various serverless paradigms: (1) FaaS (function as a service, and focus of our work) imple- mented for instance by AWS Lambda [4], (2) DBaaS (database as a service), as available through Microsoft Azure for Post- greSQL [42]; and (3) STaaS (storage as a service), via Google KEYWORDS Cloud [26]. FaaS can be considered an hybrid between Plat- serverless, function as a service, benchmark suite, open- form as a Service (PaaS) and the Software as a Service (SaaS) source service model: data and infrastructure are fully managed by CC-BY 4.0. This is the author’s preprint version of the camera-ready article. A full version of this paper is published in the proceedings of 14th ACM International Conference on Distributed and Event-Based Systems (DEBS 2020). the provider, while the application is handled by the user. Fig- 2 BACKGROUND ure 1 illustrates a typical FaaS infrastructure. In the typical This section provides technical details about the four ma- FaaS approach, developers bypass the setup, maintenance jor mainstream serverless providers, namely Amazon Web and management of a compute node (i.e., bare metal, vir- Services (§2.1), Microsoft Azure (§2.2), Google Cloud (§2.3) tual machines, or even containers). Instead, users provide and IBM Cloud (§2.4). We compare the performance of all of the application code for specific functions to be deployed to them in our evaluation (§6). the cloud. Specific eventse ( .g., HTTP requests, storage or database conditions) trigger their execution, typically imple- 2.1 Amazon Web Services Lambda menting data processing [4, 19]. The provider then handles AWS Lambda [4] was released in November in 2014 [8]. AWS the load, as well as availability and scaling requirements. Lambda spans 18 geographical regions, plus China [10]. At Despite the convenience of the approach, it is currently hard the time of writing, it supports six different runtime systems to decide on a specific FaaS provider based on criteria such and seven different programming languages [9]. Depending as performance, workload adaptability or costs. on the region where the function is deployed, Lambda sup- This paper introduces FaaSdom, a testing suite that tack- ports up to 3,000 instances to serve the user functions [6]. les this problem. In a nutshell, application developers can The memory allocated to a function instance can vary from use the FaaSdom suite to automatically deploy several per- 128 MB up to 3,008 MB in steps of 64 MB [5]. The CPU power formance tests a set of FaaS providers. The results can then increases linearly with its memory allocation. For instance, be easily compared along several dimensions. Based on this according to the documentation, at 1,792 MB the function information, deployers can evaluate the “FaaS-domness” of will get 1 vCPU [5]. the providers and decide which one is best adapted for their As observed in [58], Lambda executes functions using applications, in terms of performance, reliability or cost. two different CPUs, namely Intel Xeon E5-2666 clocked at While few efforts exist to benchmark serverless comput- 2.90 GHz and Intel Xeon E5-2680, clocked at 2.80 GHz. ing (often FaaS oriented) [14–16, 27, 35, 38–41, 51, 53], they are relatively limited in terms of supported providers, com- 2.2 Microsoft Azure Functions parison metrics, diversity of benchmarks (e.g., user-defined functions, elasticity), or operating modes (e.g., stress-test vs. Microsoft Azure Functions [43] was released publicly in No- continuous monitoring). Similarly, studies on benchmarking vember 2016 [37]. It supports five runtime systems and seven cloud platforms lack an in-depth evaluation of serverless different programming languages [49]. In contrast with the computing platforms [17]. In this paper, we introduce FaaS- other three cloud providers, Azure offers three different host- dom, a modular and extensible benchmark suite for evaluat- ing plans [46]: Azure Functions offers billing plans that adapt ing serverless computing. FaaSdom natively supports major to the load and popularity of the deployed function (“Con- FaaS providers (AWS, Azure, Google, IBM) but can be easily sumption” plan), plans with finer-grain control over the com- extended to benchmark additional platforms. It includes a puting instance size and pre-warming support (“Premium” wide range of workloads, including user-defined functions, plan), and a billing plan customized on a given application and implementation languages. FaaSdom uses several met- needs (“App Service” plan). This work only considers the con- rics to compare FaaS platforms against multiple dimensions, sumption plan (generation 2.0 [45]), as it is the only one to in terms of latency, throughput and operating costs. It pro- be fully managed by the cloud provider and the most similar vides a Web-based interface that allows users to perform in terms of features to the plans from alternative providers. benchmarks in a fully automatic way (including deployment, Azure Functions can use as many as 200 instances and up execution and clean-up of the tests), and keeps track of his- to 1.5 GB memory [46]. The service can run either on Win- torical data. In that way, FaaSdom can be used both for dows or Linux hosts, and is offered in 28 out of 46 publicly one-shot benchmarks and for continuous monitoring over accessible regions [48]. Note that the consumption plan is time of the providers. FaaSdom is the first system to support only available in 11 regions for both Linux and Windows, such in-depth, comprehensive and extensible benchmarking hence we restrict our deployment to those in our experiments. of serverless computing. Computing nodes can be characterized by their Azure Com- The remainder of this paper is organised as follows. We pute Unit (ACU), with 100 ACU roughly mapped to 1 vCPU. According to our investigations, we believe Azure Functions first introduce background concepts (§2) and the supported 1 frameworks (§3). We then describe the FaaSdom architecture to be executed by virtual machines of type Av2. These VMs (§4) and its different benchmarks (§5). We present and discuss use three different CPUs: Intel Xeon 8171M at 2.1 GHz, Intel evaluation results (§6), before concluding with a summary Xeon E5-2673 v4 at 2.3 GHz and Intel Xeon E5-2673 v3 at of lessons learned (§7) and open perspectives(§8). 2.4 GHz [47]. 1https://docs.microsoft.com/de-ch/azure/virtual-machines/av2-series CC-BY 4.0. This is the author’s preprint version of the camera-ready article.