SeBS: A Serverless Benchmark Suite for Function-as-a-Service Computing Marcin Copik Grzegorz Kwaśniewski Maciej Besta [email protected] ETH Zürich ETH Zürich ETH Zürich Switzerland Switzerland Switzerland Michał Podstawski Torsten Hoefler ComeOn Group ETH Zürich Poland Switzerland Abstract User’s perspective Function-as-a-Service (FaaS) is one of the most promising - Pay-as-you-go billing ŏ High computing cost directions for the future of cloud services, and serverless - Massive parallelism ŏ Variable performance - Simplified deployment ŏ Vendor lock-in functions have immediately become a new middleware for - Architecture agnostic ŏ Black-box platform building scalable and cost-efficient microservices and appli- Provider’s perspective cations. However, the quickly moving technology hinders - Higher machine utilization ŏ Handling heterogeneity reproducibility, and the lack of a standardized benchmark- - Fine-grained scheduling ŏ Micro-architecture effects ing suite leads to ad-hoc solutions and microbenchmarks Table 1. Intuitive summary of the FaaS model. Quantitative measure- being used in serverless research, further complicating meta- ments are needed for assessing advantages and disadvantages. analysis and comparison of research solutions. To address this challenge, we propose the Serverless Benchmark Suite: the first benchmark for FaaS computing that systematically covers a wide spectrum of cloud resources and applications. The flexible FaaS model may be seen as a necessary con- Our benchmark consists of the specification of represen- nection between ever-increasing demands of diverse work- tative workloads, the accompanying implementation and loads on the one hand, and huge data centers with special- evaluation infrastructure, and the evaluation methodology ized hardware on the other. Serverless functions have al- that facilitates reproducibility and enables interpretability. ready become the software glue for building stateful applica- We demonstrate that the abstract model of a FaaS execution tions [24, 89, 97]. It is already adopted by most major commer- environment ensures the applicability of our benchmark cial providers, such as AWS Lambda [1], Azure Functions [3], to multiple commercial providers such as AWS, Azure, and Google Cloud Functions [5], and IBM Cloud Functions [4], Google Cloud. Our work facilities experimental evaluation of marking the future of cloud computing. From a user perspec- serverless systems, and delivers a standardized, reliable and tive, it promises more savings and the pay-as-you-go model evolving evaluation methodology of performance, efficiency, where only active function invocations are billed, whereas a scalability and reliability of middleware FaaS platforms. standard IaaS virtual machine rental incurs costs even when they are idle [94]. From a provider’s perspective, the fine- grained execution model enables high machine utilization through efficient scheduling and oversubscription. Table1 arXiv:2012.14132v2 [cs.DC] 2 Jul 2021 1 Introduction provides an overview of FaaS advantages and issues. Clouds changed the computing landscape with the promise While serverless computing gained significant traction of plentiful resources, economy of scale for everyone, and both in industry and academia, many authors raised several on-demand availability without up-front or long-term com- important issues, for example, vendor lock-in on commer- mitment. Reported costs are up to 7× lower than that of cial platforms, lack of standardized tools for development a traditional in-house server [22]. The deployment of mid- and debugging, unpredictable overheads due to high-latency dleware in cloud evolved from a more hardware-oriented cold starts [75], and surprisingly high costs of computation- Infrastructure as a Service (IaaS) to a more software-oriented intensive codes [60]. Cost and performance analyses of FaaS Platform as a Service, where the cloud service provider takes applications are further inhibited by the black-box nature of the responsibility of deploying and scaling resources [37]. serverless platforms, and existing analyses rarely generalize Function-as-a-Service (FaaS) is a recent development towards beyond a given vendor or a tested system. Yet, efficient and fine-grained computing and billing, where stateless functions scalable software systems cannot be designed without in- are used to build modular applications without managing sights into the middleware they’re being built upon. Hence, infrastructure and incurring costs for unused services. there is an urgent need for a benchmarking design that would Copik, et al. SeBS: The Serverless 1 (External) 2 Execution environment (4) Warm invoca�on Benchmark Suite (Internal) triggers start E sources (1) Execu�on Timer Function 1 Function 2 ... starts at the ... Cloud Payload frontend 1 EE IoT (3) Data (4) Cold E transfer start Func�on ID E E Cloud resources and services Web apps Func�on ... -end scheduler HTTP Ephemeral Front- Persistent Storage Storage ... Invocation E (2) Execu�on moves 3 4 User Pattern to the scheduler Figure 1. An abstract model of a FaaS platform. Labels: ¶ - triggers, · - execution environment, ¸ - persistent storage, ¹ - ephemeral storage, E - invocation system. Details in Section2. (1) specify clear comparison baselines for the evaluation of 2 Platform Model many FaaS workloads on different platforms, (2) enable deriv- We first build a benchmarking model of a FaaS platform that ing general performance, cost, and reliability insights about provides an abstraction of key components, see Figure 1. This these evaluations, and (3) facilitate the above with public and enables generalizing design details that might vary between easy to use implementation. providers, or that may simply be unknown due to the black- To address these challenges, we introduce the Server- box nature of a closed-source platform. less Benchmark Suite (SeBS), a FaaS benchmark suite that ¶ Triggers. The function lifetime begins with a trig- combines a systematic cover of a wide spectrum of cloud ger. Cloud providers offer many triggers to express various resources with detailed insights into black-box serverless ways to incorporate functions into a larger application. One platforms. SeBS comes with a (1) benchmark specification example is an HTTP trigger, where every request sent to based on extensive literature review, (2) a general FaaS plat- a function-specific address invokes the function using the form model for wide applicability, (3) a set of metrics for passed payload. Such triggers are often used for interactive effective analysis of cost and performance, (4) performance services or background requests from Internet-of-Things (IoT) models for generalizing evaluation insights across different and edge computing devices. Other triggers are invoked pe- cloud infrastructures, and (5) an implementation kit that fa- riodically, similarly to cron jobs, or upon events such as a cilitates evaluating existing and future FaaS platforms. new file upload or a new message in a queue. Finally, func- We evaluate SeBS on AWS, Microsoft Azure, and Google tions can be triggered as a part of larger FaaS workflows in Cloud Platform. Overall, with our benchmarks represent- dedicated services such as AWS Step and Azure Durable. ing a wide variety of real-world workloads, we provide the · Execution environment. Functions require a sand- necessary milestone for serverless functions to become an box environment to ensure isolation between tenants. One efficient and reliable software platform for complex and scal- option is containers, but they may incur overheads of up able cloud applications. to 20× over native execution [85]. Another solution is light- To summarize, we make the following contributions: weight virtual machines (microVMs). They can provide over- heads and bootup times competitive with containers, while • We propose SeBS, a standardized platform for continuous improving isolation and security [6, 73]. evaluation, analysis, and comparison of FaaS performance, ¸ Persistent Storage. The cloud service offers scalable reliability, and cost-effectiveness. storage and high bandwidth retrieval. Storage offers usu- • We offer novel metrics and experiments that, among others, ally consist of multiple containers known as buckets (AWS, enable quantifying the overheads and efficiency of FaaS Google) and containers (Azure). Storage services offer high under various configurations and workloads. throughput but also high latency for a low price, with fees • We provide a full benchmark implementation and an open- in the range of a few cents for 1 GB of data storage, retrieval, source1 software toolkit that can automatically build, deploy, or 10,000 write/read operations. and invoke functions on FaaS systems in AWS, Azure, and ¹ Ephemeral Storage. The ephemeral storage service GCP, three popular cloud providers. The toolkit is modular addresses the high latency of persistent storage [66, 67]. Ex- and can be easily extended to support new FaaS platforms. ample use cases include storing payload passed between • We provide insights into FaaS performance and consistency consecutive function invocations [20] and communication (Sec.6, Table9). We analyze performance and cost over- in serverless distributed computing [61]. The solutions in- heads of serverless functions, and model cold start patterns clude scalable, in-memory databases offered by the cloud and invocation latencies of FaaS platforms. provider and a custom solution with a VM instance holding an in-memory,