Memory Elasticity Benchmark Liran Funaro Orna Agmon Ben-Yehuda Assaf Schuster Technion—Israel Institute of Technion—Israel Institute of Technion—Israel Institute of Technology Technology Technology Haifa, Israel Haifa, Israel Haifa, Israel [email protected] [email protected] [email protected] ABSTRACT CCS CONCEPTS Cloud computing handles a vast share of the world’s com- • General and reference → Metrics; Evaluation; Exper- puting, but it is not as efficient as it could be due to its lackof imentation; Performance; Measurement. support for memory elasticity. An environment that supports memory elasticity can dynamically change the size of the KEYWORDS application’s memory while it’s running, thereby optimizing Cloud, Benchmark, Vertical Elasticity, RAM, Memory the entire system’s use of memory. However, this means at least some of the applications must be memory-elastic. ACM Reference Format: A memory elastic application can deal with memory size Liran Funaro, Orna Agmon Ben-Yehuda, and Assaf Schuster. 2020. changes enforced on it, making the most out of all of the Memory Elasticity Benchmark. In The 13th ACM International Sys- tems and Storage Conference (SYSTOR ’20), June 2–4, 2020, Haifa, memory it has available at any one time. The performance of Israel. ACM, New York, NY, USA, 12 pages. https://doi.org/10.1145/ an ideal memory-elastic application would not be hindered 3383669.3398277 by frequent memory changes. Instead, it would depend on global values, such as the sum of memory it receives over time. 1 INTRODUCTION Memory elasticity has not been achieved thus far due to a Today’s cloud providers make every effort to improve their circular dependency problem. On the one hand, it is difficult resource utilization and thereby make more money off the to develop computer systems for memory elasticity without same hardware. Rigid allocation prevents them from utilizing proper benchmarking, driven by actual applications. On the the hardware efficiently, so they offer clients various options other, application developers do not have an incentive to for resource elasticity [11]. These elastic options allow clients make their applications memory-elastic, when real-world to change their resource consumption on the fly by exploiting systems do not support this property nor do they incentivize resources that are momentarily unused by other clients. it economically. Resource elasticity is seamless in services such as To overcome this challenge, we propose a system of memory- Application-as-a-Service (AaaS) and serverless computing. elastic benchmarks and an evaluation methodology for an Here, clients rent a black-box execution environment that application’s memory elasticity characteristics. We validate exposes a limited application programming interface (API) this methodology by using it to accurately predict the per- they can use. The environment’s resource consumption and formance of an application, with a maximal deviation of 8% workload distribution are controlled by the provider. Thus, on average. The proposed benchmarks and methodology the client shares resources with other clients who occupy the have the potential to help bootstrap computer systems and same environment. In such services, the provider handles the applications towards memory elasticity. client’s resource elasticity, relieving the client of this burden. However, these environments may not suit all applications. Some clients need a broader API, have a proprietary applica- Permission to make digital or hard copies of all or part of this work for tion, or use an uncommon application that is not supported personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear by the provider. Other clients may have specific performance this notice and the full citation on the first page. Copyrights for components requirements that the provider is unable to guarantee. For of this work owned by others than ACM must be honored. Abstracting with example, clients might need to be physically closer to their credit is permitted. To copy otherwise, or republish, to post on servers or to data or maintain some continuity between runs. redistribute to lists, requires prior specific permission and/or a fee. Request Clients who require more than what is provided by AaaS permissions from [email protected]. and serverless computing will deploy their applications us- SYSTOR ’20, June 2–4, 2020, Haifa, Israel © 2020 Association for Computing Machinery. ing Infrastructure-as-a-Service (IaaS) and Container-as-a- ACM ISBN 978-1-4503-7588-7/20/06...$15.00 Service (CaaS). In such services, clients rent a bundle of rigid, https://doi.org/10.1145/3383669.3398277 exclusive, resources in the form of a single virtual machine SYSTOR ’20, June 2–4, 2020, Haifa, Israel L. Funaro et al. (VM) or an OS container. Many IaaS and CaaS providers offer CloudSigma allowed clients to change their memory alloca- CPU elasticity in the form of burstable performance, which tion and billing during runtime. Unfortunately, other com- offers a basic level of CPU performance but can ’burst’ toa mercial cloud providers did not follow in their footsteps higher level when required. Under certain conditions, this and this option is no longer promoted on the CloudSigma lets clients use more CPU than their initial allocation, in the web-page. same VM/container. These providers include Google [13], We’re seeing a circular dependency problem between Amazon [3], Azure [24], CloudSigma [7], and RackSpace [27]. memory-elastic applications and systems that require and With the current proliferation of CPU elasticity schemes, incentivize such properties. When one of these elements is CPU utilization is adequately optimized and more clients scarce, there is no incentive to develop the other because can be allocated to the same physical servers [11]. This real benefit only comes when both elements exist. leaves memory as the bottleneck resource: it is an expen- A proof that memory-elastic applications exist or can be sive resource that limits machine occupancy. Memory elas- created is essential to break this circular dependency. Our ticity schemes should be a natural extension to CPU elas- first contribution is a set of memory-elastic applications ticity, allowing clients to use more memory in the same with a memory to performance trade-off (Section 4.3). VM/container than their initial memory allocation. Clients Once the circular dependency problem is solved, and elas- who can tolerate a temporary memory shortage could bene- tic memory systems and applications exist commercially, a fit from these schemes by lowering the amount of memory language and method for quantifying application elasticity they rent exclusively. They could then compensate for the re- will help clients choose a resource bundle that best suits their duced requirement by bursting when they really need more application. In addition, quantifying an application’s elastic- memory. For example, clients who can postpone memory- ity will help cloud providers test and optimize their systems. intensive phases in their operations, can reserve a small Our second contribution comprises a methodology and initial amount of memory and make use of more memory terminology for evaluating memory elasticity (Section 3). whenever it is available (e.g., for maintenance operations). These specify how to determine a memory elasticity score This enables clients to time-share memory and the provider for each application that can be used as a memory elasticity can squeeze more applications onto the same hardware [11]. benchmark. To achieve this, clients need memory-elastic applications The methodology is implemented as an open-source mem- that can change their maximal memory usage on the fly ory elasticity evaluation framework (Section 4). We validated and whose performance is proportional to their memory us- the evaluation process using our framework (Section 5) over age. Unfortunately, memory-elastic applications are scarce. a set of memory-elastic applications. The results show that Although developers usually strive to make their applica- our memory elasticity score can accurately predict an appli- tion’s performance proportionate to its CPU and bandwidth cation’s performance, with an average deviation of 8%. availability, most applications are not designed with mem- The methods and applications we introduce, along with ory elasticity in mind. Developers generally address only our memory elasticity metrics, will allow clients to choose the maximal memory footprint of their application. They less expensive memory elastic schemes and reduce costs. treat it as constant or a value dictated by the current applica- Accordingly, the number of IaaS clients per server will no tion workload. The operating system’s swapping mechanism longer be constrained by memory, allowing elastic CPU allo- allows seamless application operation when the available cation schemes to demonstrate their full potential of nearly memory is insufficient, but this results in a graceless perfor- 80% CPU utilization [11]. mance degradation; even a minor memory loss may degrade the performance significantly. Why do developers toil towards making performance scale nicely with the CPU and bandwidth, but neglect doing this 2 MEMORY ELASTICITY METHODS for memory? Developing memory-elastic applications re- This section presents several application properties that can quires more work. With a proliferation of memory-elastic