Proceedings of The 6th IEEE International Symposium on Service Oriented System Engineering (SOSE 2011) Cloud Testing Tools Xiaoying Bai∗ y, Muyang Li∗, Bin Chen∗, Wei-Tek Tsai∗ z, Jerry Gao∗ x ∗Department of Computer Science and Technology, TNList, Tsinghua University, Beijing, China y SKLSDE, Beijing University of Aeronautics and Astronautics, Beijing, China zSchool of Computing, Informatics, and Decision Systems Engineering, Arizona State University, USA xDepartment of Computer Science and Technology, San Jose State University, USA Email: [email protected], [email protected], [email protected], [email protected], [email protected] Abstract—Cloud platform provides an infrastructure for re- running 99.95% of the year, i.e., allows for 4.4 hours of source sharing, software hosting and service delivering in a downtime per year only. Unfortunately, an unexpected crash pay-per-use approach. To test the cloud-based software systems, happened in April, 2011 due to operation mistakes during techniques and tools are necessary to address unique quality concerns of the cloud infrastructure such as massive scalability network reconfiguration [5]. More than 70 organizations are and dynamic configuration. The tools can also be built on the affected including FourSquare, the New York Times, and cloud platform to benefit from virtualized platform and services, Reddit, who pay to use AWS to run their websites on EC2. Due massive resources, and parallelized execution. The paper makes to the accident, the performance of these websites is greatly a survey of representative approaches and typical tools for cloud decreased, and some sites were even down for dozens of testing. It identifies the needs for cloud testing tools including multi-layer testing, SLA-based testing, large scale simulation, hours. Applications hosted on remote clouds may have lower and on-demand test environment. To address the needs, it controllability and observability, compared with conventional investigates the new architecture and techniques for designing in-house hosted applications. testing tools for the cloud and in the cloud. Tool implementations In addition, the mechanisms of massive resource sharing are surveyed considering different approaches including migrated and usage-based allocation introduce uncertainties. From user- conventional tools, research tools, commercial tools and facilities like benchmark and testbed. Based on the analysis of state-of- s’ perspective, it needs to ensure that system will provide the-art practices, the paper further investigates future trend of functionalities and performance with reasonable cost. From the testing tool research and development from both capability and providers’ perspective, it aims to enhance resource utilization usability perspectives. by utilizing unused capacities based on guaranteed system Keywords-Cloud Computing, Cloud Testing, Testing Tool performance. A SLA between the two parties usually defines the QoS properties, including not only quality properties I. INTRODUCTION during normal operations, but also fault-tolerance and recovery Cloud computing proposes a new architecture of multi- capabilities in reaction to failures. However, as the cloud layered resource sharing, such as infrastructure-as-a-service (I- infrastructure also changes its status (e.g., number of virtual aaS), data-as-as-service (DaaS), platform-as-a-service (PaaS), machine instances, number of applications, and usage loads) and software-as-a-service (SaaS) [1]. Resources can be dynam- continuously, making testing a difficult task because the un- ically allocated based on usage demands following negotiated derlying infrastructure is also changing and thus performance Service-Level-Agreement (SLA) in a pay-per-use business will be different even the same application is run. model to achieve cost-effective performance and resource Cloud testing issues have been addressed earlier [6]. The utilization. Cloud-based infrastructure has a significant impact paper addresses tool support for cloud testing, specifically on software technologies. It shifts the focus of software de- support for multi-layer testing, SLA-based testing, simulation- velopment from product-oriented activities to service-oriented based testing, embedding testing with execution, and on- reuse, composition, and online renting. It has a potential demand testing. Various tools are available [7], [8], [9], [10], to enhance the scalability, portability, reusability, flexibility, [11], [12] to test cloud-based systems at various layers includ- and fault-tolerance capabilities of software systems, taking ing hardware interface, platform interface, storage system, and advantages of the new cloud infrastructure [2], [3]. application systems. One can make the following observation: However, cloud-based hosting introduces risks to the de- 1) Testing faces the challenges of cost and scalability due to pendability of systems. Software is remotely deployed in a vir- increases in software size and complexity. However test- tualized runtime environment using shared hardware/software ing tools can also take advantages of from the enormous resources, and hosted in a third-party infrastructure. The resources from a cloud infrastructure, resources such quality and performance of the software highly depend on its as virtualized platforms and services, large computation runtime environment that are usually out of the users’ control. power and memory, parallel operations, and automated For example, Amazon provides a cloud infrastructure and web- recovery mechanisms that comes in a PaaS. For exam- hosting system AWS (Amazon Web Services)[4], including ples, conventional testing tools, such as JUnit test frame- services like EC2 (Elastic Compute Cloud) and S3 (Simple work and symbolic execution tools, can be migrated to Storage Service). It promises to keep customers’ sites up and a cloud platform. They can be refactored using cloud 978-1-4673-0412-2/11/$26.00 ©2011 IEEE 1 TABLE I parallel programming techniques like MapReduce. By COMPARISON OF SOFTWARE IN CLOUD AND IN-HOUSE ENVIRONMENT parallelizing job units, tools’ performance and scalability can be greatly enhanced. In-house Hosting Cloud Hosting 2) To address cloud unique quality problems, new test- Architecture Centralized, limited Built-in distributed ing methods, techniques and tools are necessary. Some parallel and fault tol- parallel computing, erance high fault-tolerance frameworks are proposed to provide testing as contin- Configuration Pre-defined Dynamic uous infrastructure services [13], [14]. Testing archi- configuration, deployment tectures like Collaborative Verification and Validation offline deployment, and online (CV&V), Testing-as-aservice (TaaS), self-testing of au- relative fixed set-up reconfiguration, transparent to tomatic services, are proposed recently. Techniques such software providers as simulation, services mocking, job parallelization, and Resource Allocation Stable dedicated re- Unlimited resource environment virtualization can also be used to enhance sources, limited up- pools, dynamic allo- the testing capabilities. per bound cation based on real- time usage 3) To compare across heterogeneous cloud infrastructure Resource Sharing Limited sharing multi-tenancy and development stacks, new metrics are defined and architecture, large- benchmarks [15], [16], [17] are developed to setup the scale resource baseline for testing and evaluation. sharing Runtime Stable dedicated en- Virtualized Cloud testing is an emerging area. This paper suggests new Environment vironment, in-house computing services, trends of cloud testing tools, including the needs for online control unpredictable and adaptive testing, cross-cloud testing, SaaS multi-tenancy environment, low controllability testing, real-time results processing, SLA conformance testing, Scalability Offline scale Online massive scal- dependability testing such as security testing and reliability up/down ability with unlimit- testing. ed resource pool, dy- This paper is organized as follows. Section II analyzes the namic scale up/down in response to usage issues of cloud testing tools. Section III introduces novel ar- chitectures for cloud testing. Section IV reviews enabling tech- niques and typical tool implementations. Section V presents Many new techniques are introduced to support the desired cloud benchmarks. Section VI briefly shows commercial test- features of cloud computing, such as virtualization and MTA. ing tools. Section VII compares the tools and discusses future These techniques also introduce new fault models and threat- needs of tool development. Finally, section VIII concludes this ens to system quality [22], [23], [24], [25], [26]. For example, paper. even an application is allocated with promised hardware like CPU and disks, its performance cannot be guaranteed. The II. CHALLENGES AND NEEDS performance of application in the cloud is affected by many As shown in TABLE I, software hosted in a cloud environ- factors such as the number of virtual machines (VM) instances, ment differs from that in an in-house environment. Essentially, VM management policies, the effectiveness of hypervisor the in-house environment is usually fully controlled by soft- scheduling, the strategy of application migration and recovery ware vendors with dedicated hardware and software resources; in case of failures. Some of these factors change at runtime while the cloud-based hosting provides infrastructure as leased and thus cannot be fully predicted and controlled. services