CloudCmp: Shopping for a Made Easy Ang Li Xiaowei Yang Srikanth Kandula Ming Zhang Duke University Research

Abstract – has gained much popular- by a cloud, and infrastructure (IaaS) [6], in ity recently, and many companies now offer a variety of which customers can run their applications within virtual public cloud computing services, such as Ap- machines of their chosen operating systems. pEngine, Amazon AWS, and . These Different cloud providers also have different pricing services differ in service models and pricing schemes, models. For instance, Amazon AWS charges customers making it challenging for customers to choose the best by the number of virtual instances a customer instan- suited cloud provider for their applications. This pa- tiates and how long it uses them, while Google’s Ap- per proposes a framework called CloudCmp to help a pEngine charges by the number of CPU cycles a cus- customer select a cloud provider. We outline the de- tomer’s application consumes. sign of CloudCmp and highlight the main technical chal- Worse yet, applications can have drastically different lenges. CloudCmp includes a set of benchmarking tools workloads, e.g., web service versus scientific comput- that compare the common services offered by cloud ing. It is difficult to predict which cloud offers the best providers, and uses the benchmarking results to predict performance for a particular application without deploy- the performance and costs of a customer’s application ing it on each potential cloud provider. However, de- when deployed on a cloud provider. We present prelimi- ploying an application on a cloud has significant over- nary benchmarking results on three representative cloud head. For instance, to use a PaaS provider, a customer providers. These results show that the performance and needs to port its application to the cloud platform’sAPIs. costs of various cloud providers differ significantly, sug- Even for an IaaS provider, deploying an application may gesting that CloudCmp, if implemented, will have prac- involve and configuration file migrations. tical relevance. In this work, we propose a framework called Cloud- Cmp that helps a potential cloud customer estimate the 1. INTRODUCTION performance and costs of running a legacy application on a cloud without actually deploying the application. Cloud computing has emerged as a revolutionary ap- The framework first characterizes the services offered proach to provide computing services. It offers sev- by various cloud providers into a set of common ser- eral key advantages over a traditional in-house comput- vice interfaces, and benchmarks the performance and ing model, including on-demand scaling, resource multi- costs of these services. It then expresses an application’s plexing, pay-as-you-go metered service, and high-speed workload using the interfaces, and estimates the appli- network access. A cloud customer only needs to pay cation’s performance and costs based on the benchmark- the usage costs of its applications, which are likely to be ing results. This approach to performance prediction re- much smaller than the upfront investments of building sembles the measurement-based prediction approaches its own infrastructure. widely adopted in evaluating different high performance The advantages of cloud computing have enticed a computing systems [15, 18]. To the best of our knowl- number of companies to enter this market [2, 4, 5, 8, 9], edge, we are the first to generalize the approach to eval- including traditional online service providers such as uate cloud providers. Amazon, Google, and Microsoft, web hosting compa- As an initial step, this paper focuses on characterizing nies such as Rackspace and GoGrid, and new start-ups and benchmarking the common services offered by dif- such as Flexiant and . A practical challenge thus ferent cloud providers. We first identify a set of services arises. How can a potential cloud customer, e.g., a small shared by six representative cloud providers: Google enterprise wishing to outsource its IT infrastructure, de- AppEngine, Amazon AWS, Microsoft Azure, GoGrid, cide whether it should migrate some services to a cloud? and Rackspace CloudServers/CloudSites. The providers And if so, which cloud provider should it choose? range from PaaS (AppEngine and CloudSites) to IaaS Answering these questions is not straightforward, due (CloudServers and GoGrid) and a combination of the to the diversity of cloud providers and the applications two (AWS and Azure). We then develop a set of bench- they support. marking tools to measure and compare the performance Different cloud providers may offer different service and monetary costs of each type of services. It is our models, including (PaaS) [13], in ongoing work to use these benchmarking results to es- which a cloud customer builds its applications using the timate the performance and costs of an application on computing platform (e.g., a Python sand-) provided different cloud providers, and we outline the challenges A cloud provider typically offers the following ser- and possible solutions. vices for web applications: Our preliminary benchmarking results indicate that cloud providers differ significantly in both performance • Elastic compute cluster. The cluster includes an and costs of the services they provide, and one provider elastic number of virtual instances that run the ap- is unlikely to ace all services. For example, provider X plication’s code and process incoming requests. is able to execute the CPU intensive tasks in less than • Persistent storage. The storage service stores ap- half of the time used by provider Y, while only charg- plication data in a similar fashion as traditional ing <10% more per task. However, provider X’s stor- . age service is slower than that of others when the data • Intra-cloud network. The network inside a cloud set is large, and has larger latency variation. In another that connects the virtual instances of an applica- example, provider Z has good performance in computa- tion, and connects them to cloud-providedservices tion, but its scaling speed is much slower than that of the such as the persistent storage service. other providers. The results suggest that choosing the • Wide-area delivery network. The wide-area de- best suited cloud platform for a particular application is livery network of a cloud delivers an application’s a non-trivial problem, and a cloud comparison frame- contents to the end hosts from multiple geographi- work will be valuable to guide cloud selection. cally distributed data centers of the cloud. We note that our measurements represent a snapshot in time. Changes in the load offered by customers as Table 1 summarizes the services offered by the six well as changes to the , hardware, and network cloud providers. Next, we describe how the cloud infrastructure or the pricing model offered by a provider providers differ in terms of performance and costs for can qualitatively impact the results. Our goal is to de- each type of service. sign a comprehensive benchmark that closely approxi- mates customer use-cases. By not being specific to to- 2.1 Elastic Compute Cluster day’s cloud offerings, we conjecture that such a com- A compute cluster provides virtual instances that host parison framework will help clarify customer choice as and run a customer’s application code. The virtual in- cloud services evolve. stances among various cloud providers differ in at least three aspects: the underlying hardware, the virtualiza- tion technologies, and the hosting environment. These 2. CLOUD COMPUTING SERVICES differences determine how efficient an application can In this section, we summarize the common ser- run and how many concurrent requests an instance can vices offered by six representative cloud providers: serve. They may also affect the cost to host an applica- Amazon AWS [2], Microsoft Azure [8], Google Ap- tion, as the customer may need more instances to serve pEngine [5], GoGrid [4], and - the same amount of workload if each instance is weaker. Servers/CloudSites [9]. The providers have different The compute cluster is also “elastic” in the sense that service models: AppEngine and CloudSites are PaaS a customer can dynamically and efficiently scale up and providers, whereas CloudServers and GoGrid are IaaS down the number of instances to match the application’s ones. AWS and Azure provide both PaaS and IaaS workload. An application with low scaling latency can functionality. The IaaS providers (AWS, Azure, Cloud- absorb workload surges without introducing additional Servers, and GoGrid) support native application code, processing latency, and requires fewer always-running with most of them further supporting different guest op- instances. For example, assume an application’s work- erating systems such as and Windows. In contrast, load increases at a maximum rate of 100 requests per AppEngine and CloudSites only provide sandbox envi- second, and each instance can handle 50 concurrent re- ronments to run managed code written in languages such quests. With an unrealistic (short) one second scaling la- as Java and Python. tency to allocate new instances, the customer only needs We find that the services provided by different cloud to maintain two always-running instances in addition to providers do not fully overlap. As a preliminary step, those necessary to support the current load. Since two we focus on the set of common services that support new instances can be allocated every second, the scaling web applications, which is the only type of applications speed of the cluster keeps up with the workload increase supported by all six cloud providers, and is also one of speed and ensures that requests are never bottlenecked the most popular types of applications supported by to- for processing. In contrast, if the scaling latency is 10 day’s cloud providers. Extending to other types such as seconds, to maintain the same guarantee, the customer backup and computation services (e.g., HPC, MapRe- would have to maintain at least 20 additional always- duce, and Dryad) remains future work. running instances. Cloud Elastic Compute Persistent Intra-cloud Wide-area Provider Cluster Storage Service Network Delivery Network AWS -based VM SimpleDB 3 locations (2 in US, 1 in Europe) Azure Azure VM Azure Table 6 data center locations (2 in each of US, Europe, and Asia) AppEngine Sandbox DataStore Unpublished number of Google data centers Proprietary CloudServers Xen-based VM N/A 2 data center locations in US CloudSites Sandbox MySQL/MSSQL 2 data center locations in US GoGrid Xen-based VM N/A 1 data center location in US

Table 1: The services offered by the cloud providers. DataStore also charges based on each operation’s CPU Pricing: All providers except AppEngine and consumption, but offers 60 free CPU hours per applica- CloudSites charge customers based on how many in- tion per day. Azure Table Storage currently charges a stances are allocated and how long each instance is used, customer only based on the number of operations, re- regardless of whether the instances are fully utilized. gardless of each operation’s complexity. AppEngine charges based on the number of CPU cycles a customer’s code consumes; an idle application incurs 2.3 Intra-cloud Network no cost. It also offers 6.5 free CPU hours per applica- The network infrastructure inside a cloud provider’s tion per day. CloudSites charges a monthly fee which data centers connects the virtual instances of an appli- includes a fixed amount of CPU cycles the application cation among themselves and with the shared cloud- can consume. It also charges any over-limit fee based on provided services. All providers promise high band- how many extra cycles are used beyond the quota. width network paths (typically from hundreds of Mbps We do not consider the differences in the customiz- to Gbps), approximating a private data center network. ability of a virtual instance, because it does not directly The actual network infrastructures of all the cloud impact the performance or costs of running an applica- providers we studied are proprietary. tion. We also do not consider the engineering cost in Pricing: Currently intra-cloud network is offered at no porting an application to a cloud, because such cost is cost by all the examined providers. hard to measure by a technical method. Our experience suggests that the more complex the application and the 2.4 Wide-area Delivery Network ecosystem supporting the application (dependencies on The wide-area delivery network serves an applica- specific databases such as Oracle or specific libraries) tion’s content from geographically dispersed locations. the greater the cost. We havemanually ported our bench- Each end user’s request can be directed to the instances marking tasks to run on both PaaS and IaaS providers. close to the user to minimize the wide-area network la- We had to make minimal modifications to code in this tency. Here we focus on the delivery network for dy- process, whereas the main monetary costs for this re- namic content, because static objects can be served by a search came from running the benchmarking tasks over third-party CDN, decoupled from a cloud platform. extended periods. Almost all cloud providers provide a handful of ge- 2.2 Persistent Storage Service ographically distributed data centers to host an applica- tion. Different clouds have data centers at different loca- All Cloud providers except CloudServers and GoGrid tions. For example, Azure currently has data centers in offer persistent storage services that store the dynamic US, Europe, and Asia, while CloudServers only has data application data in lieu of the traditional database back- centers in US. Such differencescan affect the application ends in legacy web applications. Most of the services response time observed by an end user, which in turn provide SQL-like interfaces but with limited support for has significant impact on the application’s revenue [16]. complex operations such as select. Compared to tradi- This is particularly important as today’s web applica- tional databases, the storage services offered by cloud tions face a much broader and geographically more di- providers are highly scalable and robust against failures verse user base than before. via replication [1]. The storage services have similar interfaces across Pricing: Charges for using the wide-area delivery net- providers, but are implemented with different propri- work are based on the amount of data delivered through etary technologies. Consequently, they vary in perfor- the cloud boundaries to the end users. Currently all the mance, as suggested by our benchmarking results in § 4. providers we studied have similar prices for this service. Pricing: Amazon’s SimpleDB and CloudSites charge a customer based on the CPU cycles consumed to execute 3. CloudCmp FRAMEWORK each storage operation. Therefore, a complex operation The ultimate goal of the CloudCmp framework is to can incur a higher cost than a simple one. AppEngine’s estimate the performance and costs of a legacy appli- Wide-area Frontend Application cation if it is deployed on a chosen cloud provider. A Network Server Server potential cloud customer can use the results to compare 20 5 different providers and decide whether it should migrate Database 53 to the cloud and which cloud provider is best suited for 13 257 its applications. 130

In this section, we outline a preliminary design of the 7 CloudCmp framework. CloudCmp estimates the per- 6 formance and costs of an application via three steps: service benchmarking, application workload collection, Figure 1: An example of a request in a standard three- and performance prediction. For each step, we describe tier web application. The execution path of the request is the challenges in realizing it and the possible solutions. shown and the time (ms) is marked at each component on the path. The processing time of each component is com- 3.1 Service Benchmarking puted from the service performance profiles, and the over- all processing time (491ms) is the sum of each component’s The goal of service benchmarking is to generate a processing time as the execution path is single-threaded. cloud service’s performance and costs profile which in- cludes the time to finish a task and the associated cost. Fortunately, there has been much study in obtaining Challenges: The benchmarking tasks need to be care- execution paths annotated with causal relationships. One fully chosen to represent generic customer applications technique of particular interest is vPath [19]. It ex- and to avoid measurement bias. For example, to unbi- ploits the synchronous intra-thread communication pat- asedly benchmark the performance of a compute clus- tern of the thread-based programming model and ex- ter, we develop a set of standard CPU, memory, and tracts the causality of events within the same thread. I/O benchmarking tasks written in Java (some are based vPath can infer the precise request execution path in on the tests in SPECjvm2008 [11], an existing stan- multi-threaded programs without modifying the appli- dard benchmarking suite for Java Virtual Machines). We cation. Another technique //trace [17] uses throttling to chose Java to ensure fair comparison and mitigate the infer event causality: if throttling a parent event leads to bias introduced by different compilers/interpreters, be- the absence of a child event, it indicates a causal relation- cause Java is one of the two languages supported by all ship between the two events. In our design, we propose cloud providers we study (the other language is Python, to use a vPath-like technique to obtain a request’s ex- which is not as widely used as Java in web applications). ecution path if the application uses the common thread This design allows us to run the same Java bytecode on programming model. For applications that do not con- different cloud providers. form to the model, we will use the throttling technique We also design benchmarks to measure the per- as in //trace to infer inter-event dependencies. formance and costs of other services (i.e., besides CPU/memory/disk benchmarks). Some preliminary 3.3 Performance Prediction benchmarking results are shown in § 4. Due to space limitation, we omit the design details of those tools. In this step, we will try to predict the processing time and the costs of each application request we collect by using the pre-generated performance and costs profile of each cloud service. We first use the profiles to estimate 3.2 Application Workload Collection the time and costs spent at each component on the exe- The second step toward performance/costs prediction cution path, and then combine the individual processing is to obtain workload representative of an application. time and costs together to obtain the overall time and To do so, we propose to collect the application’s request costs, as shown in Figure 1. traces and each request’s execution path. A request’s Challenges: To compute the overall processing time trace should includethe sourceIP of the request, the time of a request, we need to identify the critical path that when the request arrives, the size of the request, etc. takes the longest time among all branches in the exe- Challenges: It is a non-trivial task to collect a re- cution path. This can be obtained by simulating how quest’s execution path, as it involves recording the se- a request is processed. The simulated processing time quence of events (e.g., queries/answers sent to/from a at each component is estimated from the service perfor- database backend) triggered by the request and inferring mance profiles. We can simplify the performance pre- their causal relationships. It may be necessary to infer diction step if a request is processed in a single thread, the causal relationships between events across layers to i.e., the execution path contains only one branch, be- predict the impact on an application’s performance from cause in this case the overall processing time equals the changes at one layer. sum of each component’s processing time. 1.2 instance. To benchmark a virtual instance’s efficiency, Cloud X Cloud Y (Linux) 1 Cloud Y (Windows) we develop a set of standard benchmarking tasks in Java Cloud Z 0.8 and run them on different compute clusters. For each

0.6 provider, we choose to benchmark the default instance type. For cloud X, we run the tasks on its default sand- 0.4 box environment. For Y , we run the tasks on both Linux 0.2 Normalized Finishing Time ( 9.04) and Windows (Server 2008) based in- 0 scimark.fft crypto.aes crypto.rsa memory.smallmemory.large stances to test whether different OSes would affect the benchmarking results. For Z, we run the tasks on Win- (a) Normalized finishing time dows Server 2008. Figure 2(a) shows the results. We include the finishing 1.2 Cloud X time of five tasks, with three CPU intensive ones (sci- Cloud Y (Linux) 1 Cloud Y (Windows) mark.fft, crypto.aes, and crypto.rsa) and two memory in- Cloud Z 0.8 tensive ones (memory.small and memory.large). We can

0.6 see from the figure that for CPU intensive tasks, cloud

0.4 X has the best performance, followed by Z, while Y

0.2 has the worst performance. For cryptographic compu- Normalized Cost per Task tations, cloud X instances are almost twice as fast as 0 scimark.fft crypto.aes crypto.rsa memory.smallmemory.large Y instances. However, for memory intensive tasks, X’s performance does not show significant advantages. In (b) Normalized cost per task fact, for the task with a large memory allocation size (memory.large) that does not fit into the L2 cache, X Figure 2: The benchmarking results of the compute clus- performsthe worst. This result suggests that X instances ters. Figure (a) shows the finishing time of each bench- might have faster CPUs but slower memory controllers. marking task. The time is normalized using the longest The two types of Y instances (Linux and Windows- finishing time in all tests. Figure (b) shows the normalized cost per task. based) have similar performance. This suggests that the benchmarkingresults are independent of the guest OSes. 4. BENCHMARKING RESULTS Therefore, they are likely to reflect the differences in the In this section, we present some preliminary bench- underlying hardware and the virtualization technologies marking results of the common services offered by three used by different cloud providers. popular and representative cloud providers. Due to legal Cost We further measure the cost of running each task. concerns, we anonymize the names of the providers we Cloud Y and Z charge a customer based on how long study, and refer to them as cloud X, Y , and Z. each virtual instance is used. We thus compute the cost The goal of the measurement study is to examine per task using the task’s finishing time and the per hour whether the performance of the cloud services varies price for them. X charges a customer based on each across different providers. It is our future work to further task’s CPU consumption. We compute the task’s cost understand the causes of the performance differences. by multiplying the CPU time with the per CPU second The results in this section show a snapshot of the perfor- price for X. The pricing information is obtained from mance and costs of each cloud provider at the provider’s each provider’s public website. default data center over a two-week period from March Figure 2(b) shows the results. Despite having the 13th to March 25th, 2010. We are currently extending worst performance in the previous test, the Linux-based the measurements to run continuouslyin all the data cen- cloud Y instance turns out to have the lowest cost for ters offered by these cloud providers. More results will the first two tasks. This is because the Linux-based Y be available on the project’s website [3]. instance has the lowest per hour price among the three We stress that the specifics of the results will change instance types charged by usage time (Linux/Windows- as providers evolve their software, hardware and net- based Y and Z instances). The cost per task of the work. Our goal is to formulate a methodology to com- Linux-based Y instance is even lower than that of X, pare providers and we use these results to only demon- although the task takes less time to finish on X. The re- strate the value of such a comparison tool. sults suggest that the metric of cost per task captures the cost-effectiveness of each service regardless of its pric- 4.1 Elastic Compute Cluster ing scheme. We first compare the performance of the elastic com- Scaling We then benchmark the scaling service by mea- pute clusters. We benchmark both the efficiency of a suring the latency between when a new instance is re- virtual instance and the scaling latency to allocate a new Cloud TCP throughput (Mbps) 800 Provision Latency Provider Median 5th Percentile 95th Percentile 700 Booting Latency Total Latency Y 630 465 742 600 Z 626 15.1 687 500 Table 2: The TCP throughput between two instances in 400

Time (s) each cloud. 300

200 1

100 0.8 0 Cloud Y (Linux) Cloud Z (Windows) 0.6

Figure 3: The scaling latency to allocate a new virtual 0.4 instance. Cumulative Fraction 0.2 Cloud X Cloud X Cloud Y 1000 Cloud Y Cloud Z Cloud Z 0 0.05 0.1 0.15 0.2 0.25 0.3 Round Trip Time (s)

The distributions of the round trip time from 100 Figure 5: 260 PlanetLab vantage points to their optimal data centers.

Operation Latency (ms) forms well when the table size is small, but its latency

10 increases significantly with a large variance when the fetch/1K insert/1K fetch/100K insert/100K table size increases. The results suggest that different storage services can offer different performance at dif- Figure 4: The time it takes to insert/fetch a random entry ferent data scales with different operations. Hence, it to/from a data table in a storage service. The x-axis label is important to design benchmarking tools that cover a shows the operation type (fetch or insert) and the size of comprehensive set of operations. the table (1K entries or 100K entries). The y-axis is in a logarithmic scale. 4.3 Intra-cloud Network quested and when the instance is ready. We do not con- The intra-cloud network performance is benchmarked sider X because its scaling is done automatically and by measuring the available bandwidth between two in- cannot be triggered manually. The scaling latency is fur- stances in the cloud. We use the iperf tool [7] to ther split into a provisioning latency, during which the measure the average TCP throughput between two in- cloud is preparing to launch a new instance, and a boot- stances, and repeat the experiment on many different ing latency, during which the instance’s operating sys- pairs of instances. We do not consider cloud X be- tem is booting. cause it does not allow direct communication between Figure 3 shows the result. It show that cloud Y has instances. Table 2 shows the results. From the table, we a much shorter scaling latency than cloud Z. This sug- see both providers have an average throughput higher gests that the two cloud providers perhaps use different than 600Mbps. Cloud Z’s throughput has a large vari- strategies to manage virtual instances. Furthermore, the ation, ranging from 10Mbps to around 700Mbps. We booting time of Linux on Y appears shorter than that of speculate that it might be caused by network congestion Windows 2008 on Z, suggesting that choosing the right inside Z’s data center. combination of the guest OS and the virtualization layer can be important for scaling performance. 4.4 Wide-area Delivery Network We measure the latency of each cloud’s delivery net- 4.2 Persistent Storage Service work by sending ping packets to the application in- We benchmark the performance of the persistent stor- stances deployed at a cloud provider’s data centers. We age services by measuring the latency to insert (or fetch) send the ping packets from geographically distributed a random entry to and from a pre-defined data table in vantage points. For each vantage point, we choose the each service. We run the tests against two tables with 1K data center with the minimum round trip time as the op- and 100K entries respectively, and Figure 4 shows the timal data center selected by a perfect geographic load results. All the storage services exhibit significant varia- balancing algorithm. tions in latency across requests (indicated by the lengths Figure 5 shows the distributions of the round trip time of the error bars whose ends show the 5th and 95th per- measured from 260 randomly chosen vantage points on centiles respectively). For fetch operations, Y performs PlanetLab to their optimal data centers. We see from best under both table sizes, while for insert operations, the figure that X has the lowest wide-area network la- Z outperforms the other two. X’s storage service per- tency distribution among the three providers. This is not surprising because X also offers popular online ser- other types of applications), and more data center loca- vices globally, and is known to have a large number of tions. We also plan to provide continuous rather than data centers around the world. Y and Z have similar snapshot benchmarking results. We can then build a latency distributions and Y has slightly more vantage complete CloudCmp system to predict application per- points with latencies greater than 100ms. Closer exam- formance and costs based on the benchmarking results ination of the data reveals that those vantage points are we collect. mostly in Asia and South America, where cloud Y does not have any data center presence. Acknowledgements Thiswork was fundedin partbyan NSF CAREER Award 5. RELATED WORK CNS-0845858 and Award CNS-0925472. We thank the Garfinkel [14] and Walker [20] investigate the perfor- anonymousreviewers for their helpful feedback and sug- mance of one specific cloud provider: Amazon’s AWS. gestions. In contrast, this work focuses on comparing the per- formance of multiple cloud providers. It serves as an 7. REFERENCES [1] Amazon SimpleDB. http://aws.amazon.com/simpledb/. initial step toward a framework that can assist a cus- [2] Amazon Web Service. http://aws.amazon.com. tomer to choose a cloud provider based on its cost and [3] CloudCmp Project Website. http://cloudcmp.net. performance requirements. A few industry reports per- [4] GoGrid Cloud Hosting. http://gogrid.com. form simple pair-wise evaluations on the computation [5] Google AppEngine. http://code.google.com/appengine. [6] Infrastructure-as-a-Service (IaaS). Gni, Tech. Rep. performance of different cloud instances [10, 12]. This [7] Iperf. http://iperf.sourceforge.net. work compares a broader range of cloud services includ- [8] Azure. http://www.microsoft.com/ ing computation, scaling, storage, and networking ser- windowsazure. [9] Rackspace Cloud. http://www.rackspacecloud.com. vices. Ward compares the performance of a public cloud [10] Rackspace Cloud Servers versus Amazon EC2: Performance provider AWS with that of a private cloud software suite: Analysis. http://www.thebitsource.com/featured- Ubuntu Enterprise Cloud [21], while this work focuses posts/rackspace-cloud-servers-versus-amazon- ec2-performance-analysis/. on comparing the performance and costs of the public [11] SPEC Java Virtual Machine Benchmark 2008. http://www. cloud providers. spec.org/jvm2008/. [12] VPS Performance Comparison. http://journal.uggedal. 6. CONCLUSIONANDFUTUREWORK com/vps-performance-comparison. [13] D. Cheng. PaaS-onomics: A CIOs Guide to using Cloud computing has gained much momentum in re- Platform-as-a-Service to Lower Costs of Application Initiatives cent years due to its economicadvantages. Various cloud While Improving the Business Value of IT. Longjump, Tech. Rep. providers offer different types of services with different [14] S. Garfinkel. An Evaluation of Amazon s Grid Computing pricing schemes, posing a practical challenge on how to Services : EC2 , S3 and SQS. Harvard University, Tech. Rep. choose the best performing cloud provider for an appli- TR-08-07. cation. This paper proposes a framework to estimate and [15] S. Hammond, G. Mudalige, J. Smith, S. Jarvis, J. Herdman, and A. Vadgama. WARPP: a toolkit for simulating compare the performance and costs of deploying an ap- high-performance parallel scientific codes. In International plication on a cloud. This framework characterizes the Conference on Simulation Tools and Techniques, page 19, 2009. common set of services a cloud provides, includingcom- [16] R. Kohavi, R. M. Henne, and D. Sommerfield. Practical guide to controlled experiments on the web. In ACM KDD, volume 2007. putation, storage, and networking services, benchmarks ACM Press, 2007. each service’s performance and costs, and combines the [17] M. P. Mesnier, M. Wachs, R. R. Sambasivan, J. Lopez, benchmarking results with an application’s workload to J. Hendricks, G. R. Ganger, and D. O’Hallaron. //trace: parallel trace replay with approximate causal events. In USENIX FAST, estimate the application’s processing time and costs. As 2007. an initial step toward building such a framework, we [18] G. Nudd, D. Kerbyson, E. Papaefstathiou, S. Perry, J. Harper, benchmark and compare the cloud services provided by and D. Wilcox. Pace–A Toolset for the Performance Prediction of Parallel and Distributed Systems. International Journal of three representative cloud providers to support web ap- High Performance Computing Applications, 14(3):228, August plications. The results show that the performance and 2000. costs of different cloud providers can differ significantly, [19] B. C. Tak, C. Tang, C. Zhang, S. Govindan, B. Urgaonkar, and suggesting that a systematic cloud comparison frame- R. N. Chang. vPath: Precise Discovery of Request Processing Paths from Black-Box Observations of Thread and Network work is highly relevant. Activities. In USENIX ATC, 2009. In the future, we plan to extend the benchmarking [20] E. Walker. Benchmarking amazon EC2 for high-performance tools to cover more cloud providers, more types of cloud scientific computing. USENIX Login, 2008. [21] J. S. Ward. A Performance Comparison of Clouds: Amazon services (e.g., the different tiers of instances offered by EC2 and Ubuntu Enterprise Cloud. SICSA DemoFEST, 2009. the same cloud provider and the services that support