Performance Mimicking Benchmarks for Multi-Tier Applications
Total Page：16
File Type：pdf, Size：1020Kb
Performance Mimicking Benchmarks for Multi-tier Applications Subhasri Duttagupta Mukund Kumar Varsha Apte Performance Engineering Performance Engineering Dept of Computer Science Research Center Research Center and Engg Tata Consultancy Services Tata Consultancy Services Indian Institute of Technology Mumbai, India Mumbai, India Mumbai, India [email protected] [email protected] [email protected] ABSTRACT plications are deployed and performance-tested on a sepa- Predicting performance of multi-tier enterprise applications rate test environment which is often different from the pro- for a target platform is of significant importance to IT in- duction environment. Thus, there is a need for predicting dustries especially when target environment is unavailable application performance on the production (target) platform for deployment. Performance modeling techniques depend given its performance on the test (source) platform. Predic- on accurate estimation of resource demands for a specific tion of metrics such as request throughput and server utiliza- application. This paper proposes a methodology for deriv- tion can be done using existing performance modeling tools ing Performance Mimicking benchmarks (PMBs) that can [3] provided the resource demands of the application (e.g. predict resource demand of application server of multi-tier CPU execution time of a request) are known. In this pa- on-line transaction processing applications on a target en- per, we describe a methodology for generating benchmarks vironment. PMBs do not require the actual application to which can be exploited for obtaining estimate of CPU ser- be deployed on the target itself. These benchmarks invoke vice demand of multi-tier applications, specifically, that of similar method calls as the application at different layers the application tier. in the technology stack that contribute significantly to CPU There are a large number of factors that can affect the utilization. Further, they mimic all send and receive interac- CPU service time of an enterprise application|workload tions with external servers (e.g., database server) and web characteristics, the specific technology stack used by the ap- clients. Ability of PMBs for service demand prediction is plication, the amount of data exchanged with other servers validated with a number of sample multi-tier applications and the platform architecture characteristics. Given the including SPECjEnterprise2010 on disparate hardware con- above, a constructive approach of building a parameterized figurations. These service demands when used in a modi- model of the execution time of an application program is pro- fied version of Mean Value Analysis algorithm, can predict hibitively complex. Instead, the paper presents an approach throughput and response time with accuracy close to 90%. for developing a set of simpler \benchmarks" which are ex- ecuted on the target platform. The information acquired from the benchmark execution along with application pro- CCS Concepts filing information collected on the source platform are used •General and reference ! Performance; •Software to estimate application performance on the target. Unlike and its engineering ! Software performance; applications, these benchmarks may not require installation and population of the application database or depend on Keywords multiple distributed services running on different servers. Hence, their deployments are expected to be simpler. Performance, Prediction, Benchmarks, Cross-platform Benchmarks built using the proposed approach mimic the performance of the actual application by having the right 1. INTRODUCTION mix of the type of CPU operations as used by the applica- Performance characterization of a multi-tier enterprise ap- tion. These Performance Mimicking Benchmarks (PMBs) plication which is to be deployed on a particular production make similar lower-level method calls corresponding to dif- platform is a critical step prior to its release. However, due ferent layers in the technology stack. In addition, they mimic to various practical constraints, the production platform is the send and receive calls of the application as shown in Fig- often not available for application deployment. Instead, ap- ure 1. While profiling a number of mult-tier applications, it is observed that much of the application server CPU resource demand come from interactions among the tiers and calls to Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed various layers as compared to the methods doing the actual for profit or commercial advantage and that copies bear this notice and the full cita- business processing. Hence, the benchmarks are designed to tion on the first page. Copyrights for components of this work owned by others than capture the method calls made across different tiers. As a ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or re- publish, to post on servers or to redistribute to lists, requires prior specific permission result, we expect that the ratio of service demands of PMB and/or a fee. Request permissions from [email protected]. and that of the application remains almost unchanged on ICPE’16 Companion, March 12-18, 2016, Delft, Netherlands the source as well as the target platform. Thus, PMB helps c 2016 ACM. ISBN 978-1-4503-4147-9/16/03. $15.00 DOI: http://dx.doi.org/10.1145/2859889.2859898 45 similarity with a previously profiled set of benchmarks is proposed in [5], [10]. However, enterprise applications dif- fer dramatically from scientific ones in their use of disk and network IO and CPU instruction mix. There are several ap- proaches using Palladio Component Model for performance prediction of component-based applications. Krogman et al [9] extract behavior model of the application using reverse engineering and rely on bytecode benchmarking for predict- ing performance on target platforms. Authors in [2] have used the model successfully for multi-tier J2EE applications. Tak et al propose an approach where a lightweight pro- gram (\PseudoApp") [11] is written that mimics essential application program characteristics, but is much easier to deploy on the target platform. The PseudoApp is built by Figure 1: Illustration of Performance Mimicking capturing system calls of the original application and re- Benchmark. producing them on the target , and a basic integer looping technique is used to capture the CPU usage between system calls. For complex transactions involving a large number of in estimating the demand of the application on the target string processing routines this approach may fail to provide platform. good accuracy. Our paper is different from existing work in the sense that we aim for the benchmark to attain a representative estimate of service demand of the application rather than the same 3. GENERATING PERFORMANCE resource demand as the application. Since exact similarity MIMICKING BENCHMARK is not required, there is potential for the benchmark to be In this section, we describe the methodology for deriving a re-used to represent multiple applications. We believe these performance mimicking benchmark (PMB) for a real-world PMBs can also help in reducing the time to market by pro- enterprise application. Aim of PMB is to emulate appli- viding a baseline performance estimate of the application cation in terms of resource utilization at the application on a given target platform. This paper makes the following server. Since it is difficult to mimic application function- specific contributions: ality without access to the source code, we mimic method calls at the technology layers (like application container or • We propose a method for creating benchmarks that re- databases system) and to the operating system. The PMB quire only application level profiling rather than kernel- is implemented as a web application and deployed using the level tracing or execution environment instrumenta- same container as the actual application. Further, PMB tion. uses the same software stack of the application server and • The benchmarks are designed to provide a represen- the database server (e.g., JBoss and Postgres). We hypoth- tative value of service demand of the application on a esize that a PMB built for a sample application achieves the deployment platform and can help in obtaining early same ratio of service demands between source and target performance estimate in the software life-cycle. platforms. If the resource demands of the sample applica- tion and PMB are AP Psource and PMBsource on the source • We validate our approach by carrying out performance platform, then by obtaining resource demand of PMB on profiling of a number of applications across three archi- the target environment as PMBtarget, the resource demand tectural platforms. The maximum error in prediction of the application can be estimated as follows: of service demand is found to be under 13%. AP Psource × PMBtarget AP Ptarget = PMBsource 2. RELATED WORK In the following sections, the main steps involved in generat- Basic two most commonly used methods for performance ing PMB and actions that the PMB performs are discussed. prediction are simulation and analytical modeling [3],[8]. These methods have been tried out earlier for enterprise 3.1 Main Steps applications but usage of these techniques for doing cross- In order to generate PMB, a sample application is first platform performance prediction is limited due to the effort profiled