PerfGuard: Binary-Centric Application Performance Monitoring in Production Environments Chung Hwan Kimy Junghwan Rheez Kyu Hyung Lee{ Xiangyu Zhangy Dongyan Xuy yPurdue University zNEC Laboratories America {University of Georgia West Lafayette, IN, USA Princeton, NJ, USA Athens, GA, USA {chungkim, xyzhang, [email protected] [email protected] dxu}@cs.purdue.edu ABSTRACT other software issues, preventing performance problems be- Diagnosis of performance problems is an essential part of fore software distribution is challenging [42] for the following software development and maintenance. This is in partic- reasons. First, modern software has complex dependency ular a challenging problem to be solved in the production on many components developed by multiple parties. For ex- environment where only program binaries are available with ample, an application may have dependency on third-party limited or zero knowledge of the source code. This problem libraries as well as the system libraries to use the underlying is compounded by the integration with a significant number operating system. Therefore, finding the root causes of per- of third-party software in most large-scale applications. Ex- formance problems requires investigation of the whole soft- isting approaches either require source code to embed man- ware stack of various software component layers [43]. Sec- ually constructed logic to identify performance problems or ond, it is very challenging to identify performance issues support a limited scope of applications with prior manual during the development because software vendors have lim- analysis. This paper proposes an automated approach to ited time and environments to test various complex usage analyze application binaries and instrument the binary code scenarios. Consequently, there have been efforts to diagnose transparently to inject and apply performance assertions on performance problems during production deployment, long application transactions. Our evaluation with a set of large- after the development stage [43, 38, 32, 54]. scale application binaries without access to source code dis- Production-run performance diagnosis has been performed covered 10 publicly known real world performance bugs au- generally in two major ways, which complement each other tomatically and shows that PerfGuard introduces very low and often are used together. First, software vendors main- overhead (less than 3% on Apache and MySQL server) to tain bug reporting systems [15, 12, 23]. These systems are production systems. used for reporting software issues such as performance and failures issues. Users can voluntarily report the details of their performance issues, for instance, how to reproduce the CCS Concepts symptom, the specifications of their system, etc. Second, •Software and its engineering ! Software perfor- some software vendors embed code logic to detect unex- mance; Software testing and debugging; Software pected performance delay and to report the incident to the post-development issues; vendors automatically [16]. Specifically such logic monitors the performance of semantically individual operations of a 1 Keywords program and raises an alarm if their latency exceeds pre- determined thresholds. However, the cost of human efforts Performance diagnosis, post-development testing to support such logic and thresholds is high due to require- ments to perform in-depth analysis on possible application 1. INTRODUCTION behaviors and to determine the range of its reasonable execu- Diagnosis and troubleshooting of performance problems is tion time. In addition, the location to insert the logic needs an essential part of software development and maintenance. to be manually determined considering its functionality and Traditionally, various performance tools [22, 14, 36, 21, 13, run-time impact. Such manual efforts may involve human 34] have been extensively used by developers during the de- errors due to the misunderstanding of complex program be- velopment and testing stages in order to identify inefficient haviors, particularly when dealing with large-scale software. code and prevent performance problems. However, unlike Although automating the process could save significant ef- forts in performance debugging and testing, such feature is not implemented by many software vendors in practice. Permission to make digital or hard copies of all or part of this work for personal or Furthermore, software users at the deployment stage re- classroom use is granted without fee provided that copies are not made or distributed quire performance diagnostics for production software with- for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM out source code or deep knowledge of the target application. must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, For instance, service providers use open source programs to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]. 1 FSE’16, November 13–18, 2016, Seattle, WA, USA Such operations are also known as application transactions, c 2016 ACM. 978-1-4503-4218-6/16/11...$15.00 user transactions [54], units [44], or business transactions [5, http://dx.doi.org/10.1145/2950290.2950347 19, 10]. We will use application transactions herein. 595 or 3rd-party software as part of their large infrastructure. Unit and Metadata Instrumenting App Performance Guard Application with Monitoring their performance in the production stage is im- Identification Performance Guards portant due to their impact on the overall quality of the Pre-distribution service. However, the lack of source code and code knowl- Feedback Production-run Deploy Unit Performance Unit Performance edge for instrumentation make the monitoring challenging. Monitoring Inspection Trigger There are application performance management (APM) ser- vices available in the market [5, 19, 10], but those services Figure 1: PerfGuard Architecture. require developers to modify the application code for insert- ing the monitoring API provided by the service or support a • Implementation and evaluation of a prototype with limited range of known applications and platforms with prior various applications in the Microsoft Windows plat- analysis. Table 1 lists the APM services and existing tech- form showing its practicality discovering performance niques for application performance diagnosis in comparison. bugs automatically in the real world software. Among the compared approaches, only PerfGuard automat- ically identifies application transactions and monitors their x2 presents the design overview of PerfGuard. The key performance with no code knowledge using only binaries. idea of unit-based performance analysis is presented in x3. x4 To provide a generally applicable, binary-centric frame- shows how to identify units and performance guards. Instru- work for performance monitoring, we designed and imple- mentation of applications with performance guards is pre- mented PerfGuard, which automatically transforms the bi- sented in x5. Implementation and evaluation of PerfGuard naries of an application without its source code or prior code are presented in x6 and x7. x8 and x9 respectively show knowledge to monitor its performance at the production run. related work and discussions. x10 concludes this paper. PerfGuard performs dynamic analysis of a target applica- tion and generates a performance profile, which is a \sum- 2. DESIGN OVERVIEW mary" of the application performance behavior. It is used The overall architecture of PerfGuard is presented in Fig- as a \hint" to create a set of monitoring code in the binary ure 1. PerfGuard takes the binaries of a target application format, called performance guards, to be inserted into the as an input and automatically discovers application trans- binaries of the application to diagnose its performance in actions from a set of training runs to produce a perfor- the production environment. During the production run, mance profile incrementally. The preciseness of the perfor- the inserted performance guards automatically monitor the mance profile increases as more training runs are performed. performance of specific application transactions that are au- By analyzing the performance profile PerfGuard generates tomatically determined by our analysis. An unexpected per- a set of binary code snippets, called performance guards, formance delay triggers a performance assertion set by the that are injected into the binaries on the discovered applica- performance guard, and invokes a performance diagnosis to tion transactions to monitor application performance. While help developers and users resolve the issue. the instrumented application is in the production stage, the PerfGuard utilizes program execution partitioning (a.k.a. performance guards detect potential performance anomalies units [44]) to automate the recognition of application trans- and inspect the application's state to find their root causes. actions. Each unit is a segment of the execution of a process To discover application transactions, PerfGuard leverages that handles a specific workload. During the profiling of the the fact that a majority of large-scale applications are event- application, the units of the identical or similar control flow driven and incorporate a small number of loops in each event are clustered into the same unit type. PerfGuard determines