AUTOMATED ANALYSIS OF LOAD TESTING RESULTS by ZHEN MING JIANG A thesis submitted to the School of Computing in conformity with the requirements for the degree of Doctor of Philosophy in Computer Science Queen’s University Kingston, Ontario, Canada January 2013 Copyright c Zhen Ming Jiang, 2013 Abstract ANY software systems must be load tested to ensure that they can scale up un- der high load while maintaining functional and non-functional requirements. M Studies show that field problems are often related to systems not scaling to field workloads instead of feature bugs. To assure the quality of these systems, load testing is a required testing procedure in addition to conventional functional testing procedures, such as unit and integration testing. Current industrial practices for checking the results of a load test remain ad-hoc, involving high-level manual checks. Few research efforts are devoted to the automated analysis of load testing results, mainly due to the limited access to large scale systems for use as case studies. Approaches for the automated and systematic analysis of load tests are needed, as many services are being offered online to an increasing number of users. This dissertation proposes automated approaches to assess the quality of a system under load by mining some of the recorded load testing data (execution logs). Execution logs, which are readily available yet rarely used, are generated by output state- ments which developers insert into the source code. Execution logs are hard to parse and analyze automatically due to their free-form structure. We first propose a log abstraction approach that uncovers the internal structure of each log line. Then we propose auto- mated approaches to assess the quality of a system under load by deriving various models (functional, performance and reliability models) from the large set of execution logs. Case studies show that our approaches scale well to large enterprise and open source systems and output high precision results that help load testing practitioners effectively analyze the quality of the system under load. i Dedication To my dad, mum and Kehan. ii Related Publications The following publications are related to this thesis: 1. Zhen Ming Jiang. Automated Analysis of Load Testing Results. In Proceedings of the Doctoral Symposium of the 2010 International Conference on Software Testing and Analysis (ISSTA), pages 143-146. Trento, Italy. July, 2010. [Chapter1] 2. Zhen Ming Jiang, Ahmed E. Hassan, Parminder Flora and Gilbert Hamann. Abstract- ing Execution Logs to Execution Events for Enterprise Applications. In Proceedings of the 8th International Conference on Quality Software (QSIC), pages 181-186. Ox- ford, UK. August, 2008. [Chapter3] 3. Zhen Ming Jiang, Ahmed E. Hassan, Parminder Flora and Gilbert Hamann. An Au- tomated Approach for Abstracting Execution Logs to Execution Events. In the Special Issue on “Program Comprehension through Dynamic Analysis” of Wiley’s Journal of Software Maintenance and Evolution: Research and Practice, pages 249-267. August, 2008. [Chapter3] 4. Zhen Ming Jiang, Ahmed E. Hassan, Parminder Flora and Gilbert Hamann. Auto- matic Identification of Load Testing Problems. In Proceedings of the 24th IEEE Interna- tional Conference on Software Maintenance (ICSM), pages 307-316. Beijing, China. September, 2008. [Chapter4] 5. Zhen Ming Jiang, Ahmed E. Hassan, Gilbert Hamann and Parminder Flora. Automatic Performance Analysis of Load Tests. In Proceedings of the 25th IEEE International iii Conference on Software Maintenance (ICSM), pages 125-134. Edmonton, Canada. September, 2009. [Chapter5] 6. Zhen Ming Jiang, Alberto Avritzer, Emad Shihab, Ahmed E. Hassan and Parminder Flora. An Industrial Case Study on Speeding up User Acceptance Testing by Mining Execution Logs. In Proceedings of the 4th IEEE International Conference on Secure Software Integration and Reliability Improvement (SSIRI), pages 131-140. Singapore. June, 2010. [Chapter6] iv Acknowledgments This thesis would not have been possible without the support of many exceptional people to whom I am grateful. First of all, I would like to thank my parents, who always believe in me and give me the will to succeed. Thank you for supporting me to study in Canada and for being there every step of the way. I am greatly indebted to my supervisor Dr. Ahmed E. Hassan for his support and guid- ance. I met Ahmed when I was an undergraduate research student at the University of Waterloo. Then he co-supervised me for my Master’s degree. Ahmed is like a big brother to me. In many occasions, he offered advice and assistance on various aspects of academic and personal life throughout the journey. I would like to thank my PhD examining committee members: Dr. Saeed Gazor, Dr. Mohammad Zulkernine and Dr. Jerome Rolia for taking time out of their busy schedule to read my thesis and to provide valuable feedback. I would also like to thank Dr. James Cordy and Dr. Patrick Martin for serving on my PhD supervisory committee and providing insightful comments and suggestions. I would like to thank my collaborators: Dr. Emad Shihab, Dr. Bram Adams, Weiyi Shang, Thanh Nguyen, Haroon Malik, Dharmesh Thakkar, Derek Foo, Dr. Abram Hindle, Dr. Michael Godfrey and Dr. Alberto Avritzer. I also appreciate the help of Mark Syer for proof-reading my depth report, Dr. Stephen Thomas for sharing his thesis template and everyone else from the SAIL lab for their support and encouragement. v I am also very fortunate to work with many great people at the Performance Engineering team in Research In Motion (RIM). Many of the techniques in this thesis were developed to address the daily challenges facing these practitioners. In particular, I want to thank Parminder Flora for providing me with an opportunity to work as an embedded researcher in RIM. I also want to thank Gilbert Hamann, Mohamed Nasser, Terry Green and Denny Chiu for many interesting discussions and feedbacks during the biweekly research meetings. I want to thank my good friends, who I spent many days and nights working on school projects and have fun: Jian Wang, Yuli Ye, Yang Gao and HaoQing Zhu. Finally, I am blessed to have a wonderful wife, Kehan Chen. I thank you for your love, support and sacrifice. vi Table of Contents Abstract i Dedication ii Related Publications iii Acknowledgmentsv Table of Contents vii List of Tables ix List of Figures xi Chapter 1: Introduction................................. 1 1.1 Motivation..................................... 1 1.2 Research Hypothesis............................... 5 1.3 Thesis Overview ................................. 5 1.4 Thesis Contributions............................... 9 1.5 Thesis Organization ............................... 10 Chapter 2: A Survey on Load Testing Large Scale Software Systems . 11 2.1 Introduction.................................... 11 2.2 Background.................................... 14 2.3 Research Question 1: How is a proper load designed?............. 21 2.4 Research Question 2: How is a load test executed? .............. 47 2.5 Research Question 3: How is the result of a load test analyzed? . 68 2.6 Conclusion .................................... 81 Chapter 3: Automated Abstraction of Execution Logs................ 82 3.1 Introduction.................................... 82 3.2 Related Work................................... 85 3.3 Measuring the Performance of Approaches for Abstraction Logs . 87 3.4 Our Log Abstraction Approach.......................... 89 3.5 Case Study .................................... 98 vii 3.6 Conclusion .................................... 106 Chapter 4: Automatic Detection of Functional Problems . 107 4.1 Introduction.................................... 107 4.2 Functional Problems in a Load Test....................... 110 4.3 Our Anomaly Detection Approach........................ 112 4.4 Case Studies.................................... 119 4.5 Discussions and Limitations........................... 125 4.6 Related Work................................... 126 4.7 Conclusion .................................... 128 Chapter 5: Automatic Detection of Performance Problems . 129 5.1 Introduction.................................... 129 5.2 Performance Analysis Report........................... 131 5.3 Our Approach to Create the Performance Analysis Report . 135 5.4 Case Studies.................................... 139 5.5 Discussion of Results............................... 147 5.6 Related Work................................... 149 5.7 Conclusion .................................... 150 Chapter 6: Automatic Estimation of System Reliability . 152 6.1 Introduction.................................... 152 6.2 Motivating Example ............................... 155 6.3 Approach Overview................................ 159 6.4 Industrial Case Studies.............................. 169 6.5 Threats to Validity ................................ 172 6.6 Related Work................................... 175 6.7 Conclusion .................................... 177 Chapter 7: Conclusions and Future Work.......................179 7.1 Thesis Findings and Contributions ....................... 180 7.2 Future Work.................................... 183 7.3 Closing Remarks ................................. 184 Bibliography........................................185 Chapter A: Our Paper Selection Process........................219 viii List of Tables 2.1 Interpretations of Load Testing, Performance Testing and Stress Testing . 16 2.2 Load Design Techniques ............................. 25