Performance Model Extraction Using Kernel Event Tracing by Shieryn Tjandra A thesis submitted to the Faculty of Graduate and Postdoctoral Affairs in partial fulfillment of the requirements for the degree of Master of Applied Science in Electrical and Computer Engineering Department of Systems and Computer Engineering Faculty of Engineering Carleton University Ottawa, Ontario, Canada, K1S 5B6 © Copyright 2019, Shieryn Tjandra Abstract Models are used in performance analysis when the analyst needs to be able to predict the effect of system changes that go beyond what can be measured. The model can be obtained from a combination of system knowledge and experimentation. This thesis addresses an experimental approach to obtaining layered queueing network (LQN) models of distributed systems. It applies and extends an approach called SAME (Software Architecture and Model Extraction)which was developed to interpret application-level traces, to interpreting Kernel-level traces. Kernel-level traces have the benefit that application instrumentation is not required, and communication with attached devices can be modeled, but they lack application context information. The research shows that modeling from Kernel traces is feasible in systems which communicate via TCP messages, including Java remote procedure calls. This covers most web-based systems. Systems using middleware pose special problems. The combination of Kernel and application-level tracing was included in some experiments. Tools are described that adapt the Kernel traces to SAME, and that extract CPU demand parameter calibration information. ii Acknowledgements Study in Canada was not an easy decision for me. With all supports, I could finish my study. I would first like to thank God for giving me a chance to fulfil my dream and give me strength whenever I feel exhausted. Also, thank God for listening to the prayers of my grand parents, my mother, the Tjandra family and me. To my beloved mother who always trusts my decisions, has my back all the time, sacrifices yourself to be the best single-mother in the world. You made me what I became today. I know my love to you never been enough for what you have given to me but thank you. To my mentor, Dr. C. Murray Woodside, thank you for helping me to solve my problems, for trusting me when I doubted myself. If it has not been you, I would never finish my study. You have been more than a great teacher for me. Thank you. To my cousins, my best friends thank you for supporting me and standing up for me when people doubted me. To Geneviève Bastien and Houssem Daoud from École Polytechnique de Montréal, thank you for the LTTng and Trace Compass one-week training, for helping me to figure out the concept of TCP algorithm. And never forget to say thank you to colleagues in the RADS lab for all the encouragements, and to my Company colleagues who supported me while I was doing research. Thank you. Last but not least, financial support provided by Mitacs Accelerate is greatly appreciated. iii Table of Contents Abstract ......................................................................................................................... ii Acknowledgements ...................................................................................................... iii Table of Contents ......................................................................................................... iv List of Tables ............................................................................................................. viii List of Illustrations ...................................................................................................... ix Chapter 1: Introduction ...............................................................................................1 1.1 Motivation and Objectives .......................................................................................... 1 1.2 Thesis Contributions .................................................................................................. 2 1.3 Challenges and Solutions............................................................................................ 2 1.4 Industrial Case Study ................................................................................................. 3 1.5 Thesis outline ............................................................................................................. 3 Chapter 2: Background ................................................................................................4 2.1 Performance Engineering ........................................................................................... 4 2.2 Layered Queueing Network (LQN) ............................................................................ 4 2.2.1 Types of Communication in LQN .......................................................................... 7 2.2.2 Making a Model by Hand ....................................................................................... 7 2.3 Inter-Process Communication in Linux ....................................................................... 7 2.4 Linux Kernel Tracing ................................................................................................. 8 2.4.1 LTTng Tracing Tools ............................................................................................. 9 2.4.2 LTTng User-Space Tracepoints .............................................................................. 9 2.4.3 LTTng and Common Trace Format (CTF) ............................................................ 10 2.4.4 LTTng Traces Viewer (Trace Compass) ............................................................... 10 2.5 Perf Tools ................................................................................................................ 10 2.6 Java Remote Method Invocation ............................................................................... 11 iv 2.7 State of The Art of Model Building .......................................................................... 12 Chapter 3: SAME Tools ............................................................................................. 18 3.1 SAME phase 1 ......................................................................................................... 18 3.2 SAME Phase 2 (ExtractModel)................................................................................. 22 Chapter 4: LTTng Traces .......................................................................................... 25 4.1 LTTng Kernel Tracepoints ....................................................................................... 25 4.2 Challenges of using LTTng Kernel ........................................................................... 27 4.3 Dealing with LTTng Kernel challenges .................................................................... 28 Chapter 5: Filtering Tools .......................................................................................... 29 5.1 Obtain the Application Name for each Process ......................................................... 29 5.2 Extraction of Messages Using TCP .......................................................................... 30 5.3 Conversion of File Operations .................................................................................. 33 5.4 Inferring CPU Times: CPUStat Algorithms .............................................................. 35 5.5 Add CPU Times ....................................................................................................... 38 5.6 Combining Parts of a Message ................................................................................. 39 5.7 Dealing with Non-Traced Components ..................................................................... 40 Chapter 6: The Method .............................................................................................. 42 6.1 Overall Workflow of the Method .............................................................................. 42 6.2 Executing the Program ............................................................................................. 44 6.3 Filtering the Trace Data, and SAME Phase 1 ............................................................ 49 6.4 Extracting the Model with SAME ............................................................................. 52 Chapter 7: Client-Server Case Study......................................................................... 57 7.1 Description............................................................................................................... 57 7.2 Model Expectation ................................................................................................... 57 7.3 The Collected Data ................................................................................................... 58 v 7.4 SAME Result ........................................................................................................... 58 7.5 Lessons Learned ....................................................................................................... 59 Chapter 8: Java File Transfer Case Study ................................................................ 61 8.1 Description............................................................................................................... 61 8.2 Model Expectation ................................................................................................... 62 8.3 Data cleaning ..........................................................................................................