Performance Model Extraction Using Kernel Event Tracing

Total Page:16

File Type:pdf, Size:1020Kb

Performance Model Extraction Using Kernel Event Tracing Performance Model Extraction Using Kernel Event Tracing by Shieryn Tjandra A thesis submitted to the Faculty of Graduate and Postdoctoral Affairs in partial fulfillment of the requirements for the degree of Master of Applied Science in Electrical and Computer Engineering Department of Systems and Computer Engineering Faculty of Engineering Carleton University Ottawa, Ontario, Canada, K1S 5B6 © Copyright 2019, Shieryn Tjandra Abstract Models are used in performance analysis when the analyst needs to be able to predict the effect of system changes that go beyond what can be measured. The model can be obtained from a combination of system knowledge and experimentation. This thesis addresses an experimental approach to obtaining layered queueing network (LQN) models of distributed systems. It applies and extends an approach called SAME (Software Architecture and Model Extraction)which was developed to interpret application-level traces, to interpreting Kernel-level traces. Kernel-level traces have the benefit that application instrumentation is not required, and communication with attached devices can be modeled, but they lack application context information. The research shows that modeling from Kernel traces is feasible in systems which communicate via TCP messages, including Java remote procedure calls. This covers most web-based systems. Systems using middleware pose special problems. The combination of Kernel and application-level tracing was included in some experiments. Tools are described that adapt the Kernel traces to SAME, and that extract CPU demand parameter calibration information. ii Acknowledgements Study in Canada was not an easy decision for me. With all supports, I could finish my study. I would first like to thank God for giving me a chance to fulfil my dream and give me strength whenever I feel exhausted. Also, thank God for listening to the prayers of my grand parents, my mother, the Tjandra family and me. To my beloved mother who always trusts my decisions, has my back all the time, sacrifices yourself to be the best single-mother in the world. You made me what I became today. I know my love to you never been enough for what you have given to me but thank you. To my mentor, Dr. C. Murray Woodside, thank you for helping me to solve my problems, for trusting me when I doubted myself. If it has not been you, I would never finish my study. You have been more than a great teacher for me. Thank you. To my cousins, my best friends thank you for supporting me and standing up for me when people doubted me. To Geneviève Bastien and Houssem Daoud from École Polytechnique de Montréal, thank you for the LTTng and Trace Compass one-week training, for helping me to figure out the concept of TCP algorithm. And never forget to say thank you to colleagues in the RADS lab for all the encouragements, and to my Company colleagues who supported me while I was doing research. Thank you. Last but not least, financial support provided by Mitacs Accelerate is greatly appreciated. iii Table of Contents Abstract ......................................................................................................................... ii Acknowledgements ...................................................................................................... iii Table of Contents ......................................................................................................... iv List of Tables ............................................................................................................. viii List of Illustrations ...................................................................................................... ix Chapter 1: Introduction ...............................................................................................1 1.1 Motivation and Objectives .......................................................................................... 1 1.2 Thesis Contributions .................................................................................................. 2 1.3 Challenges and Solutions............................................................................................ 2 1.4 Industrial Case Study ................................................................................................. 3 1.5 Thesis outline ............................................................................................................. 3 Chapter 2: Background ................................................................................................4 2.1 Performance Engineering ........................................................................................... 4 2.2 Layered Queueing Network (LQN) ............................................................................ 4 2.2.1 Types of Communication in LQN .......................................................................... 7 2.2.2 Making a Model by Hand ....................................................................................... 7 2.3 Inter-Process Communication in Linux ....................................................................... 7 2.4 Linux Kernel Tracing ................................................................................................. 8 2.4.1 LTTng Tracing Tools ............................................................................................. 9 2.4.2 LTTng User-Space Tracepoints .............................................................................. 9 2.4.3 LTTng and Common Trace Format (CTF) ............................................................ 10 2.4.4 LTTng Traces Viewer (Trace Compass) ............................................................... 10 2.5 Perf Tools ................................................................................................................ 10 2.6 Java Remote Method Invocation ............................................................................... 11 iv 2.7 State of The Art of Model Building .......................................................................... 12 Chapter 3: SAME Tools ............................................................................................. 18 3.1 SAME phase 1 ......................................................................................................... 18 3.2 SAME Phase 2 (ExtractModel)................................................................................. 22 Chapter 4: LTTng Traces .......................................................................................... 25 4.1 LTTng Kernel Tracepoints ....................................................................................... 25 4.2 Challenges of using LTTng Kernel ........................................................................... 27 4.3 Dealing with LTTng Kernel challenges .................................................................... 28 Chapter 5: Filtering Tools .......................................................................................... 29 5.1 Obtain the Application Name for each Process ......................................................... 29 5.2 Extraction of Messages Using TCP .......................................................................... 30 5.3 Conversion of File Operations .................................................................................. 33 5.4 Inferring CPU Times: CPUStat Algorithms .............................................................. 35 5.5 Add CPU Times ....................................................................................................... 38 5.6 Combining Parts of a Message ................................................................................. 39 5.7 Dealing with Non-Traced Components ..................................................................... 40 Chapter 6: The Method .............................................................................................. 42 6.1 Overall Workflow of the Method .............................................................................. 42 6.2 Executing the Program ............................................................................................. 44 6.3 Filtering the Trace Data, and SAME Phase 1 ............................................................ 49 6.4 Extracting the Model with SAME ............................................................................. 52 Chapter 7: Client-Server Case Study......................................................................... 57 7.1 Description............................................................................................................... 57 7.2 Model Expectation ................................................................................................... 57 7.3 The Collected Data ................................................................................................... 58 v 7.4 SAME Result ........................................................................................................... 58 7.5 Lessons Learned ....................................................................................................... 59 Chapter 8: Java File Transfer Case Study ................................................................ 61 8.1 Description............................................................................................................... 61 8.2 Model Expectation ................................................................................................... 62 8.3 Data cleaning ..........................................................................................................
Recommended publications
  • Building Embedded Linux Systems ,Roadmap.18084 Page Ii Wednesday, August 6, 2008 9:05 AM
    Building Embedded Linux Systems ,roadmap.18084 Page ii Wednesday, August 6, 2008 9:05 AM Other Linux resources from O’Reilly Related titles Designing Embedded Programming Embedded Hardware Systems Linux Device Drivers Running Linux Linux in a Nutshell Understanding the Linux Linux Network Adminis- Kernel trator’s Guide Linux Books linux.oreilly.com is a complete catalog of O’Reilly’s books on Resource Center Linux and Unix and related technologies, including sample chapters and code examples. ONLamp.com is the premier site for the open source web plat- form: Linux, Apache, MySQL, and either Perl, Python, or PHP. Conferences O’Reilly brings diverse innovators together to nurture the ideas that spark revolutionary industries. We specialize in document- ing the latest tools and systems, translating the innovator’s knowledge into useful skills for those in the trenches. Visit con- ferences.oreilly.com for our upcoming events. Safari Bookshelf (safari.oreilly.com) is the premier online refer- ence library for programmers and IT professionals. Conduct searches across more than 1,000 books. Subscribers can zero in on answers to time-critical questions in a matter of seconds. Read the books on your Bookshelf from cover to cover or sim- ply flip to the page you need. Try it today for free. main.title Page iii Monday, May 19, 2008 11:21 AM SECOND EDITION Building Embedded Linux SystemsTomcat ™ The Definitive Guide Karim Yaghmour, JonJason Masters, Brittain Gilad and Ben-Yossef, Ian F. Darwin and Philippe Gerum Beijing • Cambridge • Farnham • Köln • Sebastopol • Taipei • Tokyo Building Embedded Linux Systems, Second Edition by Karim Yaghmour, Jon Masters, Gilad Ben-Yossef, and Philippe Gerum Copyright © 2008 Karim Yaghmour and Jon Masters.
    [Show full text]
  • Linux-Cookbook.Pdf
    LINUX COOKBOOK ™ Other Linux resources from O’Reilly Related titles Linux Device Drivers Exploring the JDS Linux Linux in a Nutshell Desktop Running Linux Learning Red Hat Enterprise Building Embedded Linux Linux and Fedora Systems Linux Pocket Guide Linux Security Cookbook Understanding the Linux Kernel Linux Books linux.oreilly.com is a complete catalog of O’Reilly’s books on Resource Center Linux and Unix and related technologies, including sample chapters and code examples. ONLamp.com is the premier site for the open source web plat- form: Linux, Apache, MySQL, and either Perl, Python, or PHP. Conferences O’Reilly brings diverse innovators together to nurture the ideas that spark revolutionary industries. We specialize in document- ing the latest tools and systems, translating the innovator’s knowledge into useful skills for those in the trenches. Visit conferences.oreilly.com for our upcoming events. Safari Bookshelf (safari.oreilly.com) is the premier online refer- ence library for programmers and IT professionals. Conduct searches across more than 1,000 books. Subscribers can zero in on answers to time-critical questions in a matter of seconds. Read the books on your Bookshelf from cover to cover or sim- ply flip to the page you need. Try it today with a free trial. LINUX COOKBOOK ™ Carla Schroder Beijing • Cambridge • Farnham • Köln • Paris • Sebastopol • Taipei • Tokyo Linux Cookbook™ by Carla Schroder Copyright © 2005 O’Reilly Media, Inc. All rights reserved. Printed in the United States of America. Published by O’Reilly Media, Inc., 1005 Gravenstein Highway North, Sebastopol, CA 95472. O’Reilly books may be purchased for educational, business, or sales promotional use.
    [Show full text]
  • Linux-FAQ.Pdf
    The Linux FAQ David C. Merrill david −AT− lupercalia.net 2003−09−19 Revision History Revision 1.20 2001−12−04 Revised by: rk Revision 2.0 2002−04−25 Revised by: dcm Some reorganization and markup changes. Revision 2.1 2003−05−19 Revised by: dcm Fairly complete reorganization and conversion to WikiText. Revision 2.1.1 2003−09−19 Revised by: dcm Minor corrections. Revision 2.1.2 2004−02−28 Revised by: dcm Minor corrections. This is the list of Frequently Asked Questions for Linux, the Free/Open Source UNIX−like operating system kernel that runs on many modern computer systems. The Linux FAQ Table of Contents 1. Introduction.....................................................................................................................................................1 1.1. About the FAQ..................................................................................................................................1 1.2. Asking Questions and Sending Comments.......................................................................................1 1.3. Authorship and Acknowledgments...................................................................................................1 1.4. Copyright and License......................................................................................................................2 1.5. Disclaimer.........................................................................................................................................2 2. General Information.......................................................................................................................................3
    [Show full text]
  • Automata-Based Formal Analysis and Verification of the Real-Time Linux Kernel
    Universidade Federal de Santa Catarina Scuola Superiore Sant'Anna Red Hat, Inc. Ph.D. in Automation and Systems Engineering Ph.D. in Emerging Digital Technologies - Embedded and Real-Time Systems R&D Platform - Core-Kernel - Real-Time Ph.D. Candidate: Daniel Bristot de Oliveira Ph.D. Supervisors: Tommaso Cucinotta Rômulo Silva de Oliveira Automata-based Formal Analysis and Verification of the Real-Time Linux Kernel Pisa 2020 ACKNOWLEDGEMENTS In the first place, I would like to thank my friends and family for supporting me during the development of this research. In particular, I would like to thank Alessandra de Oliveira for motivating me to pursue a Ph.D. degree since I was a teenager. But also for allowing me to use her computer, which was “my” first computer. I also would like to thank Bianca Cartacci, for the unconditional support in the hardest days of these last years. I would like to thank Red Hat, Inc. for allowing me to continue my academic research in concurrency with my professional life. I would also like to thank the pro- fessionals that motivated me to remain in the academy and supported this research, including Nelson Campaner, Steven Rostedt, Juri Lelli and Arnaldo Carvalho de Mello. In special, I would like to thank Clark Williams, not only for his effort in supporting my research at Red Hat but, mainly, for his adivises and personal support. I would like to thank my Ph.D. colleagues from UFSC and Sant’Anna, in special Karila Palma Silva, Marco Pagani, Paolo Pazzaglia and Daniel Casini, all the members of the Retis Lab, and the administrave staff of the TeCIP institute at Sant’Anna and the PPGEAS at UFSC for the support in the cotutela agreement, in special for Enio Snoeijer.
    [Show full text]
  • Gender Dimensions of Free and Open Source Development
    THE MAKING OF LABORER SUBJECTIVITY AND KNOWLEDGE IN THE INFORMATION INDUSTRY: GENDER DIMENSIONS OF FREE AND OPEN SOURCE DEVELOPMENT Yeon Ju Oh A Dissertation Submitted to the Graduate College of Bowling Green State University in partial fulfillment of the requirements for the degree of DOCTOR OF PHILOSOPHY August 2013 Committee: Radhika Gajjala, Advisor Deborah G. Wooldridge Graduate Faculty Representative Oliver Boyd-Barrett Victoria S. Ekstrand ii ABSTRACT Radhika Gajjala, Advisor This study examines female software developers as knowledge laborers with a special emphasis on free and open source software (FOSS) development. In examining female developers as knowledge laborers, this study focuses on both labor and knowledge. Women’s low participation in FOSS development is not an issue of recent years, but a consequence of women’s overall status in the computing field over the last three decades. In order to explicate women’s low participation in FOSS development, a broader historical and economic analysis is needed. Thus, this study explores the historical context of computer science education and industry in the 1980s since this is when the groundwork for FOSS development was laid. Furthermore, the power of cultural discourses that maintain and reinforce the gendered construction of FOSS development is discussed to unpack how the gendered construction is interrelated with the labor relations in the knowledge industry. In addition to the labor relations in FOSS development, this study attends to the knowledge produced by FOSS development. Knowledge gains importance as a sum of values of the knowledge producers. Source codes written by software developers turn into products that engage users with certain utility.
    [Show full text]
  • Building Embedded Linux Systems ,Roadmap.18084 Page Ii Wednesday, August 6, 2008 9:05 AM
    Building Embedded Linux Systems ,roadmap.18084 Page ii Wednesday, August 6, 2008 9:05 AM Other Linux resources from O’Reilly Related titles Designing Embedded Programming Embedded Hardware Systems Linux Device Drivers Running Linux Linux in a Nutshell Understanding the Linux Linux Network Adminis- Kernel trator’s Guide Linux Books linux.oreilly.com is a complete catalog of O’Reilly’s books on Resource Center Linux and Unix and related technologies, including sample chapters and code examples. ONLamp.com is the premier site for the open source web plat- form: Linux, Apache, MySQL, and either Perl, Python, or PHP. Conferences O’Reilly brings diverse innovators together to nurture the ideas that spark revolutionary industries. We specialize in document- ing the latest tools and systems, translating the innovator’s knowledge into useful skills for those in the trenches. Visit con- ferences.oreilly.com for our upcoming events. Safari Bookshelf (safari.oreilly.com) is the premier online refer- ence library for programmers and IT professionals. Conduct searches across more than 1,000 books. Subscribers can zero in on answers to time-critical questions in a matter of seconds. Read the books on your Bookshelf from cover to cover or sim- ply flip to the page you need. Try it today for free. main.title Page iii Monday, May 19, 2008 11:21 AM SECOND EDITION Building Embedded Linux SystemsTomcat ™ The Definitive Guide Karim Yaghmour, JonJason Masters, Brittain Gilad and Ben-Yossef, Ian F. Darwin and Philippe Gerum Beijing • Cambridge • Farnham • Köln • Sebastopol • Taipei • Tokyo Building Embedded Linux Systems, Second Edition by Karim Yaghmour, Jon Masters, Gilad Ben-Yossef, and Philippe Gerum Copyright © 2008 Karim Yaghmour and Jon Masters.
    [Show full text]
  • Enea® Linux User's Guide
    Enea® Linux User's Guide 4.0-docupdate1 Enea® Linux User's Guide Enea® Linux User's Guide Copyright Copyright © Enea Software AB 2014. This User Documentation consists of confidential information and is protected by Trade Secret Law. This notice of copyright does not indicate any actual or intended publication of this information. Except to the extent expressly stipulated in any software license agreement covering this User Documentation and/or corresponding software, no part of this User Documentation may be reproduced, transmitted, stored in a retrieval system, or translated, in any form or by any means, without the prior written permission of Enea Software AB. However, permission to print copies for personal use is hereby granted. Disclaimer The information in this User Documentation is subject to change without notice, and unless stipulated in any software license agreement covering this User Documentation and/or corresponding software, should not be construed as a commitment of Enea Software AB. Trademarks Enea®, Enea OSE®, and Polyhedra® are the registered trademarks of Enea AB and its subsidiaries. Enea OSE®ck, Enea OSE® Epsilon, Enea® Element, Enea® Optima, Enea® Linux, Enea® LINX, Enea® LWRT, Enea® Accelerator, Polyhedra® Flash DBMS, Polyhedra® Lite, Enea® dSPEED, Accelerating Network Convergence™, Device Software Optimized™, and Embedded for Leaders™ are unregistered trademarks of Enea AB or its subsidiaries. Any other company, product or service names mentioned in this document are the registered or unregistered trade- marks of their respective owner. Acknowledgements and Open Source License Conditions Information is found in the Release Information manual. © Enea Software AB 2014 4.0-docupdate1 ii Enea® Linux User's Guide Table of Contents 1 - Introduction ...............................................................................................................
    [Show full text]
  • With Hacks That Any User Can Follow, Linux Desktop Hacks Demonstrates How Easy It Is to Modify Linux to Suit Your Desires. the B
    Linux Desktop Hacks By Jono Bacon, Nicholas Petreley Publisher: O'Reilly Pub Date: March 2005 ISBN: 0-596-00911-9 Pages: 342 Table of • Contents • Index • Reviews Reader With hacks that any user can follow, Linux Desktop Hacks demonstrates • Reviews how easy it is to modify Linux to suit your desires. The book is packed with • Errata tips on customizing and improving the interface, boosting performance, • Academic administering your desktop, and generally making the most out of what X, KDE, Gnome, and the console have to offer. Linux Desktop Hacks By Jono Bacon, Nicholas Petreley Publisher: O'Reilly Pub Date: March 2005 ISBN: 0-596-00911-9 Pages: 342 Table of • Contents • Index • Reviews Reader • Reviews • Errata • Academic Copyright Credits About the Authors Contributors Acknowledgments Preface Why Linux Desktop Hacks? How to Use This Book How This Book Is Organized Conventions Used in This Book Using Code Examples How to Contact Us Got a Hack? Safari Enabled Chapter 1. Booting Linux Hacks 1-9 Hack 1. Give Your Computer the Boot Hack 2. Kill and Resurrect the Master Boot Record Hack 3. Bypass the Boot Manager Hack 4. Set a Bitmap Boot Screen for LILO Hack 5. Create Your Own LILO Boot Splash Hack 6. Display a GRUB Boot Splash Background Hack 7. Create a GRUB Boot Splash Background Hack 8. Jazz Up Your Debian System Boot Hack 9. Graphics on the Console Chapter 2. Console Hacks 10-15 Hack 10. Redefine Keyboard Actions Hack 11. Energize Your Console with Macro Music Magic Hack 12. Take a Screenshot from the Command Line Hack 13.
    [Show full text]
  • Performance Analysis Using Great Performance Tools and Linux Trace Toolkit Next Generation
    Universität Hamburg Department Informatik Scientific Computing, WR Performance analysis using Great Performance Tools and Linux Trace Toolkit next generation Seminar Paper Seminar: Performance analysis in Linux Heye Vöcking Matr.Nr. 6139373 [email protected] Tutor: Julian Kunkel April 3, 2012 Heye Vöcking Performance analysis using GPerfTools and LTTng Contents 1 Introduction 3 1.1 Overview . 3 1.2 Motivation . 3 1.3 Related Tools . 3 1.3.1 Tools discussed in this work . 4 2 Related Work 4 3 History 4 4 Great Performance Tools in depth 4 4.1 Field of application . 5 4.1.1 Supported operating systems . 5 4.1.2 Supported programming languages . 5 4.2 Install / Setup . 5 4.3 How it works and how to use it . 5 4.3.1 CPU Profiler . 5 4.3.2 TCMalloc . 6 4.3.3 Heap Leak Checker . 6 4.3.4 Heap Profiler . 7 4.3.5 Profiler Insides . 8 4.4 Post-Processing with pprof . 8 4.4.1 Analyzing CPU Profiles . 8 4.4.2 Analyzing Memory Leaks . 10 4.4.3 Analyzing Heap Profiles . 10 5 Linux Trace Toolkit next generation in depth 10 5.1 Field of application . 10 5.1.1 Supported operating systems . 10 5.1.2 Supported architectures . 10 5.1.3 Supported programming languages . 10 5.2 Install / Setup . 11 5.3 How it works . 11 5.3.1 Tracing . 12 5.3.2 Tracepoints . 12 5.3.3 Probes, Trace Sessions, and Channels . 13 5.3.4 Storage and Trace Modes . 13 5.4 Post-Processing and Analysis .
    [Show full text]
  • Automata-Based Formal Analysis and Verification of the Real-Time Linux
    UNIVERSIDADE FEDERAL DE SANTA CATARINA PROGRAMA DE PÓS-GRADUAÇÃO EM ENGENHARIA DE AUTOMAÇÃO E SISTEMAS Daniel Bristot de Oliveira Automata-based Formal Analysis and Verification of the Real-Time Linux Kernel Pisa 2020 Daniel Bristot de Oliveira Automata-based Formal Analysis and Verification of the Real-Time Linux Kernel Tese submetida ao Programa de Pós-Graduação em Engenharia de Automação e Sistemas da Univer- sidade Federal de Santa Catarina para a obtenção do tí- tulo de Doutor em Engenharia de Automação e Sis- temas. Orientador: Prof. Rômulo Silva de Oliveira, Dr. Coorientador: Prof. Tommaso Cucinotta, Dr. Pisa 2020 Ficha de identificação da obra elaborada pelo autor, através do Programa de Geração Automática da Biblioteca Universitária da UFSC. Bristot de Oliveira, Daniel Automata-based Formal Analysis and Verification of the Real-Time Linux Kernel / Daniel Bristot de Oliveira ; orientador, Rômulo Silva de Oliveira, coorientador, Tommaso Cucinotta, 2020. 144 p. Tese (doutorado) - Universidade Federal de Santa Catarina, Centro Tecnológico, Programa de Pós-Graduação em Engenharia de Automação e Sistemas, Florianópolis, 2020. Inclui referências. 1. Engenharia de Automação e Sistemas. 2. Tempo Real. 3. Kernel do Linux. 4. Metodos Formais. 5. Verificação. I. Silva de Oliveira, Rômulo. II. Cucinotta, Tommaso. III. Universidade Federal de Santa Catarina. Programa de Pós Graduação em Engenharia de Automação e Sistemas. IV. Título. Daniel Bristot de Oliveira Automata-based Formal Analysis and Verification of the Real-Time Linux Kernel O presente trabalho em nível de doutorado foi avaliado e aprovado por banca examinadora composta pelos seguintes membros: Prof. Marco Di Natale, Dr. Scuola Superiore Sant’Anna Prof. Giuseppe Lipari, Dr.
    [Show full text]
  • Linuxcon Europe 2011
    LinuxCon Europe 2011 LTTng 2.0 : Application, Library and Kernel tracing within your Linux distribution. E-mail: [email protected] Mathieu Desnoyers October 26th, 2011 1 > Buzzword compliant ! ● LTT: Linux Trace Toolkit ● “ng” : Next Generation ● 2.0 ! LTTng 2.0 ! ● All we miss is a recursive acronym. ;) Mathieu Desnoyers October 26th, 2011 2 > Presenter ● Mathieu Desnoyers ● EfficiOS Inc. ● http://www.efficios.com ● Author/Maintainer of ● LTTng, LTTng-UST, Babeltrace, LTTV, Userspace RCU Mathieu Desnoyers October 26th, 2011 3 > Benefits of low-impact tracing in a multi-core world ● Understanding interaction between – Kernel – Libraries – Applications – Virtual Machines ● Debugging ● Performance tuning ● Monitoring Mathieu Desnoyers October 26th, 2011 4 > Tracing use-cases ● Telecom – Operator, engineer tracing systems concurrently with different instrumentation sets. – In development and maintenance phases. ● Embedded – System development, maintenance of deployed systems. ● Server/Desktop software – Qemu/KVM, MariaDB. Mathieu Desnoyers October 26th, 2011 5 > Why do we need a LTTng 2.0 ? ● Need more flexible trace data layout format – Introduce Common Trace Format (CTF) ● Introduction of user-space tracing (UST) – Leverage common control infrastructure for kernel and user-space tracing – Simplification of the kernel-level infrastructure ● Need more flexible ring buffer – Snapshot, mmap and splice, global and per- cpu, kernel and user-space, configurable crash dump support. Mathieu Desnoyers October 26th, 2011 6 > LTTng 2.0
    [Show full text]
  • The Lttng Tracer: a Low Impact Performance and Behavior Monitor for GNU/Linux
    The LTTng tracer: A low impact performance and behavior monitor for GNU/Linux Mathieu Desnoyers Michel R. Dagenais École Polytechnique de Montréal École Polytechnique de Montréal [email protected] [email protected] Abstract presented. The approach taken to allow nested types, variable fields and dynamic alignment of data in the trace buffers will be revealed. It will Efficient tracing of system-wide execution, show the mechanisms deployed to facilitate use allowing integrated analysis of both kernel and extension of this tool by adding custom in- space and user space, is something difficult to strumentation and analysis involving kernel, li- achieve. The following article will present you braries and user space programs. a new tracer core, Linux Trace Toolkit Next Generation (LTTng), that has taken over the It will also introduce LTTng’s trace analyzer previous version known as LTT. It has the same and graphical viewer counterpart: Linux Trace goals of low system disturbance and architec- Toolkit Viewer (LTTV). The latter implements ture independance while being fully reentrant, extensible analysis of the trace information scalable, precise, extensible, modular and easy through collaborating text and graphical plu- to use. For instance, LTTng allows tracepoints gins.1 It can simultaneously display multi- in NMI code, multiple simultaneous traces and ple multi-GBytes traces of multi-processor sys- a flight recorder mode. LTTng reuses and en- tems. hances the existing LTT instrumentation and RelayFS. This paper will focus on the approaches taken 1 Tracing goals by LTTng to fulfill these goals. It will present the modular architecture of the project. It With the increasing complexity of newer com- will then explain how NMI reentrancy requires puter systems, the overall performance of appli- atomic operations for writing and RCU lists for cations often depends on a combination of sev- tracing behavior control.
    [Show full text]