Universität Hamburg Department Informatik Scientific Computing, WR Performance analysis using Great Performance Tools and Linux Trace Toolkit next generation Seminar Paper Seminar: Performance analysis in Linux Heye Vöcking Matr.Nr. 6139373 [email protected] Tutor: Julian Kunkel April 3, 2012 Heye Vöcking Performance analysis using GPerfTools and LTTng Contents 1 Introduction 3 1.1 Overview . 3 1.2 Motivation . 3 1.3 Related Tools . 3 1.3.1 Tools discussed in this work . 4 2 Related Work 4 3 History 4 4 Great Performance Tools in depth 4 4.1 Field of application . 5 4.1.1 Supported operating systems . 5 4.1.2 Supported programming languages . 5 4.2 Install / Setup . 5 4.3 How it works and how to use it . 5 4.3.1 CPU Profiler . 5 4.3.2 TCMalloc . 6 4.3.3 Heap Leak Checker . 6 4.3.4 Heap Profiler . 7 4.3.5 Profiler Insides . 8 4.4 Post-Processing with pprof . 8 4.4.1 Analyzing CPU Profiles . 8 4.4.2 Analyzing Memory Leaks . 10 4.4.3 Analyzing Heap Profiles . 10 5 Linux Trace Toolkit next generation in depth 10 5.1 Field of application . 10 5.1.1 Supported operating systems . 10 5.1.2 Supported architectures . 10 5.1.3 Supported programming languages . 10 5.2 Install / Setup . 11 5.3 How it works . 11 5.3.1 Tracing . 12 5.3.2 Tracepoints . 12 5.3.3 Probes, Trace Sessions, and Channels . 13 5.3.4 Storage and Trace Modes . 13 5.4 Post-Processing and Analysis . 14 5.4.1 Babeltrace . 14 5.4.2 LTT Viewer . 14 5.4.3 IDE integration . 14 5.5 Performance Impact . 15 6 Areas of Application 15 7 Summary 15 1 of 17 Heye Vöcking Performance analysis using GPerfTools and LTTng 8 Conclusion 16 9 Outlook 16 2 of 17 Heye Vöcking Performance analysis using GPerfTools and LTTng Abstract Todays systems are getting more and more complex. Especially through the trend to software running on multi-core setups. In order to find bugs and improve performance of the software we develop, we need good tools for tracing the programs execution efficiently. By monitoring, recording, and analyzing the programs behavior we can diagnose problems, tune for more per- formance, and understand the interaction of our software with libraries and the operating system better. In this work we want to present the Great Performance Tools and the Linux Trace Toolkit next generation. 1 Introduction the advantage of multi-core systems. Effi- ciency is a crucial subject because it saves When we develop software we often en- time and energy during runtime. Another counter bugs or code that performs poorly. important part is software safety. To avoid To debug and tune our code we can ob- dead locks, segmentation faults, or mem- serve the programs execution. There are ory the programmer needs a good under- several more or less efficient ways to accom- standing of the software he writes and the plish this. Tracing is one of them which can libraries he uses. In order to Here we want be performed using LTTng, another way is to show what tools can help developers to profiling which is possible with gperftools. write efficient code. 1.1 Overview 1.3 Related Tools At first we will mention some tools avail- gprof (1982) gprof extended the 1979 re- able for tracing as well as discuss and com- leased Unix tool “prof”, by providing pare the Great Performance Tools (section a complete call graph analysis. [gpr12] 4) and the Linux Trace Toolkit next genera- tion (section 5) in detail. We will show what EDGE (1999) The Mentor Embedded features they offer, describe how to install EDGE Developer Suite is an IDE built and integrate them, show how to use them, upon the Eclipse framework and pro- and give an insight of how they work. The vides tools that give an insight into areas of application are discussed in section the kernel and a debugger as well as 6. some other features. [Men99] QNX Momentics Tool Suite (2001) 1.2 Motivation This tool suite can be used to get There is more to developing software than a view of real-time interactions and just to make it work. Even though the pro- memory profiles as well as porting cessing power of computers is still rising a software from single- to multi-core developer should write efficient code and use systems. [QNX01] 3 of 17 Heye Vöcking Performance analysis using GPerfTools and LTTng Valgrind (2002) Originally designed to be 3 History a free memory debugging toll for Linux on x86, Valgrind is now a Performance analysis is no recent problem. generic framework for creating dy- Tools for analyzing performance have been namic analysis tools. [Val12] around since the early 1970s. They where usually based on timer interrupts to de- tect “hot spots” in the program code. The 1.3.1 Tools discussed in this work first profiler driven analysis tool was prof in 1979. In 1994 instrumentation by inserting In this work we are going to discuss profiling code in programs during compile and compare the Great Performance Tools time came up with the ATOM platform. (GPerfTools) and the Linux Trace Toolkit Instrumentation of the Linux kernel using next generation (LTTng). Both projects a method called “Kprobes” is implemented where started in 2005, available for free on since July 2002 (Kernel 2.5.26) and still the Internet, and by now both are commu- present today (Kernel 3.3). In early 2008 nity driven. “Kernel Markers” where developed to re- place Kprobes with no success, because the so called “Tracepoints” found their way into the Kernel not even a year later and man- 2 Related Work aged to replace the Kernel Markers com- pletely since December 2009. Tracepoints Peter Smith manages to achieve a 30% are still popular and used for example in speed gain by fixing poorly performing code Systemtap and LTTng which is discussed in that he located by profiling his 3D applica- section 5. [Mö12] tion with GPerfTools and kCachegrind as described in his work [Smi11]. In his article [Tou11] Toupin describes 4 Great Performance Tools how to use tracing to diagnose or monitor in depth single and multicore systems. The author and maintainer of LTTng, The Google Performance Tools project was Mathieu Desnoyers, has written his PhD. launched in March 2005 [SR08]. In 2011 thesis [Des09] on software tracing with LT- Craig Silverstein, in the gperftools project Tng better known as “csilvers”, who was the Romik Guha Anjoy and Soumya Kanti main developer, decided to step down. As Chakraborty evaluated in their Master The- a part of that Google decided to make the sis [RGA10] the efficiency of LTTng for project completely community run, there- employing it within the telecommunication fore with the beginning of version num- company Ericsson ber 2 (released in February 2012), the tools 4 of 17 Heye Vöcking Performance analysis using GPerfTools and LTTng changed their name from “google-perftools” allocation distributed under BSD license. to “gperftools”, where the “g” now stands They offer: for “great”. The gperftools are now main- 1. cpu_profiler, a profiler for monitor- tained by David Chappelle who was the only ing the performance of functions other active developer in the past. [CS12] 2. tcmalloc, a thread caching memory allocator as a fast replacement for 4.1 Field of application 3. heap_checker, a heap checker for leak 4.1.1 Supported operating systems detection The gperftools can be used in Linux, Mac, 4. heap_profiler, a profiler for moni- as well as in Windows environments. toring heap allocations CMalloc 4.1.2 Supported programming We will discuss these below. languages 4.3.1 CPU Profiler Supported programming languages are C The CPU profiler collects information dur- and C++. But the gperftools can be used ing runtime, namely the number of func- with any language that can call C code. tion calls with caller and callee methods and classes. There are multiple ways to use the 4.2 Install / Setup CPU profiler: Setup in a Linux environment is fairly 1. Through linking -lprofiler into straight forward. You should be able to your executable. find the corresponding packages using your 2. Through $LD_PRELOAD by us- package manager. In Arch Linux for ex- ing % env LD_PRELOAD=”/usr/lib/ ample you can use a tool called yaourt libprofiler.so” /path/to/binary. to install packages from the AUR (Arch But doing so is not recommended. User Repository). The command “yaourt - S google-perftools” will download, compile, Note that CPU profiling has to be and install the gperftools as well as all re- turned on manually before execution. quired packages. Additionally you can in- This is done by defining the environ- stall perl5, dot, and gv if you want to visu- ment variable $CPUPROFILE to the file alize the collected data with pprof. you want to save the collected informa- tion to. Surround the code you want 4.3 How it works and how to use to profile with ProfilerStart(”profile name”) and ProfilerStop(). Sev- it eral more advanced functions are The gperftools are a set of open-source tools available like ProfilerFlush() and for profiling, leak checking, and memory ProfilerStartWithOptions(). 5 of 17 Heye Vöcking Performance analysis using GPerfTools and LTTng Using the environment variables tion. So it records every allocation and $CPUPROFILE_FREQUENCY checks on exit if all allocated memory has and been freed. If not all memory has been freed $CPUPROFILE_REALTIME it reports a leak. you can control the behavior of the CPU In order to run a program with the profiler during runtime.