Kernel-Level Measurement for Integrated Parallel Performance Views: the KTAU Project Aroon Nataraj Allen D. Malony Sameer Shende Alan Morris Department of Computer and Information Science University of Oregon, Eugene, OR, USA E-mail: anataraj,malony,sameer,amorris @cs.uoregon.edu { } Abstract and framework to observe OS actions relative to applica- tion activities and performance has yet to be fully devel- The effect of the operating system on application perfor- oped. Minimally, such an approach will require OS kernel mance is an increasingly important consideration in high performance monitoring. performance computing. OS kernel measurement is key to The OS influences application performance both directly understanding the performance influences and the interre- and indirectly. Actions the OS takes independent of the lationship of system and user-level performance factors. application’s execution have indirect effects on its perfor- The KTAU (Kernel TAU) methodology and Linux-based mance. In contrast, we say that the OS directly influences framework provides parallel kernel performance measure- application performance when its actions are a result of or in ment from both a kernel-wide and process-centric perspec- support of application execution. This motivates two differ- tive. The first characterizes overall aggregate kernel per- ent monitoring perspectives of OS performance in order to formance for the entire system. The second characterizes understand the effects. One perspective is to view the entire kernel performance when it runs in the context of a partic- kernel operation as a whole, aggregating performance data ular process. KTAU extends the TAU performance system from all active processes in the system and including the with kernel-level monitoring, while leveraging TAU’s mea- activities of the OS when servicing system-calls made by surement and analysis capabilities. We explain the rational applications as well as activities not directly related to ap- and motivations behind our approach, describe the KTAU plications (such as servicing hardware interrupts or keeping design and implementation, and show working examples on time). We will refer to this as the kernel-wide perspective, multiple platforms demonstrating the versatility of KTAU in which is helpful in understanding OS behavior and in iden- integrated system / application monitoring. tifying and removing kernel hot spots. However, this view does not provide comprehensive insight into what parts of a particular application spend time inside the kernel and why. Another way to view OS performance is within the con- 1. Introduction text of a specific application’s execution. Application per- formance is affected by the interaction of user-space behav- The performance of parallel applications on high- ior with the OS, as well as what is going on in the rest of the performance computing (HPC) systems is a consequence of system. By looking at how the OS behaves in the context the user-level execution of the application code and system- of individual processes (versus aggregate performance) we level (kernel) operations that occur while the application is can provide a detailed view of the interactions between pro- running. As HPC systems evolve towards ever larger and grams, daemons, and system services. This process-centric more integrated parallel environments, the ability to ob- perspective is helpful in tuning the OS for a specific work- serve all performance factors, their relative contributions load, tuning the application to better conform to the OS con- and interrelationship, will become important to comprehen- figuration, and in exposing the source of performance prob- sive performance understanding. Unfortunately, most par- lems (in the OS or the application). allel performance tools operate only at the user-level, leav- Both the kernel-wide and process-centric perspectives ing the kernel-level performance artifacts obscure. OS fac- are important in OS performance measurement and analysis tors causing application performance bottlenecks, such as on HPC systems. The challenge is how to support both per- those demonstrated in [12, 21], are difficult to assess by spectives while providing a monitoring infrastructure that user-level measurements alone. An integrated methodology gives detailed visibility of kernel actions and is easy to use 1-4244-0328-6/06/$20.00 c 2006 IEEE. ! by different tools. For example, the interactions between While KernInst incurs overhead only for dynamically in- applications and the OS mainly occur through five different strumented events, trampolining adds delay and there is a mechanisms: system-calls, exceptions, interrupts (hard and cost in altering instrumentation during execution. KernInst, soft), scheduling, and signals. It is important to understand by itself, does not support merging user and kernel per- all forms of interactions as the application performance is formance data. In Dtrace, application-level instrumenta- influenced by each. Of these, system calls are the easiest to tion also traps into the kernel where the kernel-logging fa- observe as they are synchronous with respect to the applica- cility logs both application and kernel performance data. tion. Also, importantly, system calls are serviced inside the However, because trapping into the kernel can be costly, kernel relative to the context of the calling process, allowing using DTrace may not be a scalable option for combined greater flexibility in how these actions are monitored. On user/kernel performance measurement of HPC codes. the other hand, interrupts, scheduling, and exceptions occur Static Source Instrumentation: In contrast to the dy- asynchronously and usually in the interrupt context, where namic instrumentation in KernInst and DTrace, source in- the kernel-level facilities and state constrain monitoring op- strumentation tools operate on static source code. Linux tions and make access to user-space performance data dif- Trace Toolkit (LTT) [14], based on kernel source instru- ficult. One problem is observing all the different program- mentation, is a Kernel Tracing tool. While LTT can OS interactions and correlating performance across the user provide rich information about kernel function, its use / OS boundary. Another is observing these interactions of heavy-weight timing (gettimeofday) has high overhead across a parallel environment and collectively gathering and and low precision, and its use of a single trace buffer re- analyzing the performance data. Support for this needs to quires global locks. Some of the limitations are being be explicit, especially for post-processing, analysis, and vi- addressed [11]. Source-instrumented kernels also support sualization of parallel data. tools to measure performance in the form of profile statis- Our approach is the development of a new Linux Ker- tics. SGI’s KernProf [2] (under callgraph modes) is an ex- nel performance measurement facility called KTAU, which ample of a Kernel Profiling tool.. By compiling the Linux extends the TAU [3] performance system with kernel-level Kernel with the -pg option to gcc, every function is instru- monitoring. KTAU allows both a kernel-wide and process- mented with code to track call counts and parent/child rela- centric perspective of OS performance to be obtained to tionships. As every function is being profiled the overhead show indirect and direct influences on application perfor- at runtime can be significant. Neither tool provides exten- mance. KTAU provides a lightweight profiling facility and a sive support for online merging of user and kernel data. tracing capability that can generate a detailed event log. We Statistical Sampling Tools: In contrast to instru- describe KTAU’s design and architecture in Section 4. We mentation that modifies the kernel code (source or exe- demonstrate KTAU’s features and versatility through work- cutable), statistical sampling tools periodically record Pro- ing examples on different platforms in Section 5. We be- gram Counter (PC) values to build histograms of how per- gin with a discussion of related work in Section 2 as back- formance is distributed. This approach can observe both ground to the problem. The paper concludes with final re- user-mode and kernel-mode operation across the system marks and future directions of KTAU in Section 6. including libraries, applications, kernel-image and kernel- modules on a per processor basis. For example, Opro- 2. Background Related Work file [1] is meant to be a continuous profiler for Linux, mean- ing it is always turned on. Its shortcomings include an in- To better understand the design and functionality of ability to provide online information (as it performs a type KTAU, we first review related research and technology in of partial tracing) and the requirement of a daemon. Other OS monitoring. Tools operating at the kernel level only and issues stem from the inaccuracy of sampling based profiles. and those that attempt to combine user/kernel performance Merged User-Kernel Performance Analysis: To better data are most closely related to KTAU’s objectives. The understand program-OS interaction, localize bottlenecks in different instrumentation and measurement techniques used the program/OS execution stack and identify intrusive in- are the primary distinguishing factors between the existing terrupt/scheduling effects, it is important to associate ker- approaches. For lack of space, we show only a few candi- nel actions with specific application processes. DeBox [24] date tools in each category.