Front cover IBM System Blue Gene Solution: Performance Analysis Tools PAPI installation and usage Tools to visualize and analyze your performance data I/O node and GPFS performance tuning Gary Lakner I-Hsin Chung Dr. Guojing Cong Scott Fadden Nicholas Goracke David Klepacki Jeffrey Lien Dr. Christoph Pospiech Seetharami R. Seelam Hui-Fang Wen ibm.com/redbooks Redpaper International Technical Support Organization IBM System Blue Gene Solution: Performance Analysis Tools November 2008 REDP-4256-01 Note: Before using this information and the product it supports, read the information in “Notices” on page vii. Second Edition (November 2008) This edition applies to Version 1, Release 2, Modification 0 of IBM System Blue Gene/P Solution (product number 5733-BGP). © Copyright International Business Machines Corporation 2007, 2008. All rights reserved. Note to U.S. Government Users Restricted Rights -- Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp. Contents Notices . vii Trademarks . viii Preface . ix The team that wrote this paper . ix Become a published author . .x Comments welcome. xi Summary of changes. xiii November 2008, Second Edition . xiii Part 1. Performance Application Programming Interface . 1 Chapter 1. Implementing the Performance Application Programming Interface on Blue Gene/P . 3 1.1 Installing PAPI. 4 1.2 Modifications to PAPI for Blue Gene/P . 5 1.2.1 PAPI_read. 5 1.2.2 papi_bgp_read . 6 1.2.3 PAPI presets . 6 1.2.4 Limitations. 14 Chapter 2. PAPI usage . 15 2.1 Usage example . 16 2.1.1 Headers . 17 2.1.2 Flow of events. 17 2.1.3 Running the code . 18 2.2 Suggestions . 19 2.2.1 Using PAPI preset values . 19 2.2.2 Running in Dual or Virtual Node mode . 20 2.2.3 Additional methods . 22 Chapter 3. Automatically Available Performance Counters . 29 3.1 Automatically Available Performance Counters for MPI applications . 30 3.1.1 Enabling Automatically Available Performance Counters. 30 3.1.2 Compiling with Automatically Available Performance Counters . 30 3.1.3 How performance counters work . 30 3.1.4 Changing runtime parameters. 32 3.2 Performance counters in HTC Mode. 32 3.2.1 Modifications to application code . 33 3.2.2 Compiling the application . 34 3.2.3 Running the application . 34 3.2.4 Caveats when running in HTC mode . 35 Part 2. High Performance Computing Toolkit. 37 Chapter 4. MPI Profiler and Tracer. 39 4.1 System and software requirements. 40 4.2 Compiling and linking . 40 4.3 Environment variables. 41 © Copyright IBM Corp. 2007, 2008. All rights reserved. iii 4.3.1 TRACE_ALL_EVENTS . 41 4.3.2 TRACE_ALL_TASKS . 42 4.3.3 TRACE_MAX_RANK . 42 4.3.4 TRACEBACK_LEVEL. 42 4.3.5 SWAP_BYTES . 42 4.3.6 TRACE_SEND_PATTERN (Blue Gene/L and Blue Gene/P only) . 43 4.4 Output . 43 4.4.1 Plain text file . 44 4.4.2 The VIZ file . 47 4.4.3 Trace file . 48 4.5 Configuration. 48 4.5.1 Configuration functions . 48 4.5.2 Data structure . 49 4.5.3 Utility functions . 51 4.6 Related issues. 54 4.6.1 Overhead . 54 4.6.2 Multithreading . 54 Chapter 5. CPU profiling using Xprofiler . 55 5.1 Starting Xprofiler . 56 5.2 Understanding the Xprofiler display . 58 5.2.1 Xprofiler main menus . 59 5.2.2 Elements of the function call tree . 60 5.2.3 Manipulating the Xprofiler display . 62 5.3 Getting performance data for your application . 69 Chapter 6. Hardware Performance Monitoring . 77 6.1 HPM . 78 6.2 Events and groups . ..