TotalView® User Guide Version 8.10 Copyright © 2010-2012 by Rogue Wave Software, Inc. All rights reserved. Copyright © 2007-2009 by TotalView Technologies, LLC Copyright © 1998–2007 by Etnus LLC. All rights reserved. Copyright © 1996–1998 by Dolphin Interconnect Solutions, Inc. Copyright © 1993–1996 by BBN Systems and Technologies, a division of BBN Corporation. No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, electronic, mechanical, pho- tocopying, recording, or otherwise without the prior written permission of Rogue Wave Software, Inc. ("Rogue Wave"). Use, duplication, or disclosure by the Government is subject to restrictions as set forth in subparagraph (c)(1)(ii) of the Rights in Technical Data and Computer Software clause at DFARS 252.227-7013. Rogue Wave has prepared this manual for the exclusive use of its custom- ers, personnel, and licensees. The information in this manual is subject to change without notice, and should not be construed as a commitment by Rogue Wave. Rogue Wave assumes no responsibility for any errors that appear in this document. TotalView and TotalView Technologies are registered trademarks of Rogue Wave Software, Inc. TVD is a trademark of Rogue Wave. Rogue Wave uses a modified version of the Microline widget library. Under the terms of its license, you are entitled to use these modifications. The source code is available at: http://www.roguewave.com/support/knowledge-base.aspx. All other brand names are the trademarks of their respective holders. ii Contents About This Guide 1 Content Organization 1 TotalView Family Differences 3 Using the CLI 4 Audience 5 Conventions 6 TotalView Documentation 7 Contacting Us 9 Part 1: Introduction 11 1 A Tour of TotalView 13 Basic Debugging with TotalView 13 TotalView Action Points vs. Print Statements 14 Examining Data 16 Simplifying Array Display 17 Seeing Groups of Variables 19 Setting Watchpoints 19 Debugging Multi-process and Multi-threaded Programs 20 Program Using Almost Any Execution Model 21 Supporting Multi-process and Multi-threaded Programs 21 Using Groups and Barriers 23 Memory Debugging 24 Introducing the CLI 24 What’s Next 24 2 About Threads, Processes, and Groups 25 A Couple of Processes 26 Threads 28 Complicated Programming Models 29 Types of Threads 31 Organizing Chaos 34 How TotalView Creates Groups 38 Simplifying What You’re Debugging 42 TotalView User Guide version 8.10 iii Part 2: Setting Up 45 3 Accessing TotalView Remotely 47 Using Remote Display 47 Remote Display Supported Platforms 47 Remote Display Components 47 Installing the Client 48 Installing on Linux 48 Installing on Microsoft Windows 49 Installing on Apple Mac OS X Intel 49 Sample Session 50 Advanced Options 52 Naming Intermediate Hosts 53 Submitting a Job to a Batch Queuing System 54 Setting Up Your Systems and Security 54 Session Profile Management 55 Sharing Profiles 55 Remote Display Server and Viewer 55 Batch Scripts 56 tv_PBS.csh Script 56 tv_LoadLeveler.csh Script 57 Remote Display Commands 58 Session Profiles Area 58 Remote Host and Routing Area 58 Product Area 60 Using the Submit to Batch Queuing System Entries 61 File > Profile > Save 61 File > Profile > Delete 62 File > Profile > Import 62 File > Profile > Export 62 File > Exit 62 4 Setting Up a Debugging Session 63 Compiling Programs 64 Using File Extensions 64 Starting TotalView 65 Initializing TotalView 68 Exiting from TotalView 70 Loading Programs 71 Loading Programs Using the GUI 71 Loading Programs Using the CLI 73 Attaching to Processes 73 Detaching from Processes 74 Examining Core Files 75 Setting Command-line Arguments and Environment Variables 77 Altering Standard I/O 78 Viewing Process and Thread States 79 Seeing Attached Process States 80 Seeing Unattached Process States 80 Handling Signals 82 iv Contents Setting Search Paths 85 Setting Startup Parameters 88 Setting Preferences 89 Setting Preferences, Options, and X Resources 95 5 Setting Up Remote Debugging Sessions 97 About Remote Debugging 97 Platform Issues when Remote Debugging 98 Automatically Launching a Process on a Remote Server 100 Troubleshooting Server Autolaunch 101 Changing the Remote Shell Command 101 Changing Arguments 102 Autolaunching Sequence 102 Starting the TotalView Server Manually 104 TotalView Server Launch Options and Commands 106 Server Launch Options 106 Setting Single-Process Server Launch Options 106 Setting Bulk Launch Window Options 107 Customizing Server Launch Commands 109 Setting the Single-Process Server Launch Command 109 Setting the Bulk Server Launch Command 110 Debugging Over a Serial Line 113 Starting the TotalView Debugger Server 113 6 Setting Up MPI Debugging Sessions 115 Debugging MPI Programs 116 Starting MPI Programs 116 Starting MPI Programs Using File > New Program 116 Debugging MPICH Applications 118 Starting TotalView on an MPICH Job 118 Attaching to an MPICH Job 119 Using MPICH P4 procgroup Files 121 Debugging MPICH2 Applications 122 Downloading and Configuring MPICH2 122 Starting TotalView Debugging on an MPICH2 Hydra Job 122 Starting TotalView Debugging on an MPICH2 MPD Job 123 Starting the MPI MPD Job with MPD Process Manager 123 Starting an MPICH2 MPD Job 123 Starting MPI Issues 125 MPI Rank Display 126 Displaying the Message Queue Graph Window 127 Displaying the Message Queue 130 About the Message Queue Display 130 Using Message Operations 131 Diving on MPI Processes 131 Diving on MPI Buffers 131 About Pending Receive Operations 132 About Unexpected Messages 132 About Pending Send Operations 132 Debugging Cray MPI Applications 133 Debugging IBM MPI Parallel Environment (PE) Applications 134 TotalView User Guide version 8.10 v Preparing to Debug a PE Application 134 Using Switch-Based Communications 134 Performing a Remote Login 135 Setting Timeouts 135 Starting TotalView on a PE Program 135 Setting Breakpoints 135 Starting Parallel Tasks 136 Attaching to a PE Job 136 Attaching from a Node Running poe 136 Attaching from a Node Not Running poe 136 Debugging IBM Blue Gene Applications 138 Debugging LAM/MPI Applications 139 Debugging QSW RMS Applications 140 Starting TotalView on an RMS Job 140 Attaching to an RMS Job 140 Debugging SiCortex MPI Applications 141 Debugging SGI MPI Applications 142 Starting TotalView on an SGI MPI Job 142 Attaching to an SGI MPI Job 142 Using ReplayEngine with SGI MPI 143 Debugging Sun MPI Applications 144 Attaching to a Sun MPI Job 144 Using ReplayEngine with Infiniband MPIs 145 Debugging Parallel Applications Tips 147 Attaching to Processes Tips 147 Parallel Debugging Tips 150 MPICH Debugging Tips 151 IBM PE Debugging Tips 152 7 Setting Up Parallel Debugging Sessions 153 Debugging OpenMP Applications 154 Debugging OpenMP Programs 154 About TotalView OpenMP Features 155 About OpenMP Platform Differences 156 Viewing OpenMP Private and Shared Variables 157 Viewing OpenMP THREADPRIVATE Common Blocks 158 Viewing the OpenMP Stack Parent Token Line 159 Using SLURM 160 Debugging IBM Cell Broadband Engine Programs 161 The PPU 162 The SPU 162 Cell Programing 162 PPU and SPU Executable Organization 163 PPU and SPU Executable Naming 163 Thread IDs 164 Breakpoints 164 Registers, Unions, and Casting 165 Debugging Cray XT Applications 167 Cray XT Catamount 167 Configuring Cray XT for TotalView 168 Using TotalView with your Cray XT System 169 vi Contents Cray XT CNL 169 Debugging SiCortex Applications 170 Installation Notes 170 Using TotalView on SiCortex 170 MPI Debugging 171 Debugging Global Arrays Applications 172 Debugging PVM (Parallel Virtual Machine) Applications 174 Supporting Multiple Sessions 174 Setting Up ORNL PVM Debugging 174 Starting an ORNL PVM Session 175 Automatically Acquiring PVM Processes 176 Attaching to PVM Tasks 176 About Reserved Message Tags 178 Cleaning Up Processes 178 Debugging Shared Memory (SHMEM) Code 179 Debugging UPC Programs 180 Invoking TotalView 180 Viewing Shared Objects 180 Displaying Pointer to Shared Variables 182 Part 3: Using the GUI 185 8 Using TotalView Windows 187 Using Mouse Buttons 188 Using the Root Window 189 Viewing Remote Processes 190 Viewing Data Linearly or Hierarchically 191 Controlling the Display of Processes and Threads 192 Using the Process Window 193 Viewing the Assembler Version of Your Code 196 Diving into Objects 199 Resizing and Positioning Windows and Dialog Boxes 202 Editing Text 203 Saving the Contents of Windows 204 9 Visualizing Programs and Data 205 Displaying Call Graphs 206 Parallel Backtrace View 208 Visualizing Array Data 210 Command Summary 210 How the Visualizer Works 211 Viewing Data Types in the Visualizer 211 Viewing Data 212 Visualizing Data Manually 212 Using the Visualizer 213 Using Dataset Window Commands 213 Using View Window Commands 214 Using the Graph Window 215 Displaying Graph Views 216 Using the Surface Window 217 Displaying Surface Views 218 TotalView User Guide version 8.10 vii Manipulating Surface Data 220 Visualizing Data Programmatically 221 Launching the Visualizer from the Command Line 222 Configuring TotalView to Launch the Visualizer 222 Setting the Visualizer Launch Command 223 You can preset the Visualizer launch options by setting X resources. Adapting a Third Party Visualizer 224 Part 4: Using the CLI 227 10 Using the Command Line Interface (CLI) 229 About the Tcl and the CLI 230 About The CLI and TotalView 230 Using the CLI Interface 231 Starting the CLI 232 Startup Example 232 Starting Your Program 233 About CLI Output 236 ‘more’ Processing 237 Using Command Arguments 238 Using Namespaces 239 About the CLI