Hands-on Workshop on How To Debug Codes at the Institute
H. Birali Runesha, Shuxia Zhang and Ben Lynch
(612) 626 0802 (help) [email protected]
October 13, 2005
Outline
• Debuggers at the Institute • Totalview – Starting TotalView – The TotalView GUI and basic windows – Data management and visualization – Breakpoints and Action points – Parallel debugging: MPI, OpenMP, etc.
1 Debuggers at the Institute
• Write statements • CVD • dbx • Intel IDB • pdbx • DDD • Totalview • Xpdbx • Other tools: Compiler • xldb options, profilers, • Pedb hardware counters • GNU gdb
dbx
• dbx is a command line debugger for serial and parallel Fortran and C programs respectively • Most useful Commands: stop at linenumber # set a breakpoint at the given linenumber stop in myroutine # set a breakpoint at the given routine run < infile # run your program, reading from the file infile next # to execute the next step of your code print var # see the value of a given variable whatis var # determine the data type of a given variable cont # continue execution from where it stopped • Compile your program with –g option • Start dbx with or without the executable as an argument. • More info: man dbx, on dbx prompt use help • Available: unix systems: IBM, SGI and SUN
2 pdbx
• Command-line parallel debugger built on dbx • IBM debugger for parallel programs on AIX systems and accepts all flags supported by POE • Supports C, C++, Fortran 77, Fortran 90 programs • Compile the code with -g flag • Setup POE environment variables, e.g.: setenv MP_BUFFER_MEM 28MB setenv MP_EAGER_LIMIT 4090kb • Run pdbx: pdbx a.out [program_options] [poe options] or pdbx -a poe process id [limited poe options] • Available: IBM Power3 and Power4 • More information: man pdbx
xpdbx
• Parallel debugger with graphical interface • Load executable after starting xpdbx. Use the File menu • Use X resources to customize the xpdbx window – The default resources for xpdbx are listed in /usr/lib/X11/app-defaults/Xpdbx • Needs the –rmpool option on the IBM SP/Power4 for an interactive run • Can set break/trace points
3 xldb
• Serial and multi-thread debugger with graphical interface • xldb a.out – Window pops up with source, etc • Group of blue bars at the top right – Click on bar to open window – To minimize window, click on bar at top to get menu, click on “minimize” • To set breakpoint, click on source line • To navigate, see “commands” window
pedb
• Graphical user interface based parallel debugger for Fortran and C programs • Same as xpdbx • pedb a.out
4 GNU gdb
• gdb is a command line debugger for serial programs • Support C, C++, and Fortran77 programs • Available: linux and unix systems • Compile the code : g77 –g my.f or gcc –g my.c • Start debugging: gdb ./a.out or gdb ./a.out core # to include the core or gdb ./a.out 1234 # to attach GDB to process 1234
• Most useful Commands: quit: Exit from GDB. break [file:]function: Set a breakpoint at function in file. run: Start your program bt: Backtrace: display the program stack. print expr: Display the value of an expression. c: Continue running your program (after stopping) stop:set a breakpoint at line number or routine • Getting help: – (gdb) help – http://sources.redhat.com/gdb/onlinedocs/gdb _toc.html – man gdb
5 CVD
• CVD - Invokes the WorkShop Debugger • a source-level debugging tool with GUI • set various types of breakpoints, watchpoints, and so on. • view variables, expressions, structures, arrays, call tree • debug Ada, C, C++, Fortran 77, and Fortran 90 programs. • Available on SGI system running Irix Operating systems (SGI workstations and Origin)
• To invoke the Debugger for a Fortran 77 program: $ f77 -g -o myprog myprog.f $ cvd myprog # serial job or $ cvd myprog core # serial job with core dump or $ cvd mpirun -args -np 2 a.out # For an mpi job
• More information: • Manpage: man cvd • http://techpubs.sgi.com/library/tpl/cgi-bin/init.cgi
CVD and Other Workshop Tools: • Build Analyzer (cvbuild) - Invokes the Build Analyzer that provides a graphical display of build dependency information. • Parallel Analyzer (cvpav) - Invokes the Parallel Analyzer that presents information about MP Fortran programs. • Performance Analyzer (cvperf) - Invokes the Performance Analyzer that lets you specify the name of a performance task to enable proper data collection for your experiment. • Static Analyzer (cvstatic) - Invokes the Static Analyzer that lets you display source code information about your code and facilitates the setting of breakpoints. • Tester (cvxcov) - Invokes the test coverage tool that lets you perform dynamic test coverage over any set of tests.
6 Intel IDB
• Intel debugger are part of the compiler installation • Support C, C++, Fortran 77/90 • Both dbx (default) and gdb like interfaces • Documentation: – http://www.intel.com/software/products/compilers/docs/linux/idb_m anual_l.html • Debugging of optimized code generated by intel compilers does not work • Does not support multithreaded/multiprocess applications. claims do support mpich programs (untested) • Available on the SGI Altix and systems with Intel compilers
DDD
• Graphical debugger • Part of Redhat distribution • Works best with gdb • Can hook in idb via: ddd --debugger “idb” -- dbx ./a.out • Multiple panes for – Data/array browsing – Source code – Assembler code – Command line interface • Use help menu for details
7 Other Debuggers
• Portland Group PGDBG graphical debugger • LF95: Fujitsu debugger FDB
• Write statements! • Editors
Totalview
8 Introduction
• Debugger for Unix/Linux platform • Supports Fortran, C/C++ programs • Serial, multi-threaded, parallel programs employing multiple processors • MPI, MPICH, OpenMP, PVM, SHMEM, Global arrays, and UPC • User interface and debugging style remain uniform and consistent across platforms and programming models.
Starting TotalView
• Compile with –g option • Starting Totalview on MSI machines: module load totalview • On a new process:
% totalview a.out -a
% totalview a.out core • To attach to a running process:
% totalview
9 o
• File > New Program dialog Box • Short cut: ctrl-N
• F1 function key tells Totalview to display help info about the window
TotalView Windows
Root Unattached Window Processes Window
Process Data Window Windows
Test drive: •Menu driven •Alternative keyboard shortcuts for those who prefer them •All 3 mouse buttons are used
10 Root Window
Process name Process/thread status Number of threads
Expand list
Process Window
Local variables Stack for the Trace pane selected frame
Source pane
Action Thread Points pane pane
11 Source Pane
Gridded box is Current function and source file a possible site for a breakpoint Select to set oneCurrent point of execution
• Dive on a source word to get more information • Select a line to use Run to selection command • Select or dive on a line number to set an action point
Stack Trace Pane
• Select a line to choose a new stack frame • Just dive on a routine and TotalView shows the routine in the Source Pane and its variables in the Stack frame value Stack Frame Pane.
12 Stack Frame Pane
Names of arguments
Names of function’s local variables
Stack Frame pane in the process window contains current call stack Values of variables
Dive on a variable to bring up a data pane (and follow a pointer)
Starting, Stopping, & Restarting Your Program • Method 1 – Set a breakpoint – Select Go on the icon bar – The program starts executing and stop when it reaches the breakpoint • Method 2 – Select step – Totalview starts and then stops it immediately before the first statement in your main function • To stop a running program, select halt • To restart a program, select the Group > Restart command
13 • Stepping through a Program – Use the Step and Next commands. Step and Next both execute the current line. The difference between them is that if the line has a function call, Step dives into the function while Next executes it. – If you want to get to a line without individually stepping each intervening line, select the line, and then select Run To. • Stepping Out of a Function – If you've stepped into a function and want to pop out to the statement that called it, select the Out command.
Data Management
14 Displaying Data Seeing a value of a variable • Method 1 Dive on a variable displayed in the Source Pane. The variable appears in a Variable Window. • Method 2 Scroll to the variable in the Stack Frame Pane. If you want to display a compound variable like an array or structure, dive on it. TotalView responds by displaying information in a Variable Window. • Method 3 Use the View > Lookup Variable command. The variable can be global or local. If you've defined the same variable in more than one routine, TotalView displays the variable contained in the current stack frame.
Variable Window Array Data Window • Array has an actual type as well as a declaration type • Arrays have a slice field that you can edit to specify the dimensions to display • You can update data by Selecting (left mouse button) the value and editing it
15 Array Slice
• An array slice indicates which portion of an array you want to see. I.e.: see elements 3 to 8 • Fortran example- TotalView displays a one-dimensional array by using (:). It uses (:,:) if it is displaying a two-dimensional array, and so on
Data Array Filtering
Go to the Filter line and type a filter expression You can specify the elements of an array you’re interested in
This example requests all elements with values >0.8
16 Data Array Filtering (cont.)
• Filter: 5:998 will display elements between 5 and 998 • Filter:== $nan • Filter:== $inf
Visualization • In an array data pane select Visualize from the menu • The Visualizer is automatically started • Arrays of rank up to two may be visualized • Unlike data panes visualizations are not automatically updated • $visualize in EVAL points for animated display
17 Action Points
Action Point Types “Action point” is a generic name for several types of breakpoints: – Breakpoint stops execution of processes and threads that reach it – Barrier breakpoint holds each process that reaches it until all processes in the group reach it – Conditional breakpoint stops code if condition is met – Evaluation point is point at which user-defined code fragments get executed – Data watchpoint
18 Setting a Breakpoint at a Function • Method 1: If you can see the function, click on a line number within the function. • Method 2: Use the View > Lookup Function command to find the function. After the function displays, select a line number. • Method 3: Select Action Points > At Location and then type the function's name in the dialog box.
Setting a Data Watchpoint • A data watchpoint is a type of breakpoint that monitors a variable's value. Unlike other action points, it isn't set on a source line. Instead, it's set on the memory location where your program stores the variable. When the contents of this memory location change, TotalView stops execution and displays the source line that changed it. This means you should only set watchpoints on global variables or on malloc'd memory. • Set a watchpoint by first diving on the variable to display its Variable Window, and then select the Tools > Watchpoint command.
19 Printing Something at a Breakpoint
• After creating a breakpoint, right-click on the sign, and then select Properties from the popup menu. • In the Properties dialog box, select Evaluate and then type a print statement. For example: printf("The value of foo divided by 20 is: %d\n",foo/20); • When code is associated with a breakpoint, the breakpoint is called an eval point.
Stopping Your Program Using a Condition • Create an eval point (this was discussed in the previous section), entering a condition instead of a statement. For example, here's a condition that stops execution when the value of the variable i in the eval point's line is equal to 30: if (i == 30) $stop • One of the big differences between an eval point and a breakpoint is that execution doesn't stop unless you tell it to. That is, if you don't add a $stop to your eval point, TotalView just continues executing. •
20 Stopping Your Program Every "x" Times TotalView Executes a Line • Create an eval point that uses the $count statement to tell TotalView how many times the line should execute. For example: $count 20 • $count is another of TotalView's built-in functions. $stop is implied when you use this statement.
Saving Your Action Points • You usually want this to happen automatically and, unless someone has changed things, TotalView automatically saves them. If it isn't saving them, go to the Options Tab in the File > Preferences dialog box and select Save preferences file on exit. • While you're looking at preferences, you might want to look at the other things you can set.
21 Action Point List Pane Enabled breakpoint
Disabled breakpoint
Breakpoint location • Select on action point icon enables/disables it • Dive on action point refocuses source pane to location; this behaves like a bookmark
Parallelism in TotalView
22 Supported Parallelism
TotalView supports all major parallel programming paradigms – MPI – OpenMP – Pthreads – PVM – HPF – Shmem
TotalView Multi-Process Functionality • TotalView performs actions on groups of processes – Start / stop – Share breakpoints across executables – Take action on multiple processes when at breakpoint – Attach to multiple running processes – Single step groups of processes – Set barrier breakpoints • Easily view and manipulate variables processes – Laminated variables for threads and processes – Distributed HPF arrays
23 Debugging MPI Programs on the IBM SP with Totalview
% totalview poe -a a.out [program_args][PE_args]
• Automatically attaches to all of the MPI processes • TotalView can display MPI message queues
Displaying a variable in all processes or threads
• Need to view or update the value of a variable in all of the processes or threads at once 1. First display an instance of the variable in a Variable Window 2. View > Laminate > Process, or View > Laminate > Thread 3. View > Laminate > None, delaminates the window
• Example on the right is for an OpenMP
24 Other tools
Other Tools
• Profiler: Xprofiler: – GUI based IBM performance profiling tool – Graphical function call tree – Flat report: line by line CPU time – Compile and link your program with –pg – Run your code => produce gmon.out – Invoke xprofiler
xprofiler executable [-s] gmon.out.
25 • Hardware counter: Hpm on the IBM systems – Hardware performance counters • Cycles • Instructions • Float point and fixed point operations • Loads/stores • Cache misses • TLB misses • Etc – Supports MPI, OpenMP and threaded applications – Multiple instrumentations and derived metrics
• MSI – www.msi.umn.edu – Help: help@,msi.umn.edu 612 626 0802 • Totalview – Visit www.etnus.com – Online documentation at www.etnus.com/Support/docs/index.html
26