NightTrace User’s Guide Version 7.1.2 (RedHawkTM Linux®) 0898398-166 August 2009 Copyright 2008 by Concurrent Computer Corporation. All rights reserved. This publication or any part thereof is intended for use with Concurrent products by Concurrent personnel, customers, and end–users. It may not be reproduced in any form without the written permission of the pub- lisher. The information contained in this document is believed to be correct at the time of publication. It is subject to change without notice. Concurrent Computer Corporation makes no warranties, expressed or implied, concerning the information contained in this document. Concurrent Computer Corporation and its logo are registered trademarks of Concurrent Computer Corporation. All other Concurrent product names are trademarks of Concurrent while all other product names are trademarks or registered trademarks of their respective owners. Linux® is used pursuant to a sublicense from the Linux Mark Institute. NightStar’s integrated help system is based on Qt’s Assistant from Trolltech. Preface Scope of Manual This manual is a reference document and user’s guide for NightTraceTM - a graphical, interactive debugging and performance analysis tool. Structure of Manual The manual includes four major parts as shown below: • Event Logging and Capture – Chapters 2 through 6 • Graphical Analysis – Chapters 7 through 17 • Programmatic Analysis – Chapter 18 • Reference – appendices and index Man page descriptions of programs, system calls, subroutines, and file formats appear in the system manual pages. Syntax Notation The following notation is used throughout this guide: italic Books, reference cards, and items that the user must specify appear in italic type. Special terms and comments in code may also appear in italic. list bold User input appears in list bold type and must be entered exactly as shown. Names of directories, files, commands, options and man page refer- ences also appear in list bold type. list Operating system and program output such as prompts and messages and list- ings of files and programs appears in list type. Keywords also appear in list type. window Keyboard sequences and window features such as push buttons, radio buttons, menu items, labels, and titles appear in window type. iii NightTrace RT User’s Guide [ ] Brackets enclose command options and arguments that are optional. You do not type the brackets if you choose to specify such option or arguments. { } Braces enclose mutually exclusive choices separated by the pipe (|) character, where one choice must be selected. You do not type the braces or the pipe character with the choice. ... An ellipsis follows an item that can be repeated. iv Contents Chapter 1 Introduction User Trace Point Placement. .1-2 Kernel Trace Point Placement . .1-2 Timestamps . .1-3 Languages . .1-3 Information Displayed. .1-4 Chapter 2 Using the NightTrace Logging API Language-Specific Source Considerations . .2-1 C . .2-1 Fortran . .2-2 Ada . .2-2 Java. .2-3 Inter-Process Communication and Library Routines . .2-5 Understanding NightTrace Library Calls . .2-6 trace_begin, Trace.begin. .2-7 trace_event, Trace.event and their variants . .2-13 trace_enable, trace_disable, and their variants. .2-20 trace_flush, Trace.flush, trace_trigger, and Trace.trigger. .2-24 trace_set_thread_name, Trace.setThreadName . .2-26 trace_close_thread, Trace.closeThread. .2-27 trace_end, Trace.end. .2-29 trace_diag_mode. .2-31 trace_diag_func . .2-32 Disabling Tracing . .2-33 Threads and Logging. .2-33 Compiling and Linking . .2-34 C Compilation and Linking . .2-34 Fortran Compilation and Linking. .2-34 Ada Example . .2-34 Java Example . .2-35 Chapter 3 Capturing User Events with ntraceud The ntraceud Daemon . .3-1 ntraceud Modes . .3-2 The Default User Daemon Configuration . .3-2 ntraceud Options . .3-3 Invoking ntraceud . .3-6 Chapter 4 Capturing Kernel Events with ntracekd The ntracekd Daemon . .4-1 v NightTrace RT User’s Guide ntracekd Modes . 4-1 ntracekd Options . 4-2 ntracekd Invocations . 4-5 Chapter 5 Application Illumination Overview . 5-1 Illuminator . 5-1 Work Flow Illustration . 5-1 Provided Illuminators . 5-2 Detail Levels. 5-2 Creating and Building an Illuminator . 5-4 illuminator --create . 5-4 --aggregate_limit=limit . .5-4 --config=config.xml. .5-5 --do_nodebug, --dont_nodebug. 5-5 --event_ids=N-[M] . .5-5 --install=path . .5-5 --i*, --x* . .5-6 --iunderscores, --xunderscores. 5-6 --iregex=regex, --xregex=regex . .5-6 --istd, --xstd. 5-7 illuminator --populate . 5-7 illuminator --build. 5-7 1 . 5-8 2 . 5-8 3 . 5-8 next_event.txt. 5-8 illuminator.h . 5-8 illuminator.map . 5-8 illuminator_level.fmt . 5-9 lluminator_level.o . 5-9 lluminator_level.list . 5-9 illuminator.o . 5-9 illuminator --report . 5-9 Linking With Illuminators . 5-10 illuminator --gcc . 5-11 illuminator --g77. 5-11 illuminator --cf77 . 5-11 illuminator --ada . 5-11 Predefined Illuminators . 5-11 main. 5-12 glibc . 5-12 pthread . 5-12 ccur_rt . 5-12 Activating Illuminators . 5-12 program . .5-12 ! . 5-13 main[,options] . ..