DB2 ® DB2 Version 9 for Linux, UNIX, and Windows Troubleshooting Guide GC10-4240-00 DB2 ® DB2 Version 9 for Linux, UNIX, and Windows Troubleshooting Guide GC10-4240-00 Before using this information and the product it supports, be sure to read the general information under Notices. Edition Notice This document contains proprietary information of IBM. It is provided under a license agreement and is protected by copyright law. The information contained in this publication does not include any product warranties, and any statements provided in this manual should not be interpreted as such. You can order IBM publications online or through your local IBM representative. v To order publications online, go to the IBM Publications Center at www.ibm.com/shop/publications/order v To find your local IBM representative, go to the IBM Directory of Worldwide Contacts at www.ibm.com/ planetwide To order DB2 publications from DB2 Marketing and Sales in the United States or Canada, call 1-800-IBM-4YOU (426-4968). When you send information to IBM, you grant IBM a nonexclusive right to use or distribute the information in any way it believes appropriate without incurring any obligation to you. © Copyright International Business Machines Corporation 1993, 2006. All rights reserved. US Government Users Restricted Rights – Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp. Contents Chapter 1. Learning more . .1 Mimicking databases using db2look . .32 Introduction to problem determination . .1 Listing DB2 products installed on your system About first failure data capture(FFDC) . .3 (Linux and UNIX) . .35 First failure data capture information . .3 Monitoring and troubleshooting using db2pd . .37 General First Failure Data Capture (FFDC) . .3 Collecting environment information using About administration notification log files . .5 db2support . .45 Administration notification log . .5 Traces . .47 Setting the error capture level for the Basic trace diagnostics . .47 administration notification log file . .6 DB2 traces . .49 Interpreting administration notification log file DRDA traces . .52 entries . .6 Control Center traces . .61 About DB2 diagnostic log (db2diag.log) files . .8 JDBC trace files . .62 Setting the diagnostic log file error capture level .8 CLI traces . .64 Interpreting the db2diag.log file informational Platform-specific tools . .70 record . .8 Diagnostic tools (Windows) . .70 Interpreting diagnostic log file entries . .9 Diagnostic tools (Linux and UNIX) . .71 db2cos (callout script) output files . .11 Dump Files . .13 Chapter 5. Searching knowledge bases 75 About trap files . .14 How to search effectively for known problems . .75 Trap files . .14 Problem determination resources . .76 Formatting trap files (Windows) . .15 About platform-specific error logs . .15 Chapter 6. Getting fixes . .77 Platform specific error log information . .15 Applying fix packs . .77 System core files (UNIX) . .16 Accessing system core file information (UNIX) .17 Appendix A. DB2 Database technical Accessing event logs (Windows) . .18 information . .79 Exporting event logs (Windows) . .18 Accessing the Dr. Watson log file (Windows) . .19 Overview of the DB2 technical information . .79 Combining DB2 database and OS diagnostics . .19 Documentation feedback . .79 DB2 technical library in hardcopy or PDF format . .80 Ordering printed DB2 books . .82 Chapter 2. Troubleshooting DB2 . .23 Displaying SQL state help from the command line Troubleshooting DB2 . .23 processor . .83 Troubleshooting partitioned database environments 23 Accessing different versions of the DB2 Information Issuing commands in a partitioned database Center . .84 environment . .23 Displaying topics in your preferred language in the DB2 Information Center . .84 Chapter 3. Troubleshooting DB2 Updating the DB2 Information Center installed on Connect . .25 your computer or intranet server . .85 Problem determination . .25 DB2 tutorials . .87 Gathering relevant information . .25 DB2 troubleshooting information . .87 Initial connection is not successful . .26 Terms and Conditions . .88 Problems encountered after an initial connection . .27 Diagnostic tools . .28 Appendix B. Notices . .89 Trademarks . .91 Chapter 4. Tools for troubleshooting 29 Overview of the db2dart tool . .29 Index . .93 Analyzing db2diag.log files using db2diag . .29 Displaying and altering the Global Registry (UNIX) Contacting IBM . .95 using db2greg . .31 Identifying the version and service level of your product . .31 © Copyright IBM Corp. 1993, 2006 iii iv Troubleshooting Guide Chapter 1. Learning more Introduction to problem determination The first step in good problem analysis is to describe the problem completely. Without a problem description, you will not know where to start investigating the cause of the problem. This step includes asking yourself such basic questions as: v What are the symptoms? v Where is the problem happening? v When does the problem happen? v Under which conditions does the problem happen? v Is the problem reproducible? Answering these and other questions will lead to a good description to most problems, and is the best way to start down the path of problem resolution. What are the symptoms?: ″ When starting to describe a problem, the most obvious question is What is the ″ problem? This may seem like a straightforward question; however, it can be broken down into several other questions to create a more descriptive picture of the problem. These questions can include: v Who or what is reporting the problem? v What are the error codes and error messages? v How does it fail? For example: loop, hang, crash, performance degradation, incorrect result. v What is the business impact? Where is the problem happening?: Determining where the problem originates is not always easy, but it is one of the most important steps in resolving a problem. Many layers of technology can exist between the reporting and failing components. Networks, disks, and drivers are only a few components to be considered when you are investigating problems. v Is the problem platform specific, or common to multiple platforms? v Is the current environment and configuration supported? v Is the application running locally on the database server or on a remote server? v Is there a gateway involved? v Does the database reside on individual disks, or on a RAID disk array? These types of questions will help you isolate the problem layer, and are necessary to determine the problem source. Remember that just because one layer is reporting a problem, it does not always mean the root cause exists there. Part of identifying where a problem is occurring is understanding the environment in which it exists. You should always take some time to completely describe the problem environment, including the operating system, its version, all corresponding software and versions, and hardware information. Confirm you are running within an environment that is a supported configuration, as many © Copyright IBM Corp. 1993, 2006 1 problems can be explained by discovering software levels that are not meant to run together, or have not been fully tested together. When does the problem happen?: Developing a detailed time line of events leading up to a failure is another necessary step in problem analysis, especially for those cases that are one-time occurrences. You can most easily do this by working backwards --start at the time an error was reported (as exact as possible, even down to milliseconds), and work backwards through available logs and information. Usually you only have to look as far as the first suspicious event that you find in any diagnostic log, however, this is not always easy to do and will only come with practice. Knowing when to stop is especially difficult when there are multiple layers of technology each with its own diagnostic information. v Does the problem only happen at a certain time of day or night? v How often does it happen? v What sequence of events leads up to the time the problem is reported? v Does the problem happen after an environment change such as upgrading existing or installing new software or hardware? Responding to questions like this will help you create a detailed time line of events, and will provide you with a frame of reference in which to investigate. Under which conditions does the problem happen?: Knowing what else is running at the time of a problem is important for any complete problem description. If a problem occurs in a certain environment or under certain conditions, that can be a key indicator of the problem cause. v Does the problem always occur when performing the same task? v Does a certain sequence of events need to occur for the problem to surface? v Do other applications fail at the same time? Answering these types of questions will help you explain the environment in which the problem occurs, and correlate any dependencies. Remember that just because multiple problems may have occurred around the same time, it does not necessarily mean that they are always related. Is the problem reproducible?: ″ ″ From a problem description and investigation standpoint, the ideal problem is one that is reproducible. With reproducible problems you almost always have a larger set of tools or procedures available to use to help your investigation. Consequently, reproducible problems are usually easier to debug and solve. However, reproducible problems can have a disadvantage: if the problem is of significant business impact, you don’t want it recurring. If possible, recreating