Front cover Problem Solving and Troubleshooting in AIX 5L Introduction of new problem solving tools and changes in AIX 5L Step by step troubleshooting in real environments Complete guide to the problem determination process HyunGoo Kim JaeYong An John Harrison Tony Abruzzesse KyeongWon Jeong ibm.com/redbooks International Technical Support Organization Problem Solving and Troubleshooting in AIX 5L January 2002 SG24-5496-01 Take Note! Before using this information and the product it supports, be sure to read the general information in “Special notices” on page 439. Second Edition (January 2002) This edition applies to IBM ^ pSeries and RS/6000 Systems for use with the AIX 5L Operating System Version 5.1 and is based on information available in August, 2001. Comments may be addressed to: IBM Corporation, International Technical Support Organization Dept. JN9B Building 003 Internal Zip 2834 11400 Burnet Road Austin, Texas 78758-3493 When you send information to IBM, you grant IBM a non-exclusive right to use or distribute the information in any way it believes appropriate without incurring any obligation to you. © Copyright International Business Machines Corporation 1999-2002. All rights reserved. Note to U.S Government Users – Documentation related to restricted rights – Use, duplication or disclosure is subject to restrictions set forth in GSA ADP Schedule Contract with IBM Corp. Contents Figures . xiii Tables . xv Preface . xvii The team that wrote this redbook. xvii Special notice . xix IBM trademarks . xix Comments welcome. xix Chapter 1. Problem determination introduction. 1 1.1 Problem determination process . 3 1.1.1 Defining the problem . 3 1.1.2 Gathering information from the user . 3 1.1.3 Gathering information from the system . 5 1.1.4 Resolving the problem . 5 1.1.5 Obtaining software fixes and hardware microcode updates . 6 1.1.6 Other relevant documentation. 7 1.2 Troubleshooting starting point . 10 1.2.1 Boot path flowchart . 10 1.2.2 Symptom index . 15 1.3 Avoiding problems . 17 1.3.1 System health check . 17 Chapter 2. Error logging . 19 2.1 Error logging overview . 20 2.2 Error log file processing . 21 2.2.1 Error templates . 21 2.2.2 Error messages. 22 2.3 Error log management . 22 2.3.1 Viewing the error log . 23 2.3.2 Reading a summary error log . 27 2.3.3 Reading error logs in detail . 29 2.3.4 Examples of detailed error reports . 30 2.4 Finding a core dump. 34 Chapter 3. Boot problems. 39 3.1 Types of machines . 40 3.2 The boot process . 40 © Copyright IBM Corp. 2001 iii 3.2.1 Pre-boot state . 41 3.2.2 Boot stage one . 43 3.2.3 Boot stage two . 47 3.2.4 Boot stage three . 48 3.3 Boot problem determination . 49 3.3.1 Failure to locate a boot image. 49 3.3.2 Codes displayed longer than five minutes . 51 3.3.3 LED code 269 on SMP machines . 52 3.3.4 LED 549 hang . 54 3.4 Minimum configuration . 55 3.5 Accessing rootvg from bootable media . 57 3.6 LED 551, 552, 554, 555, 556, and 557 halts. 58 3.6.1 LED 551, 555, or 557 halt . 58 3.6.2 LED 552, 554, or 556 halt . 60 3.6.3 LED 553 halt . 64 3.7 No login prompt . 68 Chapter 4. System dumps . 69 4.1 Introduction . 70 4.2 Saving a system dump when system is booting . 70 4.2.1 LED 549 . 71 4.3 Preparing for the dump. 71 4.3.1 Estimate the size of the dump. 71 4.3.2 Selecting the dump device . 74 4.3.3 Create a dump device . 78 4.3.4 The dumpcheck command . 79 4.3.5 Change the size of a dump device . 82 4.3.6 Optional setup . 83 4.4 Checking the dump status . 85 4.4.1 Get the last dump information . 85 4.4.2 Dump status codes . 85 4.4.3 The sysdumpdev command . 86 4.4.4 Verifying the dump . 87 4.5 Collecting the dump and related information. 89 4.5.1 The snap command . 89 4.6 Initiating a system dump. 90 4.6.1 LED reason codes. 91 4.6.2 How to force a dump . 91 4.7 The kdb command . 93 4.7.1 Uses of kdb . 93 4.7.2 What is the kernel? . 93 4.7.3 Examining a system dump with the kdb command . 94 4.7.4 Basic kdb subcommands . 94 iv Problem Solving and Troubleshooting in AIX 5L 4.7.5 Examining a system dump with the crash command . 103 4.7.6 Types of crashes. 112 4.7.7 Data required by IBM support . 113 Chapter 5. Hardware problem determination . 115 5.1 General advice . 116 5.1.1 Diagnostic tips. 116 5.1.2 Device location notation . 117 5.2 Problem diagnosis . ..