Front cover Problem Determination for Linux on System z Learn a Linux on System z problem determination methodology Discover problem determination tools for z/VM and Linux on System z Do a case study using what you learn here Lydia Parziale Manoj S Pattabhiraman Masayuki Unno Eravimangalath P. Naveen Klaus Egeler Thiago Patricio de Jesus ibm.com/redbooks International Technical Support Organization Problem Determination for Linux on System z August 2008 SG24-7599-00 Note: Before using this information and the product it supports, read the information in “Notices” on page ix. First Edition (August 2008) This edition applies to Version 5, Release 3, of z/VM (product number 5741-A05), Novel USE Linux Enerprise Server 10 and Red Hat Linux 5. © Copyright International Business Machines Corporation 2008. All rights reserved. Note to U.S. Government Users Restricted Rights -- Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp. Contents Notices . ix Trademarks . x Preface . xi This book The team that wrote this book . xi Become a published author . xiii Comments welcome. xiii Chapter 1. Problem determination methodology . 1 1.1 Proactive approach for minimizing the effects of problems. 3 1.1.1 Design and implementation phases . 3 1.1.2 Test phase . 7 1.2 Problem determination basics . 9 1.2.1 Basic problem determination steps . 9 1.2.2 Gather information and explore the type of problem and its trigger . 11 1.2.3 Analysis of a problem . 13 1.2.4 Collect additional information . 15 1.2.5 Escalate to the support team to analyze . 16 1.2.6 How to approach complex problems. 16 1.3 Problem resolution and follow-up actions . 17 1.3.1 Expanded check for other systems . 18 1.3.2 Update. 18 1.4 Summary . 20 Chapter 2. Problem determination tools for z/VM . 21 2.1 z/VM system tools . 22 2.1.1 Collecting technical setup information. 22 2.1.2 Indicate . 23 2.1.3 query srm . 27 2.1.4 query alloc page . 28 2.1.5 VM trace . 28 2.1.6 vmdump . 33 2.2 IBM z/VM Performance Toolkit . 38 2.2.1 Modes of operation . 39 2.2.2 Selecting performance screens. 43 © Copyright IBM Corp. 2008. All rights reserved. iii 2.3 IBM Tivoli OMEGAMON XE on z/VM and Linux . 52 Chapter 3. Problem determination tools for Linux on System z. 59 3.1 Information-gathering tools . 60 3.1.1 dbginfo.sh . 60 3.1.2 vmstat . 61 3.1.3 System status (sysstat) tool . 62 3.1.4 top . 66 3.1.5 ps . 67 3.1.6 ipcs . 67 3.1.7 iostat . 68 3.1.8 netstat . 68 3.1.9 DASD statistics . 69 3.1.10 OProfile . 73 3.1.11 zFCP statistics . 76 3.1.12 Summary of information-gathering tools . 80 3.2 Dump and trace. 80 3.2.1 When to use dump . 81 3.2.2 When a trace should be run . 82 3.2.3 Stand-alone dump. 83 3.2.4 Verification of the dump . 84 3.2.5 Core dump . 87 3.2.6 Summary of dump techniques . 92 3.3 Debugging tools . 92 3.3.1 gdb . 93 3.3.2 strace . 96 3.3.3 ltrace . 98 3.3.4 SysRq . 100 3.3.5 s390dbf . 104 3.3.6 zFCP/SCSI . 106 Chapter 4. Network problem determination . 109 4.1 Overview of networking options for Linux on System z. 110 4.1.1 Open Systems Adapters . 110 4.1.2 HiperSockets. 112 4.1.3 Guest LAN. 113 4.1.4 Virtual Switch . 113 4.1.5 Summary of networking options for Linux on System z . 114 4.2 Network interface detection and configuration under Linux on System z 115 4.2.1 The Linux hotplug system . 115 4.2.2 Flow of network interface detection and activation under Linux . 116 4.2.3 Network interfaces and their interface names. 119 iv Problem Determination for Linux on System z 4.2.4 Network configuration files overview. 121 4.3 Network problem determination . 123 4.3.1 Problems related to network communication . 123 4.4 Case studies . 159 Chapter 5. Performance problem determination . 163 5.1 What a performance problem is . 164 5.2 General flow of performance problem determination . 165 5.2.1 Data necessary for investigation . 169 5.2.2 CPU bottlenecks . 170 5.2.3 Memory bottlenecks . 183 5.2.4 I/O bottlenecks . 192 5.2.5 Case studies . 205 Chapter 6. Storage problems . 211 6.1 How to determine a storage-related problem . ..