Cisco MDS 9000 Family Troubleshooting Guide, Release 3.X Cisco MDS SAN-OS for Release 3.0(1)
Total Page:16
File Type:pdf, Size:1020Kb
Send documentation comments to [email protected] Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x Cisco MDS SAN-OS for Release 3.0(1) April 2006 Corporate Headquarters Cisco Systems, Inc. 170 West Tasman Drive San Jose, CA 95134-1706 USA http://www.cisco.com Tel: 408 526-4000 800 553-NETS (6387) Fax: 408 526-4100 Text Part Number: OL-9285-01 Send documentation comments to [email protected] THE SPECIFICATIONS AND INFORMATION REGARDING THE PRODUCTS IN THIS MANUAL ARE SUBJECT TO CHANGE WITHOUT NOTICE. ALL STATEMENTS, INFORMATION, AND RECOMMENDATIONS IN THIS MANUAL ARE BELIEVED TO BE ACCURATE BUT ARE PRESENTED WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED. USERS MUST TAKE FULL RESPONSIBILITY FOR THEIR APPLICATION OF ANY PRODUCTS. THE SOFTWARE LICENSE AND LIMITED WARRANTY FOR THE ACCOMPANYING PRODUCT ARE SET FORTH IN THE INFORMATION PACKET THAT SHIPPED WITH THE PRODUCT AND ARE INCORPORATED HEREIN BY THIS REFERENCE. IF YOU ARE UNABLE TO LOCATE THE SOFTWARE LICENSE OR LIMITED WARRANTY, CONTACT YOUR CISCO REPRESENTATIVE FOR A COPY. The Cisco implementation of TCP header compression is an adaptation of a program developed by the University of California, Berkeley (UCB) as part of UCB’s public domain version of the UNIX operating system. All rights reserved. Copyright © 1981, Regents of the University of California. NOTWITHSTANDING ANY OTHER WARRANTY HEREIN, ALL DOCUMENT FILES AND SOFTWARE OF THESE SUPPLIERS ARE PROVIDED “AS IS” WITH ALL FAULTS. CISCO AND THE ABOVE-NAMED SUPPLIERS DISCLAIM ALL WARRANTIES, EXPRESSED OR IMPLIED, INCLUDING, WITHOUT LIMITATION, THOSE OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT OR ARISING FROM A COURSE OF DEALING, USAGE, OR TRADE PRACTICE. IN NO EVENT SHALL CISCO OR ITS SUPPLIERS BE LIABLE FOR ANY INDIRECT, SPECIAL, CONSEQUENTIAL, OR INCIDENTAL DAMAGES, INCLUDING, WITHOUT LIMITATION, LOST PROFITS OR LOSS OR DAMAGE TO DATA ARISING OUT OF THE USE OR INABILITY TO USE THIS MANUAL, EVEN IF CISCO OR ITS SUPPLIERS HAVE BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. CCSP, CCVP, the Cisco Square Bridge logo, Follow Me Browsing, and StackWise are trademarks of Cisco Systems, Inc.; Changing the Way We Work, Live, Play, and Learn, and iQuick Study are service marks of Cisco Systems, Inc.; and Access Registrar, Aironet, BPX, Catalyst, CCDA, CCDP, CCIE, CCIP, CCNA, CCNP, Cisco, the Cisco Certified Internetwork Expert logo, Cisco IOS, Cisco Press, Cisco Systems, Cisco Systems Capital, the Cisco Systems logo, Cisco Unity, Enterprise/Solver, EtherChannel, EtherFast, EtherSwitch, Fast Step, FormShare, GigaDrive, GigaStack, HomeLink, Internet Quotient, IOS, IP/TV, iQ Expertise, the iQ logo, iQ Net Readiness Scorecard, LightStream, Linksys, MeetingPlace, MGX, the Networkers logo, Networking Academy, Network Registrar, Packet, PIX, Post-Routing, Pre-Routing, ProConnect, RateMUX, ScriptShare, SlideCast, SMARTnet, The Fastest Way to Increase Your Internet Quotient, and TransPath are registered trademarks of Cisco Systems, Inc. and/or its affiliates in the United States and certain other countries. All other trademarks mentioned in this document or Website are the property of their respective owners. The use of the word partner does not imply a partnership relationship between Cisco and any other company. (0601R) Any Internet Protocol (IP) addresses used in this document are not intended to be actual addresses. Any examples, command display output, and figures included in the document are shown for illustrative purposes only. Any use of actual IP addresses in illustrative content is unintentional and coincidental. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x © 2002–2006 Cisco Systems, Inc. All rights reserved. Send documentation comments to [email protected] CONTENTS New and Changed Information xxiii Preface xxv Document Organization xxv Document Conventions xxvi Related Documentation xxviii Release Notes xxviii Compatibility Information xxviii Regulatory Compliance and Safety Information xxviii Hardware Installation xxviii Cisco Fabric Manager xxix Command-Line Interface xxix Troubleshooting and Reference xxix Installation and Configuration Note xxix Obtaining Documentation xxix Cisco.com xxx Product Documentation DVD xxx Ordering Documentation xxx Documentation Feedback xxx Cisco Product Security Overview xxxi Reporting Security Problems in Cisco Products xxxi Obtaining Technical Assistance xxxii Cisco Technical Support & Documentation Website xxxii Submitting a Service Request xxxii Definitions of Service Request Severity xxxiii Obtaining Additional Publications and Information xxxiii CHAPTER 1 Troubleshooting Overview 1-1 Overview of the Troubleshooting Process 1-1 Overview of Best Practices 1-2 Troubleshooting Basics 1-2 Troubleshooting Guidelines 1-2 Gathering Information Using Common Fabric Manager Tools and CLI Commands 1-3 Common Fabric Manager Tools 1-3 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-01 iii Contents Send documentation comments to [email protected] Common CLI Commands 1-4 Verifying Basic Connectivity 1-4 Verifying SAN Element Registration 1-5 Fibre Channel End-to-End Connectivity 1-5 Fabric Issues 1-5 Port Issues 1-6 Primary Troubleshooting Flowchart 1-8 Overview of Symptoms 1-8 System Messages 1-9 System Message Text 1-9 Syslog Server Implementation 1-10 Implementing Syslog with Fabric Manager 1-10 Implementing Syslog with the CLI 1-11 Troubleshooting with Logs 1-12 Viewing Logs with Fabric Manager 1-12 Viewing Logs with the CLI 1-13 Viewing the Log from the Supervisor 1-13 Viewing NVRAM logs 1-13 Contacting Customer Support 1-14 CHAPTER 2 Troubleshooting Installs, Upgrades, and Reboots 2-1 Overview 2-1 Best Practices 2-2 Best Practices for Installations 2-2 Best Practices for Upgrading 2-2 Best Practices for Reboots 2-4 Disruptive Module Upgrades 2-4 Troubleshooting Fabric Manager Installations 2-4 Verifying Cisco SAN-OS Software Installations 2-5 Troubleshooting Cisco SAN-OS Software Upgrades and Downgrades 2-6 Software Installation Reports an Incompatibility 2-6 Diagnosing Compatibility Issues 2-6 Software Installation Ends with Error 2-8 Installing SAN-OS Software Using Fabric Manager 2-9 Installing Cisco SAN-OS Software from the CLI 2-10 Troubleshooting Cisco SAN-OS Software System Reboots 2-12 Power On or Switch Reboot Hangs 2-12 Corrupted Bootflash Recovery 2-13 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x iv OL-9285-01 Contents Send documentation comments to [email protected] Recovery Using BIOS Setup 2-15 Recovery from the loader> Prompt 2-18 Recovery from the switch(boot)# Prompt 2-19 Recovery for Switches with Dual Supervisor Modules 2-20 Recovering One Supervisor Module With Corrupted Bootflash 2-20 Recovering Both Supervisor Modules With Corrupted Bootflash 2-21 Recognizing Error States 2-22 Switch or Process Resets 2-23 Recoverable System Restarts 2-24 Unrecoverable System Restarts 2-28 Recovering the Administrator Password 2-29 Miscellaneous Software Image Issues 2-29 All Ports Down Because of System Health Failure 2-29 Switch Reboots after FCIP Reload 2-30 FCIP Link Fails to Come Up 2-30 Cannot Create, Modify, or Delete Admin Role 2-30 FC IDs Change after Link Reset 2-31 Switch Displays Wrong User 2-31 CHAPTER 3 Managing Storage Services Modules 3-1 SSM Overview 3-1 Best Practices 3-3 Licensing Requirements 3-3 Initial Troubleshooting Checklist 3-3 Common Troubleshooting Tools in Fabric Manager 3-3 Common Troubleshooting Commands in the CLI 3-4 SSM Issues 3-4 SSM Fails to Boot 3-4 Upgrading the SSI Image 3-5 Verifying the SSI Boot Image 3-5 Using the install ssi Command 3-7 Recovering a Replacement SSM 3-9 SSM Upgrade Is Disruptive 3-9 Installing EPLD Images on Modules 3-10 CHAPTER 4 Troubleshooting Hardware 4-1 Overview 4-1 Best Practices 4-2 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-01 v Contents Send documentation comments to [email protected] Best Practices for Switch Installation 4-2 Best Practices for System Initialization 4-2 Best Practices for Supervisor Modules 4-3 Troubleshooting Startup Issues 4-3 Troubleshooting Power Supply Issues 4-4 All Power Supply LEDS Are Off 4-5 Power Supply Input Ok LED is Red 4-6 Power Supply Output Failed LED is On 4-7 Power Supply Fan Ok LED is Red 4-7 Troubleshooting the Power Supplies 4-8 Troubleshooting Fan Issues 4-9 Fan Is Not Spinning 4-9 Fan Is Spinning; Fan LED is Red 4-9 Troubleshooting a Fan Failure Using Device Manager 4-10 Troubleshooting a Fan Failure Using the CLI 4-11 Temperature Threshold Violations 4-12 Troubleshooting Clock Module Issues 4-13 Troubleshooting Other Hardware Issues 4-14 Troubleshooting Supervisor Issues 4-15 Active Supervisor Reboots 4-16 Standby Supervisor Not Recognized by Active Supervisor 4-18 Verifying That a Standby Supervisor Failed to Synchronize Using the CLI 4-18 Standby Supervisor Stays in Powered-Up State 4-20 Verifying That a Standby Supervisor Is in the Powered-Up State Using Device Manager 4-21 Verifying That a Standby Supervisor Is in Powered-Up State Using the CLI 4-21 Troubleshooting Supervisor Modules 4-21 Troubleshooting Switching and Services Modules 4-22 Overview of Module Status 4-22 Module Initialization Overview 4-23 Module Bootup 4-24 Image Download 4-24 Runtime Diagnostics 4-25 Runtime Configuration 4-25 Online and Operational 4-25 Analyzing The Logs 4-26 Troubleshooting Module Issues 4-26 Troubleshooting Powered-Down Modules 4-27 Diagnosing a Powered-Down Module 4-29 Troubleshooting Reloaded Modules 4-32 Cisco MDS 9000 Family Troubleshooting