SGI® InfiniteStorage CXFSTM Administration Guide 007–4016–018 CONTRIBUTORS Written by Lori Johnson Illustrated by Chrystie Danzer Engineering contributions to the book by Rich Altmaier, Neil Bannister, François Barbou des Places, Ken Beck, Felix Blyakher, Laurie Costello, Mark Cruciani, Dave Ellis, Brian Gaffey, Philippe Gregoire, Dean Jansa, Erik Jacobson, Dennis Kender, Chris Kirby, Ted Kline, Dan Knappe, Kent Koeninger, Linda Lait, Bob LaPreze, Steve Lord, Aaron Mantel, Troy McCorkell, LaNet Merrill, Terry Merth, Nate Pearlstein, Bryce Petty, Alain Renaud, John Relph, Elaine Robinson, Dean Roehrich, Wesley Smith, Kerm Steffenhagen, Paddy Sreenivasan, Andy Tran, Rebecca Underwood, Connie Waring, Geoffrey Wehrman COPYRIGHT © 1999–2003 Silicon Graphics, Inc. All rights reserved; provided portions may be copyright in third parties, as indicated elsewhere herein. No permission is granted to copy, distribute, or create derivative works from the contents of this electronic documentation in any manner, in whole or in part, without the prior written permission of Silicon Graphics, Inc. LIMITED RIGHTS LEGEND The electronic (software) version of this document was developed at private expense; if acquired under an agreement with the USA government or any contractor thereto, it is acquired as "commercial computer software" subject to the provisions of its applicable license agreement, as specified in (a) 48 CFR 12.212 of the FAR; or, if acquired for Department of Defense units, (b) 48 CFR 227-7202 of the DoD FAR Supplement; or sections succeeding thereto. Contractor/manufacturer is Silicon Graphics, Inc., 1600 Amphitheatre Pkwy 2E, Mountain View, CA 94043-1351. TRADEMARKS AND ATTRIBUTIONS Silicon Graphics, SGI, the SGI logo, IRIS, IRIX, O2, Octane, Onyx, Onyx2, Origin, and XFS are registered trademarks and Altix, CXFS, FailSafe, IRISconsole, IRIS FailSafe, FDDIXPress, NUMAlink, Octane2, Performance Co-Pilot, Silicon Graphics Fuel, SGI FailSafe, and Trusted IRIX are trademarks of Silicon Graphics, Inc., in the United States and/or other countries worldwide. AIX is a registered trademark of IBM Corporation. Brocade is a trademark of Brocade Communication Systems, Inc. Inc. Digi is a trademark of Digi International, Inc. FLEXlm is a trademark of Macrovision Corporation. Java and Sun are registered trademarks and Solaris is a trademark of Sun Microsystems, Inc. HP-UX is a registered trademark of Hewlett-Packard Company. Linux is a registered trademark of Linus Torvalds, used with permission by Silicon Graphics, Inc. Legato NetWorker is a registered trademark of Legato Systems, Inc. Netscape is a trademark of Netscape Communications Corporation. Red Hat is a registered trademark and RPM is a trademark of Red Hat, Inc. UNIX is a registered trademark of The Open Group in the United States and other countries. VERITAS is a trademark of VERITAS Software Corporation. Windows and Windows NT are registered trademarks of Microsoft Corporation. X/Open is a registered trademark of X/Open Company Ltd. Cover design by Sarah Bolles, Sarah Bolles Design, and Dany Galgani, SGI Technical Publications. New Features in This Guide Note: Relocation is not supported in this release. Recovery is supported only when using standby nodes. A standby node is a metadata server-capable administration node that is configured as a potential metadata server for a given filesystem, but does not currently run any applications that will use that filesystem. To use relocation or recovery, you must not run any applications on any of the potential metadata servers for a given filesystem; after the active metadata server has been chosen by the system, you can then run applications that use the filesystem on the active metadata server and client-only nodes. Relocation and recovery are fully implemented, but the number of associated problems prevents full support of these features in the current release. Although data integrity is not compromised, cluster node panics or hangs are likely to occur. Relocation and recovery will be fully supported in a future release when these issues are resolved. This update contains the following: • Support for SGI ProPack for Linux 64-bit metadata servers on SGI Altix 3000 family of servers and superclusters. A CXFS cluster can contain either Linux 64-bit for SGI ProPack 2.3 server-capable nodes on Altix systems or IRIX server-capable nodes; you cannot mix IRIX and Linux 64-bit server-capable nodes within one cluster. CXFS does not support the relocation or recovery of DMAPI filesystems that are being served by Linux 64-bit metadata servers. Coexecution with FailSafe is not supported on Linux 64-bit nodes. • Due to packaging enhancements, CXFS may now be installed on the M stream or the F stream. The IRIX CXFS software will no longer be bundled in the IRIX overlay CDs but instead is on a separate CXFS IRIX Server and Client 3.0 for IRIX 6.5.22 CD. This changes the installation procedure; see Chapter 6, "IRIX CXFS Installation", page 61. 007–4016–018 iii New Features in This Guide Note: If you are upgrading from a previous IRIX release and have CXFS installed, you must upgrade both IRIX and CXFS. If you try to upgrade one without the other, conflicts will occur. • Information about defining networks for CXFS kernel messaging (in addition to the network used for heartbeat/control). If you supply multiple interfaces, the network will fail over from a higher-priority network to a lower-priority network. However, use of these networks is deferred in this release and may be available with a patch. See: – "Define a Node with the GUI", page 152 – "Modify a Node Definition with the GUI", page 163 – "Define a Node with cmgr", page 202 – "Modify a Node with cmgr", page 212 • Support for IRIX real-time filesystems. See "Configuring Real-Time Filesystems For IRIX Nodes", page 262. • Suggestions for configuring large clusters. See "Configuring a Large Cluster", page 125. • Information about using ping to verify general connectivity and CXFS heartbeat in a multicast environment; see "Verifying Connectivity in a Multicast Environment", page 378. • The GUI has been changed to show a single display for the nodes in the cluster and nodes that are in the pool but not in the cluster. This new selection is View: Nodes and Cluster. • Information about information retaining system core files and the output from the cxfsdump utility when reporting problems. See "Reporting Problems to SGI", page 379. • Information about monitoring heartbeat timeouts for IRIX using Performance Co-Pilot or the icrash command. See "Heartbeat Timeout Status", page 320. • The ability to define multiple CXFS filesystems at one time with the GUI; see "Define CXFS Filesystems with the GUI", page 183. iv 007–4016–018 Record of Revision Version Description 001 September 1999 Supports the CXFS 1.1 product in the IRIX 6.5.6f release. 002 October 1999 Supports the CXFS 1.1 product in the IRIX 6.5.6f release. 003 December 1999 Supports the CXFS product in the IRIX 6.5.7f release. 004 March 2000 Supports the CXFS product in the IRIX 6.5.8f release. 005 June 2000 Supports the CXFS product in the IRIX 6.5.9f release. 006 September 2000 Supports the CXFS product in the IRIX 6.5.10f release. 007 January 2001 Supports the CXFS product in the IRIX 6.5.11f release. 008 March 2001 Supports the CXFS product in the IRIX 6.5.12f release. 009 June 2001 Supports the CXFS product in the IRIX 6.5.13f release. 011 September 2001 Supports the CXFS product in the IRIX 6.5.14f release. (Note, there was no 010 version due to an internal numbering mechanism.) 012 December 2001 Supports the CXFS Version 2 product in IRIX 6.5.15f. 013 March 2002 Supports the CXFS Version 2 product in IRIX 6.5.16f. 007–4016–018 v Record of Revision 014 June 2002 Supports the CXFS Version 2 product in IRIX 6.5.17f. 015 September 2002 Supports the CXFS Version 2 product in IRIX 6.5.18f. 016 December 2002 Supports the CXFS Version 2 product in IRIX 6.5.19f. 017 March 2003 Supports the CXFS Version 2 product in IRIX 6.5.20f. 018 September 2003 Supports the CXFS 3.0 product in IRIX 6.5.22 and CXFS 3.0 for SGI Altix 3000 running SGI ProPack 2.3 for Linux. vi 007–4016–018 Contents About This Guide .....................xxix Related Publications . ...................xxix Obtaining Publications . ...................xxxi Conventions . ........................xxxi Reader Comments .......................xxxii 1. Introduction to CXFS . ................. 1 What is CXFS? ........................ 1 Comparison of XFS and CXFS ................... 2 Supported XFS Features . ................... 3 When to Use CXFS . ................... 4 Performance Considerations ................... 5 Comparison of Network and CXFS Filesystems .............. 6 Network Filesystems . ................... 6 CXFS Filesystems . ................... 6 Features ........................ 7 Restrictions ....................... 8 Cluster Environment . ................... 8 Terminology ........................ 8 Cluster . ........................ 9 Node . ........................ 9 Pool . ........................ 9 Cluster Database . ................... 10 Node Functions . ................... 11 Membership ....................... 18 007–4016–018 vii Contents Private Network . ................... 18 Relocation ........................ 19 Recovery ........................ 20 Isolating Failed Nodes . ................... 22 I/O Fencing ....................... 23 Serial Hardware Reset . ................... 27 The Cluster Database and CXFS Clients . .............. 30 Metadata Server Functions . ..................