Front cover
Implementing the IBM General Parallel File System (GPFS) in a Cross-Platform Environment
Describes features, use cases, and implementation scenarios of GPFS v3.4
Explains management, configuration, performance, and tuning factors
Gives problem determination and disaster recovery help
Dino Quintero Matteo Barzaghi Randy Brewster Wan Hee Kim Steve Normann Paulo Queiroz Robert Simon Andrei Vlad
ibm.com/redbooks
International Technical Support Organization
Implementing the IBM General Parallel File System (GPFS) in a Cross-Platform Environment June 2011
SG24-7844-00
Note: Before using this information and the product it supports, read the information in “Notices” on page vii.
First Edition (June 2011)
This edition applies to IBM AIX 6.1 TL05, IBM Virtual IO Server 2.1.3.10-FP23, IBM General Parallel File System 3.4.0.1, RedHat Enterprise Linux 5.5.
© Copyright International Business Machines Corporation 2011. All rights reserved. Note to U.S. Government Users Restricted Rights -- Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp. Contents
Notices ...... vii Trademarks ...... viii
Preface ...... ix The team who wrote this book ...... ix Now you can become a published author, too! ...... xi Comments welcome...... xi Stay connected to IBM Redbooks ...... xii
Chapter 1. Introduction...... 1 1.1 Overview and strategy ...... 2 1.2 Features ...... 2 1.3 Licensing ...... 3 1.4 Operating systems support ...... 3 1.5 Hardware support ...... 3 1.6 Contact information ...... 4
Chapter 2. Infrastructure planning ...... 5 2.1 Network design ...... 6 2.1.1 The 1 Gigabit Ethernet and 10 Gigabit Ethernet...... 6 2.1.2 InfiniBand (IP over IB/RDMA) ...... 6 2.1.3 Configuring the NIC ...... 9 2.1.4 IBM data center networking (DCN) products ...... 10 2.2 Storage design ...... 11 2.2.1 Host bus adapter (HBA) ...... 12 2.2.2 Multipath driver ...... 13 2.2.3 The characteristic of storage hardware...... 16 2.2.4 Tape library ...... 20 2.3 InfiniBand Storage DCS9900 ...... 22 2.4 Industry solution design ...... 23 2.4.1 High-Performance Computing...... 23 2.4.2 RDBMS ...... 25 2.4.3 Web application server ...... 33 2.4.4 Media contents delivery ...... 33 2.4.5 Commercial industry ...... 35 2.5 GPFS planning guidance ...... 38 2.5.1 Operating system and file systems with multiplatform GPFS ...... 40 2.5.2 Security ...... 45 2.5.3 High availability ...... 46 2.5.4 Network Shared Disk (NSD) creation considerations ...... 54 2.5.5 GPFS considerations ...... 60 2.5.6 Tivoli Storage Manager for GPFS...... 67 2.6 Summary...... 68
Chapter 3. Scenarios ...... 69 3.1 ITSO laboratory environment ...... 70 3.1.1 Diagram of ITSO laboratory ...... 70 3.1.2 ITSO laboratory environment host file...... 70 3.2 Three-node GPFS cluster using internal disks ...... 73
© Copyright IBM Corp. 2011. All rights reserved. iii 3.2.1 Requirements: Hardware, software, network, storage ...... 73 3.2.2 GPFS configuration...... 74 3.2.3 Diagram of GPFS three-node cluster with internal disks ...... 75 3.2.4 Setting up and configuring a three-node cluster ...... 75 3.3 Recovery on the three-node cluster ...... 85 3.4 Linux InfiniBand cluster with RDMA and Linux for System x clients ...... 90 3.4.1 Requirements: Hardware and software...... 91 3.4.2 Ethernet GPFS cluster ...... 91 3.4.3 InfiniBand GPFS cluster ...... 94 3.5 Cross-platform cluster: Windows servers, Linux/AIX clients ...... 97 3.5.1 Requirements: Hardware, software, network, storage ...... 97 3.5.2 GPFS cluster configuration ...... 97 3.5.3 GPFS cluster diagram...... 98 3.5.4 Installing Windows Server 2008 R2 ...... 98 3.5.5 Install the IBM GPFS Windows version ...... 108 3.5.6 GPFS Windows cluster setup ...... 109 3.5.7 Simulating a GPFS failure on the cluster ...... 115 3.5.8 Adding x86 Linux node to the Windows cluster ...... 117 3.6 DB2 pureScale InfiniBand cluster on AIX ...... 120 3.6.1 Requirements: Hardware, software, network, storage ...... 120 3.6.2 GPFS configuration...... 121 3.6.3 GPFS cluster diagram...... 122 3.6.4 Cluster setup and configuration ...... 122 3.6.5 Use the mmpmon tool to collect statistics from each node...... 141 3.6.6 Simulating a GPFS failure on member 0 (algeria) ...... 145 3.6.7 Troubleshooting DB2 pureScale issues ...... 147 3.7 Multi-cluster configuration ...... 149 3.7.1 Requirements: Hardware, software, network, storage ...... 149 3.7.2 GPFS configuration...... 149 3.7.3 GPFS multi-cluster diagram ...... 150 3.7.4 Multi-cluster setup and configuration ...... 150 3.7.5 Multi-cluster scenario ...... 150 3.7.6 Add security at the GPFS communication network on Bronx cluster ...... 155 3.7.7 Verify GPFS daemon is active on all nodes of both clusters ...... 157 3.8 Disaster recovery using GPFS replication ...... 162 3.8.1 Requirements: Hardware, software, network, storage ...... 162 3.8.2 GPFS configuration...... 162 3.8.3 GPFS configuration diagram...... 163 3.8.4 Set up and configure GPFS DR cluster ...... 164
Chapter 4. Management and maintenance ...... 189 4.1 Migration and update ...... 190 4.1.1 Migrating GPFS from 3.2 or 3.3 to 3.4 ...... 190 4.1.2 Applying corrective fixes to GPFS...... 193 4.1.3 Migrating data to new storage...... 195 4.1.4 Reinstalling a node (mmsrdfs) ...... 199 4.2 Managing a GPFS cluster ...... 200
4.2.1 Adding a node to the cluster ...... 200 4.2.2 Adding a disk to a file system ...... 203 4.2.3 GPFS network usage ...... 206 4.2.4 Adding a remote cluster ...... 207 4.2.5 Adding or removing a file system ...... 210 4.2.6 Enabling GPFS replication ...... 213 iv Implementing the IBM General Parallel File System (GPFS) in a Cross-Platform Environment 4.2.7 Exporting or importing a file system ...... 217 4.3 Reducing file system fragmentation: The mmdefragfs command ...... 222 4.4 Optimizing extended attributes: The fastea option ...... 224 4.5 Setting up GPFS callback functionality: The callback commands...... 224 4.6 Monitoring GPFS configuration status: The SNMPD protocol...... 226 4.7 SSH configuration ...... 227
Chapter 5. Performance and tuning ...... 229 5.1 GPFS architecture...... 230 5.2 NSD considerations ...... 231 5.3 GPFS parametrization...... 231 5.3.1 File system parametrization ...... 231 5.3.2 Block allocation methods ...... 233 5.3.3 GPFS general parametrization ...... 234 5.4 Network parametrization ...... 237 5.5 Parametrization ...... 247 5.5.1 Controlling the bandwidth ...... 247 5.5.2 Controlling GPFS parallelism ...... 248 5.5.3 Linux considerations ...... 249 5.5.4 Storage device considerations ...... 250 5.6 Monitoring performance ...... 250
Chapter 6. Problem determination...... 253 6.1 Problem determination process ...... 254 6.1.1 Defining the problem...... 254 6.1.2 Gathering information from the user ...... 254 6.1.3 Gathering application, GPFS, and system information ...... 255 6.2 GPFS debug commands...... 258 6.2.1 Data collection commands ...... 258 6.2.2 Data analysis commands and scripts ...... 263 6.2.3 mmfs logs ...... 265 6.3 GPFS problem scenarios ...... 266 6.3.1 Considerations ...... 266 6.3.2 Scenario 1: Upgrading GPFS ...... 268 6.3.3 Scenario 2: Analyzing waiters...... 269 6.3.4 Scenario 3: Application failure ...... 271 6.3.5 Scenario 4: GPFS file system hang ...... 273 6.3.6 Scenario 5: File system unmounting...... 275
Chapter 7. IBM Power Systems virtualization and GPFS ...... 279 7.1 IBM Power Systems (System p) ...... 280 7.1.1 Introduction ...... 280 7.1.2 Virtual I/O Server (VIOS)...... 281 7.1.3 VSCSI and NPIV...... 283 7.1.4 Virtual SCSI target adapters (VSCSI) ...... 283 7.1.5 N_Port ID Virtualization (NPIV)...... 300 7.2 Shared Ethernet Adapter and Host Ethernet Adapter...... 306 7.2.1 Shared Ethernet Adapter (SEA) ...... 306 7.2.2 Host Ethernet Adapter (HEA) ...... 313
Chapter 8. Information lifecycle management (ILM) ...... 317 8.1 Explaining the ILM concept ...... 318 8.1.1 Snapshot management tasks ...... 318 8.1.2 Storage pools ...... 319
Contents v 8.1.3 File sets...... 322 8.1.4 Policies and rules ...... 326 8.1.5 ILM data flow...... 328 8.2 Tivoli Storage Manager ...... 329 8.2.1 Preparing the Tivoli Storage Manager server ...... 330 8.2.2 Tivoli Storage Manager installation...... 332 8.2.3 Administrator Center installation ...... 339 8.2.4 Tivoli Storage Manager server configuration ...... 342 8.2.5 Tivoli Storage Manager client configuration ...... 344 8.3 Information lifecycle management scenarios ...... 346 8.3.1 Working with snapshots ...... 347 8.3.2 Working with storage pools and policies...... 350 8.3.3 Working with external storage pool and GPFS policies ...... 360 8.3.4 Working with hierarchical storage management ...... 363 8.3.5 Working with hierarchical storage management and GPFS policies ...... 367 8.3.6 End-to-end implementation of ILM and HSM ...... 371 8.4 Backup and restore ...... 376 8.4.1 GPFS backup tools ...... 376 8.4.2 GPFS advanced backup tools ...... 382 8.4.3 Conclusions ...... 384 8.5 Advanced external pool scenario ...... 384 8.5.1 Off-line compressed storage...... 385 8.5.2 Implementation scenario...... 385
Chapter 9. Disaster recovery using GPFS ...... 393 9.1 Disaster recovery solution using GPFS replication ...... 394 9.1.1 Configuration...... 394 9.1.2 Characteristics of this DR configuration ...... 395 9.2 The GPFS mmfsctl command ...... 396
Related publications ...... 397 IBM Redbooks ...... 397 Other publications ...... 397 Online resources ...... 398 How to get Redbooks...... 399 Help from IBM ...... 399
Index ...... 401
vi Implementing the IBM General Parallel File System (GPFS) in a Cross-Platform Environment Notices
This information was developed for products and services offered in the U.S.A.
IBM may not offer the products, services, or features discussed in this document in other countries. Consult your local IBM representative for information on the products and services currently available in your area. Any reference to an IBM product, program, or service is not intended to state or imply that only that IBM product, program, or service may be used. Any functionally equivalent product, program, or service that does not infringe any IBM intellectual property right may be used instead. However, it is the user's responsibility to evaluate and verify the operation of any non-IBM product, program, or service.
IBM may have patents or pending patent applications covering subject matter described in this document. The furnishing of this document does not give you any license to these patents. You can send license inquiries, in writing, to: IBM Director of Licensing, IBM Corporation, North Castle Drive, Armonk, NY 10504-1785 U.S.A.
The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law: INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF NON-INFRINGEMENT, MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Some states do not allow disclaimer of express or implied warranties in certain transactions, therefore, this statement may not apply to you.
This information could include technical inaccuracies or typographical errors. Changes are periodically made to the information herein; these changes will be incorporated in new editions of the publication. IBM may make improvements and/or changes in the product(s) and/or the program(s) described in this publication at any time without notice.
Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites. The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk.
IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you.
Information concerning non-IBM products was obtained from the suppliers of those products, their published announcements or other publicly available sources. IBM has not tested those products and cannot confirm the accuracy of performance, compatibility or any other claims related to non-IBM products. Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products.
This information contains examples of data and reports used in daily business operations. To illustrate them as completely as possible, the examples include the names of individuals, companies, brands, and products. All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental.
COPYRIGHT LICENSE:
This information contains sample application programs in source language, which illustrate programming techniques on various operating platforms. You may copy, modify, and distribute these sample programs in any form without payment to IBM, for the purposes of developing, using, marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written. These examples have not been thoroughly tested under all conditions. IBM, therefore, cannot guarantee or imply reliability, serviceability, or function of these programs.
© Copyright IBM Corp. 2011. All rights reserved. vii Trademarks
IBM, the IBM logo, and ibm.com are trademarks or registered trademarks of International Business Machines Corporation in the United States, other countries, or both. These and other IBM trademarked terms are marked on their first occurrence in this information with the appropriate symbol (® or ™), indicating US registered or common law trademarks owned by IBM at the time this information was published. Such trademarks may also be registered or common law trademarks in other countries. A current list of IBM trademarks is available on the Web at http://www.ibm.com/legal/copytrade.shtml
The following terms are trademarks of the International Business Machines Corporation in the United States, other countries, or both:
AFS® IBM® Redbooks® AIX 5L™ InfoSphere™ Redbooks (logo) ® AIX® LoadLeveler® System p® BladeCenter® NetView® System Storage® Blue Gene® POWER Hypervisor™ System x® Cognos® Power Systems™ System z® DB2® POWER6® Tivoli® DS4000® PowerHA™ TotalStorage® Enterprise Storage Server® PowerPC® WebSphere® ESCON® PowerVM™ XIV® FlashCopy® POWER® z/OS® GPFS™ pSeries® zSeries® HACMP™ pureScale™
The following terms are trademarks of other companies:
Intel Xeon, Intel, Intel logo, Intel Inside logo, and Intel Centrino logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States and other countries.
Linear Tape-Open, LTO, Ultrium, the LTO Logo and the Ultrium logo are trademarks of HP, IBM Corp. and Quantum in the U.S. and other countries.
Microsoft, Windows NT, Windows, and the Windows logo are trademarks of Microsoft Corporation in the United States, other countries, or both.
Snapshot, and the NetApp logo are trademarks or registered trademarks of NetApp, Inc. in the U.S. and other countries.
Java, and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle and/or its affiliates.
UNIX is a registered trademark of The Open Group in the United States and other countries.
Linux is a trademark of Linus Torvalds in the United States, other countries, or both.
Other company, product, or service names may be trademarks or service marks of others.
viii Implementing the IBM General Parallel File System (GPFS) in a Cross-Platform Environment Preface
This IBM® Redbooks® publication provides a documented deployment model for IBM GPFS™ in a cross-platform environment with IBM Power Systems™, Linux, and Windows servers. With IBM GPFS, customers can have a planned foundation for file systems management for cross-platform access solutions.
This book examines the functional, integration, simplification, and usability changes with GPFS v3.4. It can help the technical teams provide file system management solutions and technical support with GPFS, based on Power Systems virtualized environments for cross-platform file systems management.
The book provides answers to your complex file systems management requirements, helps you maximize file system availability, and provides expert-level documentation to transfer the how-to skills to the worldwide support teams.
The audience for this book is the technical professional (IT consultants, technical support staff, IT architects, and IT specialists) who is responsible for providing file system management solutions and support for cross-platform environments that are based primarily on Power Systems.
The team who wrote this book
This book was produced by a team of specialists from around the world working at the International Technical Support Organization, Poughkeepsie Center.
Front row (left to right): Wan Hee Kim, Paulo Queiroz, Andrei Vlad, Matteo Barzaghi Back row (left to right): Randy Brewster, Dino Quintero (Project Leader), Robert Simon, Steve Normann
© Copyright IBM Corp. 2011. All rights reserved. ix Dino Quintero is a technical Project Leader and a IT Generalist with the International Technical Support Organization (ITSO) in Poughkeepsie, NY. His areas of expertise include enterprise continuous availability planning and implementation, enterprise systems management, virtualization, and clustering solutions. He is currently an Open Group Master Certified IT Specialist - Server Systems. He holds a Masters degree in Computing Information
Systems, and a Bachelor of Science in Computer Science from Marist College.
Matteo Barzaghi is a Support Center Representative in Italy. He has 13 years of experience at IBM, mainly working with the Power Systems software products (AIX®, PowerHA™). He worked for four years in the field for a large bank customer as an AIX System Administrator. He is a former PowerHA Virtual Front End Team member, and now works for Virtual Front End Poughkeepsie European Team supporting all products, mainly GPFS, for all Europe, Africa, and Middle East customers. He has written extensively about GPFS maintenance.
Randy Brewster is a GPFS Software Test Specialist at IBM in Poughkeepsie, New York. He has 18 years of experience in pSeries® systems and clustering software. Prior to his work on pSeries systems, he spent over 10 years on zSeries® systems. He has worked at IBM for over 28 years. His areas of expertise include clustering software, GPFS, HAGEO, HACMP™ and Blue Gene®. He has written extensively on GPFS and DB2®. He hold a Master of Science in Computer Science from the City University of New York.
Wan Hee Kim is a Technical Sales Specialist in Korea. He has 5 years of experience with IBM. He holds LPI, OCP, and IBM Certification 061,086,071. His areas of expertise include open source technology with e1350 solution, based on System x® and BladeCenter® System. In particular, he is a technical sales focal point of the GPFS solution in Korea. He has vast experience in complex GPFS solutions with various storage systems in the public and media industries. He also hosted a major IBM Business Partner Residency Program regard GPFS, xCAT, and Virtualization. Recently, his focus is on cross-platform technology, system management, and cloud computing infrastructure.
Steve Normann is a Senior Software Engineer in MTS working in Poughkeepsie, New York. He has worked with IBM since 1984. He currently is a Team Leader for GPFS in the Software Technical Support Group, which supports the High Performance Clustering software (LoadLeveler®, CSM, GPFS, RSCT, and PPE) and High Performance Clustering software.
Paulo Queiroz is an IT Specialist with IBM in Brazil. He has 10 years of experience in UNIX and Linux, ranging from systems design and development to systems support. His areas of expertise include AIX Power Systems, AIX, GPFS, RHCS, KVM, and Linux. He is a Certified Advanced Technical Expert for Power Systems with AIX (CATE) and a Red Hat Certified Engineer (RHCE).
Robert Simon is a Senior Software Engineer in STG working in Poughkeepsie, New York. He has worked with IBM since 1987. He currently is a Team Leader in the Software Technical Support Group, which supports the High Performance Clustering software (LoadLeveler, CSM, GPFS, RSCT and PPE). He has extensive experience with IBM System p® hardware, AIX, HACMP, and High Performance Clustering software. He has participated in the development of four other IBM Redbooks publications.
Andrei Vlad is an IT Specialist with IBM in Romania, working in the Sever and Storage department since 2002. His areas of expertise include Linux performance and clustering, GPFS, CSM, xCAT, and HPC infrastructure. He has implemented several GPFS and CSM clusters, and provided worldwide customer training. Currently, his focus is on developing HPC courses and workshops. He is certified in AIX and Linux, and has a Masters degree in Electronic Engineering from Polytehnical University in Bucharest, Romania. He is currently pursuing a Ph.D. in Electronic Engineering.
x Implementing the IBM General Parallel File System (GPFS) in a Cross-Platform Environment Thanks to the following people for their contributions to this project: David Bennin, Ella Buslovich, Richard Conway, Diane Sherman International Technical Support Organization, Poughkeepsie Center
Gordon McPheeters, Heena Raval, Kuei-Yu Wang-Knop, Puneet Chaudhary, Scott Fadden, Sridhar Murthy, Scott Denham IBM USA
Octavian Lascu, Andrei Socoliuc IBM Romania
Steve Schormann IBM Canada
Rik Foote IBM Australia
Andre Posthuma IBM UK
Now you can become a published author, too!
Here's an opportunity to spotlight your skills, grow your career, and become a published author - all at the same time! Join an ITSO residency project and help write a book in your area of expertise, while honing your experience using leading-edge technologies. Your efforts will help to increase product acceptance and customer satisfaction, as you expand your network of technical contacts and relationships. Residencies run from two to six weeks in length, and you can participate either in person or as a remote resident working from your home base.
Find out more about the residency program, browse the residency index, and apply online at: ibm.com/redbooks/residencies.html
Comments welcome
Your comments are important to us!
We want our books to be as helpful as possible. Send us your comments about this book or other IBM Redbooks publications in one of the following ways: Use the online Contact us review Redbooks form found at: ibm.com/redbooks Send your comments in an email to: [email protected]