Front cover Implementing InfiniBand on IBM System p Understanding and exploiting InfiniBand HPC and commercial solution explored AIX 5L V5.3 and SLES 9 implementation Dino Quintero Norbert Conrad Rob Desjarlais Marc-Eric Kahle Jung-Hoon Kim Hoang-Nam Nguyen Tony Pirraglia Fernando Pizzano Robert Simon Shi Lei Yao Octavian Lascu ibm.com/redbooks International Technical Support Organization Implementing InfiniBand on IBM System p September 2007 SG24-7351-00 Note: Before using this information and the product it supports, read the information in “Notices” on page ix. First Edition (September 2007) This edition applies to Version 5, Release 3, Modification 4, APAR IY84006 of AIX 5L, of SUSE LINUX Enterprise Server 9 for POWER Service Pack 3, SLES 9 SP3 and Release 6, Version 1.0 and APAR MB01795 of the Hardware Management Console (HMC). © Copyright International Business Machines Corporation 2007. All rights reserved. Note to U.S. Government Users Restricted Rights -- Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp. Contents Notices . ix Trademarks . x Preface . xi The team that wrote this book . xi Become a published author . xiv Comments welcome. xv Part 1. InfiniBand architecture. 1 Chapter 1. Introduction. 3 1.1 Introduction to InfiniBand. 4 Chapter 2. Introduction to InfiniBand technology . 7 2.1 A technical introduction to InfiniBand . 9 2.2 Markets . 10 2.3 Application clustering . 11 2.4 I/O architectures: fabric versus bus. 12 2.4.1 Shared bus architecture . 13 2.4.2 New interconnects compliment InfiniBand . 13 2.4.3 Bandwidth out of the box. 14 2.5 InfiniBand technical overview . 14 2.6 InfiniBand layers . 16 2.6.1 Physical layer . 16 2.6.2 Link layer. 17 2.6.3 Network layer . 19 2.6.4 Transport layer . 20 2.6.5 Upper layers . 21 2.6.6 InfiniBand elements. 22 2.7 InfiniBand architecture. 23 2.7.1 Channel adapters . 23 2.7.2 The IB switch. 23 2.8 InfiniBand components . 24 2.8.1 Router . 24 2.8.2 Subnet manager . 25 2.8.3 Management infrastructure . 25 2.9 InfiniBand support for the Direct Access Programming Library (DAPL) . 26 2.10 Adapter sharing . 26 2.11 Summary . 28 © Copyright IBM Corp. 2007. All rights reserved. iii Chapter 3. InfiniBand hardware overview and implementation . 29 3.1 Limitations and considerations . 30 3.2 Features and benefits of InfiniBand on System p . 30 3.2.1 AIX supported environments. 33 3.2.2 Linux on System p: SLES9 SP3 supported environments . 34 3.3 Hardware requirements. 35 3.4 Hardware Management Console (HMC) . 37 3.4.1 Cluster Ready Hardware Server (CRHS) mode . 38 3.4.2 Why move the DHCP server . 39 3.5 Supported System p servers . 40 3.6 Supported host channel adapters (HCA) . 41 3.6.1 Sharing the host channel adapter (HCA) . 47 3.7 Logical partitioning (LPAR) . 48 3.7.1 Make it smaller (micro partitions) . 49 3.8 Cisco InfiniBand switches . 50 3.8.1 Cisco SFS 7000P InfiniBand Server Switch . 50 3.8.2 Cisco SFS 7008P InfiniBand Server Switch . 51 3.8.3 Using the switch: user and passwords . 53 3.9 InfiniBand cables . 53 3.9.1 Cabling with octopus cables . 55 3.10 Management server . 57 3.10.1 A private network DHCP IP configuration versus a static IP configuration . 58 3.11 IBM Network Manager (IBM NM) . 59 Part 2. Implementation . 71 Chapter 4. InfiniBand on AIX 5L. 73 4.1 InfiniBand on AIX. 74 4.1.1 Hardware requirements. 74 4.1.2 AIX software requirements . 74 4.1.3 Overview of cluster software components . 74 4.2 InfiniBand on System p5 running AIX . 86 4.2.1 Implementation of InfiniBand architecture (IBA) on System p5. 86 4.2.2 IP over InfiniBand (IPoIB) implementation . 89 4.2.3 AIX InfiniBand filesets and components . 91 4.3 Test cluster layout and description . 94 4.3.1 Planning for installation . 94 4.3.2 Our environment . 95 4.4 Installation and configuration of the AIX CSM Management server . 97 4.4.1 Installation of AIX . 97 4.4.2 Installing the AIX 5L management server . 98 4.4.3 NIM configuration . 100 iv Implementing InfiniBand on IBM System p 4.4.4 Updating NIM . 103 4.4.5 Verify InfiniBand filesets . 103 4.5 Installation and configuration of AIX nodes . 104 4.5.1 Pre-installation tasks . 104 4.5.2 Get network adapter information. 110 4.5.3 Further configuration . 112 4.5.4 Preparing NIM for nodes (clients) installation . 113 4.5.5 Verification of the AIX installation . 115 4.5.6 Configuring InfiniBand adapters on AIX nodes . 117 4.5.7 Verification of the InfiniBand configuration . 121 4.6 GPFS installation and configuration . 131 4.6.1 Communication considerations for GPFS . 131 4.6.2 GPFS installation . 132 4.6.3 Monitoring GPFS over InfiniBand . 138 Chapter 5. IBM System p cluster with InfiniBand and SUSE SLES 9 . 141 5.1 InfiniBand considerations for SLES 9 . ..