Front cover Implementing and Managing InfiniBand Coupling Links on IBM System z Concepts, terminology, and supported topologies Planning, migration, and implementation guidance Performance information Frank Kyne Hua Bin Chu George Handera Marek Liedel Masaya Nakagawa Iain Neville Christian Zass ibm.com/redbooks International Technical Support Organization Implementing and Managing InfiniBand Coupling Links on IBM System z January 2014 SG24-7539-03 Note: Before using this information and the product it supports, read the information in “Notices” on page vii. Fourth Edition (January 2014) This edition applies to the InfiniBand features that are available on IBM System z servers. © Copyright International Business Machines Corporation 2008, 2012, 2014. All rights reserved. Note to U.S. Government Users Restricted Rights -- Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp. Contents Notices . vii Trademarks . viii Preface . ix Authors. ix Now you can become a published author, too! . .x Comments welcome. xi Stay connected to IBM Redbooks publications . xi Summary of changes. xiii January 2014, Fourth Edition . xiii March 2012, Third Edition . xiii Chapter 1. Introduction to InfiniBand on System z . 1 1.1 Objective of this book . 2 1.2 InfiniBand architecture . 2 1.2.1 Physical layer . 3 1.3 IBM System z InfiniBand implementation . 5 1.3.1 Host channel adapters . 5 1.3.2 Processor-specific implementations . 6 1.4 InfiniBand benefits. 7 1.5 The importance of an efficient coupling infrastructure . 9 1.5.1 Coupling link performance factors. 11 1.5.2 PSIFB 12X and 1X InfiniBand links. 12 1.6 Terminology . 13 1.7 Structure of this book . 15 Chapter 2. InfiniBand technical description . 17 2.1 InfiniBand connectivity . 18 2.2 InfiniBand fanouts . 19 2.2.1 Adapter types . 20 2.3 Fanout plugging . 22 2.3.1 Fanout plugging rules for zEnterprise 196 . 22 2.3.2 Fanout plugging rules for zEnterprise 114 . 23 2.3.3 Fanout plugging rules for System z10 EC. 24 2.3.4 Fanout plugging rules for System z10 BC. 25 2.4 Adapter ID assignment and VCHIDs. 26 2.4.1 Adapter ID assignment . 26 2.4.2 VCHID - Virtual Channel Identifier . 28 2.5 InfiniBand coupling links . 30 2.5.1 12X PSIFB coupling links on System z9. 30 2.5.2 12X PSIFB coupling links on System z10. 30 2.5.3 PSIFB Long Reach coupling links on System z10 . 31 2.5.4 12X PSIFB coupling links on z196 and z114 . 32 2.5.5 Long Reach PSIFB coupling links on zEnterprise 196 and 114 . 33 2.5.6 PSIFB coupling links and Server Time Protocol . 34 © Copyright IBM Corp. 2008, 2012, 2014. All rights reserved. iii 2.6 InfiniBand cables. 34 Chapter 3. Preinstallation planning. 37 3.1 Planning considerations . 38 3.2 CPC topology . 38 3.2.1 Coexistence . 39 3.2.2 Supported coupling link types . 40 3.3 Hardware and software prerequisites . 42 3.3.1 Hardware prerequisites. 42 3.3.2 Software prerequisites . 43 3.4 Considerations for Server Time Protocol . 45 3.4.1 Considerations for STP with PSIFB coupling links . 45 3.5 Multisite sysplex considerations . 48 3.6 Planning for future nondisruptive growth. 49 3.7 Physical and logical coupling link capacity planning . 50 3.7.1 Availability . 50 3.7.2 Connectivity . 52 3.7.3 Capacity and performance . 55 3.8 Physical Coupling link addressing. 58 3.9 Cabling considerations . 61 Chapter 4. Migration planning . 63 4.1 Migration considerations . 64 4.1.1 Connectivity considerations . 65 4.2 Introduction to the scenario notation . 66 4.3 Scenario 1 . 68 4.4 Scenario 2 . 76 4.5 Scenario 3 . 86 4.6 Scenario 4 . 95 4.7 Scenario 5 . 108 4.8 Concurrent switch between IFB modes . ..