DOCSLIB.ORG

Implementing and Managing Infiniband Coupling Links on IBM System Z

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Implementing and Managing Infiniband Coupling Links on IBM System Z Front cover Implementing and Managing InfiniBand Coupling Links on IBM System z Concepts, terminology, and supported topologies Planning, migration, and implementation guidance Performance information Frank Kyne Hua Bin Chu George Handera Marek Liedel Masaya Nakagawa Iain Neville Christian Zass ibm.com/redbooks International Technical Support Organization Implementing and Managing InfiniBand Coupling Links on IBM System z January 2014 SG24-7539-03 Note: Before using this information and the product it supports, read the information in “Notices” on page vii. Fourth Edition (January 2014) This edition applies to the InfiniBand features that are available on IBM System z servers. © Copyright International Business Machines Corporation 2008, 2012, 2014. All rights reserved. Note to U.S. Government Users Restricted Rights -- Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp. Contents Notices . vii Trademarks . viii Preface . ix Authors. ix Now you can become a published author, too! . .x Comments welcome. xi Stay connected to IBM Redbooks publications . xi Summary of changes. xiii January 2014, Fourth Edition . xiii March 2012, Third Edition . xiii Chapter 1. Introduction to InfiniBand on System z . 1 1.1 Objective of this book . 2 1.2 InfiniBand architecture . 2 1.2.1 Physical layer . 3 1.3 IBM System z InfiniBand implementation . 5 1.3.1 Host channel adapters . 5 1.3.2 Processor-specific implementations . 6 1.4 InfiniBand benefits. 7 1.5 The importance of an efficient coupling infrastructure . 9 1.5.1 Coupling link performance factors. 11 1.5.2 PSIFB 12X and 1X InfiniBand links. 12 1.6 Terminology . 13 1.7 Structure of this book . 15 Chapter 2. InfiniBand technical description . 17 2.1 InfiniBand connectivity . 18 2.2 InfiniBand fanouts . 19 2.2.1 Adapter types . 20 2.3 Fanout plugging . 22 2.3.1 Fanout plugging rules for zEnterprise 196 . 22 2.3.2 Fanout plugging rules for zEnterprise 114 . 23 2.3.3 Fanout plugging rules for System z10 EC. 24 2.3.4 Fanout plugging rules for System z10 BC. 25 2.4 Adapter ID assignment and VCHIDs. 26 2.4.1 Adapter ID assignment . 26 2.4.2 VCHID - Virtual Channel Identifier . 28 2.5 InfiniBand coupling links . 30 2.5.1 12X PSIFB coupling links on System z9. 30 2.5.2 12X PSIFB coupling links on System z10. 30 2.5.3 PSIFB Long Reach coupling links on System z10 . 31 2.5.4 12X PSIFB coupling links on z196 and z114 . 32 2.5.5 Long Reach PSIFB coupling links on zEnterprise 196 and 114 . 33 2.5.6 PSIFB coupling links and Server Time Protocol . 34 © Copyright IBM Corp. 2008, 2012, 2014. All rights reserved. iii 2.6 InfiniBand cables. 34 Chapter 3. Preinstallation planning. 37 3.1 Planning considerations . 38 3.2 CPC topology . 38 3.2.1 Coexistence . 39 3.2.2 Supported coupling link types . 40 3.3 Hardware and software prerequisites . 42 3.3.1 Hardware prerequisites. 42 3.3.2 Software prerequisites . 43 3.4 Considerations for Server Time Protocol . 45 3.4.1 Considerations for STP with PSIFB coupling links . 45 3.5 Multisite sysplex considerations . 48 3.6 Planning for future nondisruptive growth. 49 3.7 Physical and logical coupling link capacity planning . 50 3.7.1 Availability . 50 3.7.2 Connectivity . 52 3.7.3 Capacity and performance . 55 3.8 Physical Coupling link addressing. 58 3.9 Cabling considerations . 61 Chapter 4. Migration planning . 63 4.1 Migration considerations . 64 4.1.1 Connectivity considerations . 65 4.2 Introduction to the scenario notation . 66 4.3 Scenario 1 . 68 4.4 Scenario 2 . 76 4.5 Scenario 3 . 86 4.6 Scenario 4 . 95 4.7 Scenario 5 . 108 4.8 Concurrent switch between IFB modes . ..
Load more

Recommended publications
  • IBM System Z Strengths and Values
    Front cover IBM System z Strengths and Values Technical presentation of System z hardware and z/OS Enterprise-wide roles for the System z platform Cost of computing considerations Philippe Comte Andrea Corona James Guilianelli Douglas Lin Werner Meiner Michel Plouin Marita Prassolo Kristine Seigworth Eran Yona Linfeng Yu ibm.com/redbooks International Technical Support Organization IBM System z Strengths and Values January 2007 SG24-7333-00 Note: Before using this information and the product it supports, read the information in “Notices” on page ix. First Edition (January 2007) This edition applies to the IBM System z platform and IBM z/OS V1.8. © Copyright International Business Machines Corporation 2007. All rights reserved. Note to U.S. Government Users Restricted Rights -- Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp. Contents Notices . ix Trademarks . x Preface . xi The team that wrote this redbook. xi Become a published author . xiii Comments welcome. xiii Chapter 1. A business view . 1 1.1 Business drivers . 2 1.2 Impact on IT . 3 1.3 The System z platform . 7 1.3.1 Using System Z technology to reduce complexity . 7 1.3.2 Business integration and resiliency. 8 1.3.3 Managing the System z platform to meet business goals. 11 1.3.4 Security . 12 1.4 Summary . 13 Chapter 2. System z architecture and hardware platform . 15 2.1 History . 16 2.2 System z architecture . 17 2.2.1 Multiprogramming and multiprocessing . 18 2.2.2 The virtualization concept . 19 2.2.3 PR/SM and logical partitions .
    [Show full text]
  • 8. IBM Z and Hybrid Cloud
    The Centers for Medicare and Medicaid Services The role of the IBM Z® in Hybrid Cloud Architecture Paul Giangarra – IBM Distinguished Engineer December 2020 © IBM Corporation 2020 The Centers for Medicare and Medicaid Services The Role of IBM Z in Hybrid Cloud Architecture White Paper, December 2020 1. Foreword ............................................................................................................................................... 3 2. Executive Summary .............................................................................................................................. 4 3. Introduction ........................................................................................................................................... 7 4. IBM Z and NIST’s Five Essential Elements of Cloud Computing ..................................................... 10 5. IBM Z as a Cloud Computing Platform: Core Elements .................................................................... 12 5.1. The IBM Z for Cloud starts with Hardware .............................................................................. 13 5.2. Cross IBM Z Foundation Enables Enterprise Cloud Computing .............................................. 14 5.3. Capacity Provisioning and Capacity on Demand for Usage Metering and Chargeback (Infrastructure-as-a-Service) ................................................................................................................... 17 5.4. Multi-Tenancy and Security (Infrastructure-as-a-Service) .......................................................
    [Show full text]
  • IBM Z Server Time Protocol Guide
    Front cover Draft Document for Review August 3, 2020 1:37 pm SG24-8480-00 IBM Z Server Time Protocol Guide Octavian Lascu Franco Pinto Gatto Gobehi Hans-Peter Eckam Jeremy Koch Martin Söllig Sebastian Zimmermann Steve Guendert Redbooks Draft Document for Review August 3, 2020 7:26 pm 8480edno.fm IBM Redbooks IBM Z Server Time Protocol Guide August 2020 SG24-8480-00 8480edno.fm Draft Document for Review August 3, 2020 7:26 pm Note: Before using this information and the product it supports, read the information in “Notices” on page vii. First Edition (August 2020) This edition applies to IBM Server Time Protocol for IBM Z and covers IBM z15, IBM z14, and IBM z13 server generations. This document was created or updated on August 3, 2020. © Copyright International Business Machines Corporation 2020. All rights reserved. Note to U.S. Government Users Restricted Rights -- Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp. Draft Document for Review August 3, 2020 8:32 pm 8480TOC.fm Contents Notices . vii Trademarks . viii Preface . ix Authors. ix Comments welcome. .x Stay connected to IBM Redbooks . xi Chapter 1. Introduction to Server Time Protocol . 1 1.1 Introduction to time synchronization . 2 1.1.1 Insertion of leap seconds . 2 1.1.2 Time-of-Day (TOD) Clock . 3 1.1.3 Industry requirements . 4 1.1.4 Time synchronization in a Parallel Sysplex. 6 1.2 Overview of Server Time Protocol (STP) . 7 1.3 STP concepts and terminology . 9 1.3.1 STP facility . 9 1.3.2 TOD clock synchronization .
    [Show full text]
  • Availability Digest
    the Availability Digest Parallel Sysplex – Fault Tolerance from IBM April 2008 IBM’s Parallel Sysplex, HP’s NonStop server, and Stratus’ ftServer are today the primary industry fault-tolerant offerings that can tolerate any single failure, thus leading to very high levels of availability. The Stratus line of fault-tolerant computers is aimed at seamlessly protecting industry- standard servers running operating systems such as Windows, Unix, and Linux. As a result, ftServer does not compete with the other two systems, Parallel Sysplex and NonStop servers, which do compete instead in the large enterprise marketplace. In this article, we will explore IBM’s Parallel Sysplex and its features that address high availability. IBM’s Parallel Sysplex IBM’s Parallel Sysplex systems are multiprocessor clusters that can support from two to thirty-two mainframe nodes (typically S/390 or zSeries systems).1 A Parallel Sysplex system is nearly linearly scalable up to its 32-processor limit. A node may be a separate system or a logical partition (LPAR) within a system. The nodes do not have to be identical. They can be a mix of any servers that support the Parallel Sysplex environment. The nodes in a Parallel Sysplex system interact as an active/active architecture. The system allows direct, concurrent read/write access to shared data from all processing nodes without sacrificing data integrity. Furthermore, work requests associated with a single transaction or database query can be dynamically distributed for parallel execution based on available processor capacity of the nodes in the Parallel Sysplex cluster. Parallel Sysplex Architecture All nodes in a Parallel Sysplex cluster connect to a shared disk subsystem.
    [Show full text]
  • IBM Z Connectivity Handbook
    Front cover IBM Z Connectivity Handbook Octavian Lascu John Troy Anna Shugol Frank Packheiser Kazuhiro Nakajima Paul Schouten Hervey Kamga Jannie Houlbjerg Bo XU Redbooks IBM Redbooks IBM Z Connectivity Handbook August 2020 SG24-5444-20 Note: Before using this information and the product it supports, read the information in “Notices” on page vii. Twentyfirst Edition (August 2020) This edition applies to connectivity options available on the IBM z15 (M/T 8561), IBM z15 (M/T 8562), IBM z14 (M/T 3906), IBM z14 Model ZR1 (M/T 3907), IBM z13, and IBM z13s. © Copyright International Business Machines Corporation 2020. All rights reserved. Note to U.S. Government Users Restricted Rights -- Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp. Contents Notices . vii Trademarks . viii Preface . ix Authors. ix Now you can become a published author, too! . xi Comments welcome. xi Stay connected to IBM Redbooks . xi Chapter 1. Introduction. 1 1.1 I/O channel overview. 2 1.1.1 I/O hardware infrastructure . 2 1.1.2 I/O connectivity features . 3 1.2 FICON Express . 4 1.3 zHyperLink Express . 5 1.4 Open Systems Adapter-Express. 6 1.5 HiperSockets. 7 1.6 Parallel Sysplex and coupling links . 8 1.7 Shared Memory Communications. 9 1.8 I/O feature support . 10 1.9 Special-purpose feature support . 12 1.9.1 Crypto Express features . 12 1.9.2 Flash Express feature . 12 1.9.3 zEDC Express feature . 13 Chapter 2. Channel subsystem overview . 15 2.1 CSS description . 16 2.1.1 CSS elements .
    [Show full text]
  • Z10 EC System Overview Level 06B, February 2011
    System z10 Enterprise Class System Overview SA22-1084-06 Level 06b, February 2011 System z10 Enterprise Class System Overview SA22-1084-06 Level 06b, February 2011 Level 06b, February 2011 Note Before using this information and the product it supports, read the information in “Safety” on page xi, Appendix D, “Notices,” on page 151, and IBM Systems Environmental Notices and User Guide, Z125-5823. | This edition, SA22-1084-06, applies to the IBM System z10 Enterprise Class (z10 EC) server. This edition replaces | SA22-1084-05. Technical changes to the text are indicated by a vertical bar (|) to the left of the change. There might be a newer version of this document in a PDF file available on Resource Link.Goto http://www.ibm.com/servers/resourcelink and click Library on the navigation bar. A newer version is indicated by a lowercase, alphabetic letter following the form number suffix (for example: 00a, 00b, 01a, 01b). © Copyright IBM Corporation 2008, 2011. US Government Users Restricted Rights – Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp. Level 06b, February 2011 Contents Figures ..............vii Power sequence controller .........31 Additional features/functions supported ....31 Tables ...............ix Monitoring and estimating CPC power consumption and temperature .......31 Preplanning and setting up the Storage Area Safety ...............xi Network (SAN) environment .......32 Safety notices ..............xi World trade safety information .......xi Chapter 3. Software support .....33 Laser safety information ..........xi z/OS ................34 Laser compliance ...........xi z/VM ................35 z/VSE ................35 About this publication ........xiii Linux on System z ............36 What is included in this publication ......xiii TPF.................36 Revisions ..............xiii Prerequisite publications..........xiii Chapter 4.
    [Show full text]
  • Fiber Optics Based Parallel Computer Architecture
    International Journal of Scientific & Engineering Research, Volume 1, Issue 2, November-2010 1 ISSN 2229-5518 Fiber Optics Based Parallel Computer Architecture A.M V RAGHAVENDRA, B.VUDA SREENIVASARAO ABSTRACT : Computer architecture is the conceptual design and fundamental operational structure of a computer system. It's a blueprint and functional description of requirements and design implementations for the various parts of a computer, focusing largely on the way by which the central processing unit (CPU) performs internally and accesses addresses in memory. In this paper, we present an overview of parallel computer architectures and discuss the use of fiber optics for clustered or coupled processors. Presently, a number of computer systems take advantage of commercially available fiber optic technology to interconnect multiple processors, thereby achieving improved performance or reliability for a lower cost than if a single large processor had been used. Optical fiber is also used to distribute the elements of a parallel architecture over large distances; these can range from tens of meters to alleviate packaging problems to tens of kilometers for disaster recovery applications. Index Terms: Network topologies, Transputer, Sky HPC-1, CMU Wrap, Tsukuba CP-PACS/2048, Parallel Sysplex & GDPS, MANs —————————— —————————— I. INTRODUCTION will use to categorize architectures we describe in other OT all of the parallel computer architectures discussed in sections of the paper. Nthis paper use fiber optic connectivity, but they are 1. Speed - There are engineering limits to how fast any single presented as a context for those systems that are currently using processor can compute using current technology. Parallel optics. We will give a few specific examples of parallel architectures can exceed these limits by splitting up the computer systems that use optical fiber, in particular the computation among multiple processors.
    [Show full text]
  • OS/390 Parallel Sysplex Configuration Volume 1: Overview
    OS/390 Parallel Sysplex Configuration Volume 1: Overview Frank Kyne, Luiz Fadel, Michael Ferguson, Rafael Garcia, Keith George, Masayuki Kojima, Alan Murphy, Henrik Thorsen International Technical Support Organization www.redbooks.ibm.com SG24-5637-00 International Technical Support Organization SG24-5637-00 OS/390 Parallel Sysplex Configuration Volume 1: Overview August 2000 Take Note! Before using this information and the product it supports, be sure to read the general information in Appendix A, “Special Notices” on page 233. First Edition (August 2000) This edition applies to Program Name, Program Number Version, Release Number CICS/ESA, 5655-018 4.1 CICS TS for OS/390, 5655-147 1.3 DB2 for OS/390, 5655-DB2 5.1 DB2 UDB for OS/390, 5645-DB2 6.1 DB2 UDB for OS/390, 5675-DB2 7.1 DFSMS/MVS, 5695-DF1 1.5 IMS/ESA, 5695-176 5.1 IMS/ESA, 5655-158 6.1 IMS/ESA, 5655-B01 7.1 System Automation for OS/390, 5645-045 1.3 for use with: Program Name Version, Release Number OS/390, 5645-001 1.3 OS/390, 5647-A01 2.4 OS/390, 5647-A01 2.5 OS/390, 5647-A01 2.6 OS/390, 5647-A01 2.7 OS/390, 5647-A01 2.8 OS/390, 5647-A01 2.9 OS/390, 5647-A01 2.10 Comments may be addressed to: IBM Corporation, International Technical Support Organization Dept. HYJ Mail Station P099 2455 South Road Poughkeepsie, New York 12601-5400 When you send information to IBM, you grant IBM a non-exclusive right to use or distribute the information in any way it believes appropriate without incurring any obligation to you.
    [Show full text]
  • OS/390 Parallel Sysplex Configuration Volume 3: Connectivity
    OS/390 Parallel Sysplex Configuration Volume 3: Connectivity Frank Kyne, Luiz Fadel, Michael Ferguson, Rafael Garcia, Keith George, Masayuki Kojima, Alan Murphy, Henrik Thorsen International Technical Support Organization www.redbooks.ibm.com SG24-5639-00 International Technical Support Organization SG24-5639-00 OS/390 Parallel Sysplex Configuration Volume 3: Connectivity August 2000 Take Note! Before using this information and the product it supports, be sure to read the general information in Appendix B, “Special Notices” on page 167. First Edition (August 2000) This edition applies to Program Name, Program Number Version, Release Number CICS/ESA, 5655-018 4.1 CICS TS for OS/390, 5655-147 1.3 DB2 for OS/390, 5655-DB2 5.1 DB2 UDB for OS/390, 5645-DB2 6.1 DB2 UDB for OS/390, 5675-DB2 7.1 DFSMS/MVS, 5695-DF1 1.5 IMS/ESA, 5695-176 5.1 IMS/ESA, 5655-158 6.1 IMS/ESA, 5655-B01 7.1 System Automation for OS/390, 5645-045 1.3 for use with: Program Name Version, Release Number OS/390, 5645-001 1.3 OS/390, 5647-A01 2.4 OS/390, 5647-A01 2.5 OS/390, 5647-A01 2.6 OS/390, 5647-A01 2.7 OS/390, 5647-A01 2.8 OS/390, 5647-A01 2.9 OS/390, 5647-A01 2.10 Comments may be addressed to: IBM Corporation, International Technical Support Organization Dept. HYJ Mail Station P099 2455 South Road Poughkeepsie, New York 12601-5400 When you send information to IBM, you grant IBM a non-exclusive right to use or distribute the information in any way it believes appropriate without incurring any obligation to you.
    [Show full text]
  • High Availability and Scalability of Mainframe Environments Using System Z and Z/OS As Example
    robert vaupel _ of mainframe environments high availability and scalability of mainframe environments r. vaupel Robert Vaupel High Availability and Scalability of Mainframe Environments using System z and z/OS as example High Availability and Scalability of Mainframe Environments using System z and z/OS as example by Robert Vaupel Impressum Karlsruher Institut für Technologie (KIT) KIT Scientific Publishing Straße am Forum 2 D-76131 Karlsruhe www.ksp.kit.edu KIT – Universität des Landes Baden-Württemberg und nationales Forschungszentrum in der Helmholtz-Gemeinschaft Diese Veröffentlichung ist im Internet unter folgender Creative Commons-Lizenz publiziert: http://creativecommons.org/licenses/by-nc-nd/3.0/de/ KIT Scientific Publishing 2013 Print on Demand ISBN 978-3-7315-0022-3 Contents 1. Introduction 1 1.1. Motivation . 1 1.2. High Availability . 3 1.3. Scalability . 5 2. z/Architecture 9 2.1. A Little History . 9 2.2. System z CMOS Heritage . 12 2.3. System zEC12 Central Electronic Complex . 13 2.4. System zEC12 Components . 14 2.5. System z Multi Chip Module . 16 2.5.1. Memory . 17 2.5.2. Book . 18 2.5.3. Processor Characterization . 18 2.6. System z High Availability Design . 19 2.6.1. Transparent CPU Sparing . 20 2.6.2. CPU Error Detection for newer System z machines . 21 2.6.3. Redundant Array of Independent Memory . 22 2.7. System z Software and Firmware Layers . 25 2.8. Instruction Execution . 27 2.8.1. CISC versus RISC Architecture . 27 2.8.2. Register Sets . 30 2.8.3. Program Status Word . 31 2.8.4.
    [Show full text]
  • IBM System Z10 Enterprise Class Technical Guide
    Front cover IBM System z10 Enterprise Class Technical Guide Describes the Enterprise Class server and related features Addresses increasing complexity, rising costs, and energy contraints Discusses infrastructure for the data center of the future Per Fremstad Wolfgang Fries Marian Gasparovic Parwez Hamid Brian Hatfield Dick Jorna Fernando Nogal Karl-Erik Stenfors ibm.com/redbooks International Technical Support Organization IBM System z10 Enterprise Class Technical Guide November 2009 SG24-7516-02 Note: Before using this information and the product it supports, read the information in “Notices” on page xi. Third Edition (November 2009) This edition applies to the IBM System z10 Enterprise Class server, as described in IBM United States Hardware Announcement 108-794, dated October 21, 2008. © Copyright International Business Machines Corporation 2008, 2009. All rights reserved. Note to U.S. Government Users Restricted Rights -- Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp. Contents Notices . xi Trademarks . xii Preface . .xv The team who wrote this book . .xv Become a published author . xvii Comments welcome. xvii Chapter 1. Introducing the System z10 Enterprise Class . 1 1.1 Wanted: an infrastructure (r)evolution. 3 1.1.1 Simplified . 4 1.1.2 Shared . 4 1.1.3 Dynamic . 5 1.1.4 z10 at the core of a dynamic infrastructure. 6 1.1.5 Storage is part of the System z10 stack . 6 1.2 System z10 EC highlights . 7 1.3 System z10 EC Models. 9 1.3.1 Model upgrade paths . 10 1.3.2 Concurrent processing unit conversions. 10 1.4 System functions and features .
    [Show full text]
  • Handelsbanken Turns to Parallel Sysplex October 2009
    the Availability Digest www.availabilitydigest.com Handelsbanken Turns to Parallel Sysplex October 2009 Founded in 1871, Svenska Handelsbanken (www.handelsbanken.com) survived two world wars and multiple recessions to become one of the largest banks in the Nordic countries. Headquartered in Stockholm, Sweden, it has more than 460 branches in Sweden and 240 branches in the rest of Scandinavia and throughout the world, including the U.S., England, Singapore, China, and Russia. The bank offers a full range of services to corporate, institutional, and private customers. It is a global universal bank that covers the traditional areas of corporate financing, investment banking and trading as well as consumer banking. It provides corporate and individual clients with deposit products, loans, credit cards, and other banking services. Subsidiaries operate in several related areas, including life insurance, mortgages, pensions, fund management, and Internet banking. An innovative feature of Handelsbanken’s Internet services is its unique branch office web sites and direct e-mail addresses to each branch for enhanced customer service. Facing the Need for Continuous Availability Handelsbanken realized that the growing dependence by its customers on online banking and credit cards required these critical services to be continuously available. Should online banking services go down, customers could not query their account balances, execute bill payments, transfer funds between accounts, or accomplish the many other banking tasks that they were accustomed to doing from their PCs. Furthermore, if the bank’s card services failed, customers using cards issued by Handelsbanken would not be able to make in-store or online purchases nor get cash from ATM machines.
    [Show full text]