IBM Spectrum Scale and IBM Elastic Storage System Network Guide
Total Page:16
File Type:pdf, Size:1020Kb
Front cover IBM Spectrum Scale and IBM Elastic Storage System Network Guide Kedar Karmarkar John Lewars Sandeep R. Patil Sandeep Naik Kevin Gildea Rakesh Chutke Larry Coyne In partnership with IBM Academy of Technology Redpaper International Technical Support Organization IBM Spectrum Scale and IBM Elastic Storage System Network Guide February 2021 REDP-5484-00 Note: Before using this information and the product it supports, read the information in “Notices” on page v. First Edition (February 2021) This edition applies to Version 5, Release 1, Modification 0 of IBM Spectrum Scale and Version 5, Release 3, Modification 4 of IBM Elastic Storage Server. This document was created or updated on February 17, 2021. © Copyright International Business Machines Corporation 2021. All rights reserved. Note to U.S. Government Users Restricted Rights -- Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp. Contents Notices . .v Trademarks . vi Preface . vii Authors. vii Now you can become a published author, too! . viii Comments welcome. ix Stay connected to IBM Redbooks . ix Chapter 1. IBM Spectrum Scale introduction . 1 1.1 How is IBM Spectrum Scale different from other network storage technologies. 2 1.2 Types of network traffic in IBM Spectrum Scale . 3 1.2.1 Control traffic. 4 1.2.2 Data traffic. 5 1.2.3 Administration traffic . 6 1.3 High-level network flow for data and control traffic . 6 1.3.1 IBM Spectrum Scale supported networks. 6 1.3.2 IBM Spectrum Scale expels . 7 Chapter 2. Network planning and best practices for IBM Spectrum Scale and IBM Elastic Storage System. 9 2.1 Network planning and sizing . 10 2.2 Bandwidth requirements . 10 2.2.1 Aggregate cluster bandwidth . 10 2.2.2 Single node bandwidth . 14 2.3 Deciding network bandwidth planning. 14 2.4 Network design considerations . 17 2.5 OSI Layer . 20 2.5.1 OSI Layer 1: Physical layer. 20 2.5.2 OSI Layer 2: Link layer . 21 2.5.3 OSI Layer 3+4. 22 2.6 Network multipathing. 25 2.6.1 Layer 2 multipathing . 25 2.6.2 Hosts-to-switch side . 25 2.6.3 Switch side . 26 2.6.4 Layer 3 multipathing . 27 2.7 Network infrastructure . 28 2.7.1 Common network topologies. 28 2.7.2 LAN and WAN considerations. 30 2.7.3 Network monitoring tools. 31 2.7.4 Flow control techniques . 31 2.8 Network considerations for special use cases . 34 Chapter 3. Implementation recommendations . 37 3.1 Network verification tools . 38 3.1.1 IBM Spectrum Scale mmnetverify command . 38 3.1.2 IBM Spectrum Scale nsdperf tool . 38 3.2 Ethernet Network Adaptor configuration settings . 38 3.2.1 Ring buffers. 39 © Copyright IBM Corp. 2021. All rights reserved. iii 3.2.2 Maximum transmission unit. 39 3.2.3 Transmission queue length. 40 3.2.4 Flow control. 40 3.3 TCP/IP settings . 41 3.3.1 TCP/IP send and receive buffers . 41 3.3.2 TCP/IP window scaling . 41 3.3.3 TCP/IP settings for network with packet loss . 42 3.3.4 Other TCP/IP setting. 42 3.4 Remote Direct Memory Access settings . 42 3.4.1 Settings to enable RDMA between nodes . 42 3.4.2 Tuning for RDMA . 43 3.5 Network multipathing settings . 43 Chapter 4. Network monitoring and troubleshooting . 45 4.1 Network-related issues overview . 46 4.2 Network monitoring with IBM Spectrum Scale GUI. 47 4.2.1 Monitoring Network Events Using the GUI . 47 4.2.2 Monitoring Network Performance Using the GUI . 47 4.2.3 Network Monitoring with IBM Spectrum Scale mmhealth command . 48 4.2.4 IBM Spectrum Scale call back commands to monitor network-related events . 49 4.3 Network troubleshooting with IBM Spectrum Scale mmdiag command . 50 4.4 Verifying network with IBM Spectrum Scale mmnetverify command . 53 4.5 Testing network performance with IBM Spectrum Scale nsdperf tool. 59 4.6 IBM Spectrum Scale mmuserauth check . 63 4.7 Using IBM Spectrum Scale log messages to debug network-related issues . 64 4.8 Node expels because of network issues. 65 4.9 Understanding and resolving the effect of network packet loss . 74 4.9.1 Overview and effect . 74 4.10 Resource issues . 79 Appendix A. IBM Spectrum Scale lease configuration variables effect on expel timing flows. 83 Lease configuration parameters related to disk leasing. 84 failureDetectionTime . ..