Mellanox OFED for Linux User Manual Rev 2.3-1.0.1 Last Updated: November 06, 2014 www.mellanox.com Rev 2.3-1.0.1 2 Mellanox Technologies Document Number: 2877 Rev 2.3-1.0.1 Table of Contents Table of Contents . 3 List of Figures . 6 List of Tables . 7 Chapter 1 Introduction . 13 1.1 Overview . 13 1.2 Stack Architecture . 13 1.2.1 mlx4 VPI Driver . 14 1.2.2 mlx5 Driver . 14 1.2.3 Mid-layer Core . 15 1.2.4 ULPs . 15 1.2.5 MPI . 16 1.2.6 InfiniBand Subnet Manager . 16 1.2.7 Diagnostic Utilities . 16 1.2.8 Mellanox Firmware Tools. 16 1.3 Mellanox OFED Package . 17 1.3.1 ISO Image . 17 1.3.2 Software Components . 17 1.3.3 Firmware . 18 1.3.4 Directory Structure . 18 1.4 Module Parameters . 19 1.4.1 mlx4 Module Parameters . 19 1.4.2 mlx5 Module Parameters . 21 1.5 Device Capabilities . 21 Chapter 2 Installation . 22 2.1 Hardware and Software Requirements . 22 2.2 Downloading Mellanox OFED . 22 2.3 Installing Mellanox OFED. 23 2.3.1 Installation Procedure . 24 2.3.2 Installation Results . 26 2.3.3 Installation Logging . 26 2.3.4 openibd Script . 26 2.3.5 mlnxofedinstall Return Codes. 27 2.4 Uninstalling Mellanox OFED . 27 2.5 Installing MLNX_OFED using YUM . 27 2.5.1 Setting up MLNX_OFED YUM Repository . 27 2.5.2 Installing MLNX_OFED using the YUM Tool . 29 2.5.3 Uninstalling Mellanox OFED using the YUM Tool. 30 2.6 Installing MLNX_OFED using apt-get . 30 2.6.1 Setting up MLNX_OFED apt-get Repository. 30 2.6.2 Installing MLNX_OFED using the apt-get Tool . 30 2.6.3 Uninstalling Mellanox OFED using the apt-get Tool . 31 2.7 Updating Firmware After Installation . 31 Mellanox Technologies 3 Rev 2.3-1.0.1 2.8 UEFI Secure Boot . 32 2.8.1 Enrolling Mellanox's x.509 Public Key On your Systems . 32 2.8.2 Removing Signature from kernel Modules . 32 2.9 Performance Tuning . 33 Chapter 3 Features Overview and Configuration. 34 3.1 Ethernet Network. 34 3.1.1 Interface. 34 3.1.2 Quality of Service (QoS). 39 3.1.3 Quantized Congestion Notification (QCN). 47 3.1.4 Ethtool . 50 3.1.5 Checksum Offload . 52 3.1.6 RDMA over Converged Ethernet (RoCE) . 52 3.1.7 Explicit Congestion Notification (ECN). 60 3.1.8 Power Management. 61 3.1.9 RSS Hash Function . 62 3.1.10 Time-Stamping . 62 3.1.11 Flow Steering . 67 3.2 InfiniBand Network . 72 3.2.1 Interface. 72 3.2.2 OpenSM . 73 3.2.3 Quality of Service (QoS). 120 3.2.4 Secure Host . 124 3.2.5 Upper Layer Protocols . 126 3.2.6 Advanced Transport . 150 3.2.7 Optimized Memory Access. 152 3.2.8 Mellanox PeerDirect™ . 160 3.3 Storage Protocols . 161 3.3.1 SCSI RDMA Protocol (SRP) . 161 3.3.2 iSCSI Extensions for RDMA (iSER) . 170 3.3.3 Lustre. 170 3.4 Virtualization . ..