Mellanox OFED for Linux User Manual Rev 2.2-1.0.1 Last Updated: July 14, 2014 www.mellanox.com Rev 2.2-1.0.1 2 Mellanox Technologies Document Number: 2877 Rev 2.2-1.0.1 Table of Contents Table of Contents . 3 List of Figures . 10 List of Tables . 11 Chapter 1 Mellanox OFED Overview . 21 1.1 Introduction to Mellanox OFED . 21 1.2 Mellanox OFED Package . 21 1.2.1 ISO Image . 21 1.2.2 Software Components . 21 1.2.3 Firmware . 22 1.2.4 Directory Structure . 22 1.3 Architecture . 23 1.3.1 mlx4 VPI Driver . 23 1.3.2 mlx5 Driver . 24 1.3.3 Mid-layer Core . 25 1.3.4 ULPs . 25 1.3.5 MPI . 26 1.3.6 InfiniBand Subnet Manager . 26 1.3.7 Diagnostic Utilities . 26 1.3.8 Mellanox Firmware Tools. 26 1.4 Quality of Service . 27 1.5 RDMA over Converged Ethernet (RoCE). 27 1.5.1 Configuring DAPL over RoCE. 28 1.5.2 A Detailed Example . 29 Chapter 2 Installation . 34 2.1 Hardware and Software Requirements . 34 2.2 Downloading Mellanox OFED . 34 2.3 Installing Mellanox OFED. 35 2.3.1 Pre-installation Notes . 35 2.3.2 Installation Script . 36 2.3.3 Installation Procedure . 39 2.3.4 Installation Results . 44 2.3.5 Post-installation Notes . 45 2.3.6 Installation Logging . 45 2.3.7 openibd Script . 45 2.4 Updating Firmware After Installation . 46 2.5 Installing MLNX_OFED using YUM . 47 2.5.1 Setting up MLNX_OFED YUM Repository . 47 2.5.2 Installing MLNX_OFED using the YUM Tool . 48 2.5.3 Updating Firmware After Installation. 49 2.6 Uninstalling Mellanox OFED . 49 2.7 Uninstalling Mellanox OFED using the YUM Tool . 49 Mellanox Technologies 3 Rev 2.2-1.0.1 2.8 UEFI Secure Boot . 50 2.8.1 Enrolling Mellanox's x.509 Public Key On your Systems . 50 Chapter 3 Configuration Files . 51 3.1 Persistent Naming for Network Interfaces. 51 Chapter 4 Driver Features . 52 4.1 SCSI RDMA Protocol . 52 4.1.1 Overview . 52 4.1.2 SRP Initiator . 52 4.2 iSCSI Extensions for RDMA (iSER). 61 4.2.1 Overview . 61 4.2.2 iSER Initiator . 61 4.3 Ethernet over IB (EoIB) vNic . 61 4.3.1 Ethernet over IB Topology . 62 4.3.2 EoIB Configuration. 63 4.3.3 Retrieving EoIB Information . 68 4.3.4 Advanced EoIB Settings . 72 4.4 IP over InfiniBand . 76 4.4.1 Introduction . 76 4.4.2 IPoIB Mode Setting. 76 4.4.3 IPoIB Configuration . 77 4.4.4 Subinterfaces . 80 4.4.5 Verifying IPoIB Functionality . 81 4.4.6 Bonding IPoIB . 82 4.5 Quality of Service InfiniBand . 83 4.5.1 Quality of Service Overview. 83 4.5.2 QoS Architecture. 84 4.5.3 Supported Policy . 84 4.5.4 CMA Features . 85 4.5.5 OpenSM Features . 86 4.6 Quality of Service Ethernet . 86 4.6.1 Quality of Service Overview. ..