Mellanox OFED for Freebsd for Connectx-4 and Above Adapter Cards User Manual
Total Page:16
File Type:pdf, Size:1020Kb
Mellanox OFED for FreeBSD for ConnectX-4 and above Adapter Cards User Manual Rev 3.5.2 www.mellanox.com Mellanox Technologies NOTE: THIS HARDWARE, SOFTWARE OR TEST SUITE PRODUCT (“PRODUCT(S)”) AND ITS RELATED DOCUMENTATION ARE PROVIDED BY MELLANOX TECHNOLOGIES “AS-IS” WITH ALL FAULTS OF ANY KIND AND SOLELY FOR THE PURPOSE OF AIDING THE CUSTOMER IN TESTING APPLICATIONS THAT USE THE PRODUCTS IN DESIGNATED SOLUTIONS. THE CUSTOMER'S MANUFACTURING TEST ENVIRONMENT HAS NOT MET THE STANDARDS SET BY MELLANOX TECHNOLOGIES TO FULLY QUALIFY THE PRODUCT(S) AND/OR THE SYSTEM USING IT. THEREFORE, MELLANOX TECHNOLOGIES CANNOT AND DOES NOT GUARANTEE OR WARRANT THAT THE PRODUCTS WILL OPERATE WITH THE HIGHEST QUALITY. ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT ARE DISCLAIMED. IN NO EVENT SHALL MELLANOX BE LIABLE TO CUSTOMER OR ANY THIRD PARTIES FOR ANY DIRECT, INDIRECT, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES OF ANY KIND (INCLUDING, BUT NOT LIMITED TO, PAYMENT FOR PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY FROM THE USE OF THE PRODUCT(S) AND RELATED DOCUMENTATION EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. Mellanox Technologies 350 Oakmead Parkway Suite 100 Sunnyvale, CA 94085 U.S.A. www.mellanox.com Tel: (408) 970-3400 Fax: (408) 970-3403 © Copyright 2019. Mellanox Technologies Ltd. All Rights Reserved. Mellanox®, Mellanox logo, Mellanox Open Ethernet®, LinkX®, Mellanox Spectrum®, Mellanox Virtual Modular Switch®, MetroDX®, MetroX®, MLNX-OS®, ONE SWITCH. A WORLD OF OPTIONS®, Open Ethernet logo, Spectrum logo, Switch-IB®, SwitchX®, UFM®, and Virtual Protocol Interconnect® are registered trademarks of Mellanox Technologies, Ltd. For the complete and most updated list of Mellanox trademarks, visit http://www.mellanox.com/page/trademarks. All other trademarks are property of their respective owners. Document Number: Rev 3.5.2 Mellanox Technologies 2 Table of Contents Table of Contents . 3 List of Tables . 5 Document Revision History . 6 About this Manual . 8 Chapter 1 Overview . 10 1.1 Mellanox OFED for FreeBSD Package Contents . 10 1.1.1 Tarball Package . 10 1.1.2 mlx5 driver. 11 Chapter 2 Installation . 11 2.1 Software Dependencies . 11 2.2 Installing Mellanox Driver for FreeBSD. 12 2.2.1 Installing Mellanox EN Driver for FreeBSD . 12 2.2.2 Installing Mellanox IB/RDMA Driver for FreeBSD. 12 2.3 Firmware Programming . 12 2.3.1 Installing Firmware Tools . 12 2.3.2 Downloading Firmware . 12 2.3.3 Updating Firmware . 13 2.3.4 Setting the Ports to ETH . 14 2.4 Driver Usage and Configuration . 14 Chapter 3 Features Overview and Configuration . 19 3.1 Hardware Large Receive Offload (HW LRO) . 19 3.2 EEPROM Cable Information Reader . 19 3.3 Packet Pacing . 21 3.3.1 Setting Rates for Packet Pacing . 21 3.3.2 Using Packet Pacing Sockets . 23 3.3.3 Feature Characteristics. 24 3.3.4 Limitations . 24 3.3.5 Performance Tuning . 24 3.4 RDMA over Converged Ethernet (RoCE). 25 3.4.1 RoCE Modes. 25 3.4.2 GID Table Population . 27 3.4.3 RoCE Packet Sniffing . 28 3.4.4 Perftest. 28 3.5 Flow Control . 29 3.5.1 Global Pause . 29 Rev 3.5.2 Mellanox Technologies 3 3.5.2 Priority Flow Control (PFC) . 29 3.5.3 PFC Hardware Buffer Configuration . 29 3.5.4 Explicit Congestion Notification (ECN) . 30 3.6 Quality of Service . 31 3.6.1 Priority Code Point (PCP) . 31 3.6.2 Differentiated Service Code Point (DSCP) . 32 3.6.3 Trust State . 32 3.6.4 QoS with RDMA. 32 3.6.5 Mapping User Priority to Traffic Class. 32 3.6.6 Mapping DSCP to Priority Mapping . 33 3.6.7 Maximum Rate Limiting . 33 3.6.8 Enhanced Transmission Selection (ETS) . 34 3.6.9 Priority Flow Control (PFC) Hardware Buffer Configuration . 34 3.7 Rx Hardware Time-Stamping. 35 3.8 Firmware Dump . 35 3.9 Forward Error Correction (FEC) . 36 3.10 IRQ Labeling . 36 3.11 Port Module Events . 37 Chapter 4 Performance Tuning . 39 4.1 Receive Queue Interrupt Moderation . 39 4.2 Tuning for NUMA Architecture . 39 4.2.1 Single NUMA Architecture . 39 4.2.2 Dual NUMA Architecture . 40 4 Mellanox Technologies Rev 3.5.2 List of Tables Table 1: Document Revision History. 6 Table 2: Abbreviations and Acronyms . 8 Table 3: Supported Uplinks to Servers . .10 Table 4: Mellanox OFED for FreeBSD Software Components. .11 Table 5: Buffer Parameters . 34 Table 6: Port Module Events . .37 Table 7: Configuration Options - General Options . .43 Table 8: Configuration Options - Ethernet . .43 Table 9: Configuration Options - InfiniBand/RDMA . .45 Table 10: Statistical Counters - Ethernet . .46 Table 11: Statistical Counters - InfiniBand/RDMA . .54 Rev 3.5.2 Mellanox Technologies 5 Document Revision History Table 1 - Document Revision History Revision Date Description 3.5.2 September 29, 2019 • Updated the following sections: • Section 2.2, “Installing Mellanox Driver for FreeBSD”, on page 12 • Added the following sections: • Section 3.6.9, “Priority Flow Control (PFC) Hardware Buffer Configuration”, on page 34 • Section 3.9, “Forward Error Correction (FEC)”, on page 36 • Section 3.10, “IRQ Labeling”, on page 36 • Section 3.11, “Port Module Events”, on page 37 3.5.1 May 29, 2019 Updated instructions under Section 2.3.3.3, “Updating Firm- ware Using Kernel Module”, on page 13. May 2, 2019 • Updated the following sections: • Section 3.6.5, “Mapping User Priority to Traffic Class”, on page 32 • Section 2.3.3, “Updating Firmware”, on page 13 • Section 4.1, “Receive Queue Interrupt Moderation”, on page 39 • Added the following sections: • Section 2.3.3.2, “Updating Firmware Using mlx5tool”, on page 13 • Section 2.3.3.3, “Updating Firmware Using Kernel Module”, on page 13 • Section A.1.1, “General”, on page 43 3.5.0 January 7, 2019 • Added the following missing Ethernet counters to Table 8, “Configuration Options - Ethernet,” on page 43: • dev.<ifnet_name>.vstats.tx_jumbo_packets • dev.<ifnet_name>.rxstat0.bytes • dev.<ifnet_name>.txstat0tc0.bytes November 29, 2018 • Added Section 3.6.8, “Enhanced Transmission Selection (ETS)”, on page 34. •Updated Section 3.6.7, “Maximum Rate Limiting”, on page 33. 6 Mellanox Technologies Rev 3.5.2 Table 1 - Document Revision History Revision Date Description 3.4.2 July 18, 2018 • Added the following: • Section 3.4.3, “RoCE Packet Sniffing”, on page 28 • Section 3.4.4, “Perftest”, on page 28 • Section 3.5.1, “Global.