Mellanox OFED Linux User's Manual

Total Page：16

File Type：pdf, Size：1020Kb

Mellanox OFED for Linux User Manual Rev 2.3-1.0.1 Last Updated: November 06, 2014 www.mellanox.com Rev 2.3-1.0.1 2 Mellanox Technologies Document Number: 2877 Rev 2.3-1.0.1 Table of Contents Table of Contents . 3 List of Figures . 6 List of Tables . 7 Chapter 1 Introduction . 13 1.1 Overview . 13 1.2 Stack Architecture . 13 1.2.1 mlx4 VPI Driver . 14 1.2.2 mlx5 Driver . 14 1.2.3 Mid-layer Core . 15 1.2.4 ULPs . 15 1.2.5 MPI . 16 1.2.6 InfiniBand Subnet Manager . 16 1.2.7 Diagnostic Utilities . 16 1.2.8 Mellanox Firmware Tools. 16 1.3 Mellanox OFED Package . 17 1.3.1 ISO Image . 17 1.3.2 Software Components . 17 1.3.3 Firmware . 18 1.3.4 Directory Structure . 18 1.4 Module Parameters . 19 1.4.1 mlx4 Module Parameters . 19 1.4.2 mlx5 Module Parameters . 21 1.5 Device Capabilities . 21 Chapter 2 Installation . 22 2.1 Hardware and Software Requirements . 22 2.2 Downloading Mellanox OFED . 22 2.3 Installing Mellanox OFED. 23 2.3.1 Installation Procedure . 24 2.3.2 Installation Results . 26 2.3.3 Installation Logging . 26 2.3.4 openibd Script . 26 2.3.5 mlnxofedinstall Return Codes. 27 2.4 Uninstalling Mellanox OFED . 27 2.5 Installing MLNX_OFED using YUM . 27 2.5.1 Setting up MLNX_OFED YUM Repository . 27 2.5.2 Installing MLNX_OFED using the YUM Tool . 29 2.5.3 Uninstalling Mellanox OFED using the YUM Tool. 30 2.6 Installing MLNX_OFED using apt-get . 30 2.6.1 Setting up MLNX_OFED apt-get Repository. 30 2.6.2 Installing MLNX_OFED using the apt-get Tool . 30 2.6.3 Uninstalling Mellanox OFED using the apt-get Tool . 31 2.7 Updating Firmware After Installation . 31 Mellanox Technologies 3 Rev 2.3-1.0.1 2.8 UEFI Secure Boot . 32 2.8.1 Enrolling Mellanox's x.509 Public Key On your Systems . 32 2.8.2 Removing Signature from kernel Modules . 32 2.9 Performance Tuning . 33 Chapter 3 Features Overview and Configuration. 34 3.1 Ethernet Network. 34 3.1.1 Interface. 34 3.1.2 Quality of Service (QoS). 39 3.1.3 Quantized Congestion Notification (QCN). 47 3.1.4 Ethtool . 50 3.1.5 Checksum Offload . 52 3.1.6 RDMA over Converged Ethernet (RoCE) . 52 3.1.7 Explicit Congestion Notification (ECN). 60 3.1.8 Power Management. 61 3.1.9 RSS Hash Function . 62 3.1.10 Time-Stamping . 62 3.1.11 Flow Steering . 67 3.2 InfiniBand Network . 72 3.2.1 Interface. 72 3.2.2 OpenSM . 73 3.2.3 Quality of Service (QoS). 120 3.2.4 Secure Host . 124 3.2.5 Upper Layer Protocols . 126 3.2.6 Advanced Transport . 150 3.2.7 Optimized Memory Access. 152 3.2.8 Mellanox PeerDirect™ . 160 3.3 Storage Protocols . 161 3.3.1 SCSI RDMA Protocol (SRP) . 161 3.3.2 iSCSI Extensions for RDMA (iSER) . 170 3.3.3 Lustre. 170 3.4 Virtualization . ..

207

Copy Link

Recommended publications

MLNX OFED Documentation Rev 5.0-2.1.8.0
MLNX_OFED Documentation Rev 5.0-2.1.8.0 Exported on May/21/2020 06:13 AM https://docs.mellanox.com/x/JLV-AQ Notice This document is provided for information purposes only and shall not be regarded as a warranty of a certain functionality, condition, or quality of a product. NVIDIA Corporation (“NVIDIA”) makes no representations or warranties, expressed or implied, as to the accuracy or completeness of the information contained in this document and assumes no responsibility for any errors contained herein. NVIDIA shall have no liability for the consequences or use of such information or for any infringement of patents or other rights of third parties that may result from its use. This document is not a commitment to develop, release, or deliver any Material (defined below), code, or functionality. NVIDIA reserves the right to make corrections, modifications, enhancements, improvements, and any other changes to this document, at any time without notice. Customer should obtain the latest relevant information before placing orders and should verify that such information is current and complete. NVIDIA products are sold subject to the NVIDIA standard terms and conditions of sale supplied at the time of order acknowledgement, unless otherwise agreed in an individual sales agreement signed by authorized representatives of NVIDIA and customer (“Terms of Sale”). NVIDIA hereby expressly objects to applying any customer general terms and conditions with regards to the purchase of the NVIDIA product referenced in this document. No contractual obligations are formed either directly or indirectly by this document. NVIDIA products are not designed, authorized, or warranted to be suitable for use in medical, military, aircraft, space, or life support equipment, nor in applications where failure or malfunction of the NVIDIA product can reasonably be expected to result in personal injury, death, or property or environmental damage.
[Show full text]
ECE 598 – Advanced Operating Systems Lecture 19
ECE 598 { Advanced Operating Systems Lecture 19 Vince Weaver http://web.eece.maine.edu/~vweaver [email protected] 7 April 2016 Announcements • Homework #7 was due • Homework #8 will be posted 1 Why use FAT over ext2? • FAT simpler, easy to code • FAT supported on all major OSes • ext2 faster, more robust filename and permissions 2 btrfs • B-tree fs (similar to a binary tree, but with pages full of leaves) • overwrite filesystem (overwite on modify) vs CoW • Copy on write. When write to a file, old data not overwritten. Since old data not over-written, crash recovery better Eventually old data garbage collected • Data in extents 3 • Copy-on-write • Forest of trees: { sub-volumes { extent-allocation { checksum tree { chunk device { reloc • On-line defragmentation • On-line volume growth 4 • Built-in RAID • Transparent compression • Snapshots • Checksums on data and meta-data • De-duplication • Cloning { can make an exact snapshot of file, copy-on- write different than link, different inodles but same blocks 5 Embedded • Designed to be small, simple, read-only? • romfs { 32 byte header (magic, size, checksum,name) { Repeating files (pointer to next [0 if none]), info, size, checksum, file name, file data • cramfs 6 ZFS Advanced OS from Sun/Oracle. Similar in idea to btrfs indirect still, not extent based? 7 ReFS Resilient FS, Microsoft's answer to brtfs and zfs 8 Networked File Systems • Allow a centralized file server to export a filesystem to multiple clients. • Provide file level access, not just raw blocks (NBD) • Clustered filesystems also exist, where multiple servers work in conjunction.
[Show full text]
Chapter 19 RECOVERING DIGITAL EVIDENCE from LINUX SYSTEMS
Chapter 19 RECOVERING DIGITAL EVIDENCE FROM LINUX SYSTEMS Philip Craiger Abstract As Linux-kernel-based operating systems proliferate there will be an in evitable increase in Linux systems that law enforcement agents must process in criminal investigations. The skills and expertise required to recover evidence from Microsoft-Windows-based systems do not neces sarily translate to Linux systems. This paper discusses digital forensic procedures for recovering evidence from Linux systems. In particular, it presents methods for identifying and recovering deleted files from disk and volatile memory, identifying notable and Trojan files, finding hidden files, and finding files with renamed extensions. All the procedures are accomplished using Linux command line utilities and require no special or commercial tools. Keywords: Digital evidence, Linux system forensics !• Introduction Linux systems will be increasingly encountered at crime scenes as Linux increases in popularity, particularly as the OS of choice for servers. The skills and expertise required to recover evidence from a Microsoft- Windows-based system, however, do not necessarily translate to the same tasks on a Linux system. For instance, the Microsoft NTFS, FAT, and Linux EXT2/3 file systems work differently enough that under standing one tells httle about how the other functions. In this paper we demonstrate digital forensics procedures for Linux systems using Linux command line utilities. The ability to gather evidence from a running system is particularly important as evidence in RAM may be lost if a forensics first responder does not prioritize the collection of live evidence. The forensic procedures discussed include methods for identifying and recovering deleted files from RAM and magnetic media, identifying no- 234 ADVANCES IN DIGITAL FORENSICS tables files and Trojans, and finding hidden files and renamed files (files with renamed extensions.
[Show full text]
MC-1200 Series Linux Software User's Manual
MC-1200 Series Linux Software User’s Manual Version 1.0, November 2020 www.moxa.com/product © 2020 Moxa Inc. All rights reserved. MC-1200 Series Linux Software User’s Manual The software described in this manual is furnished under a license agreement and may be used only in accordance with the terms of that agreement. Copyright Notice © 2020 Moxa Inc. All rights reserved. Trademarks The MOXA logo is a registered trademark of Moxa Inc. All other trademarks or registered marks in this manual belong to their respective manufacturers. Disclaimer Information in this document is subject to change without notice and does not represent a commitment on the part of Moxa. Moxa provides this document as is, without warranty of any kind, either expressed or implied, including, but not limited to, its particular purpose. Moxa reserves the right to make improvements and/or changes to this manual, or to the products and/or the programs described in this manual, at any time. Information provided in this manual is intended to be accurate and reliable. However, Moxa assumes no responsibility for its use, or for any infringements on the rights of third parties that may result from its use. This product might include unintentional technical or typographical errors. Changes are periodically made to the information herein to correct such errors, and these changes are incorporated into new editions of the publication. Technical Support Contact Information www.moxa.com/support Moxa Americas Moxa China (Shanghai office) Toll-free: 1-888-669-2872 Toll-free: 800-820-5036 Tel: +1-714-528-6777 Tel: +86-21-5258-9955 Fax: +1-714-528-6778 Fax: +86-21-5258-5505 Moxa Europe Moxa Asia-Pacific Tel: +49-89-3 70 03 99-0 Tel: +886-2-8919-1230 Fax: +49-89-3 70 03 99-99 Fax: +886-2-8919-1231 Moxa India Tel: +91-80-4172-9088 Fax: +91-80-4132-1045 Table of Contents 1.
[Show full text]
Oracle® Linux 7 Managing File Systems
Oracle® Linux 7 Managing File Systems F32760-07 August 2021 Oracle Legal Notices Copyright © 2020, 2021, Oracle and/or its affiliates. This software and related documentation are provided under a license agreement containing restrictions on use and disclosure and are protected by intellectual property laws. Except as expressly permitted in your license agreement or allowed by law, you may not use, copy, reproduce, translate, broadcast, modify, license, transmit, distribute, exhibit, perform, publish, or display any part, in any form, or by any means. Reverse engineering, disassembly, or decompilation of this software, unless required by law for interoperability, is prohibited. The information contained herein is subject to change without notice and is not warranted to be error-free. If you find any errors, please report them to us in writing. If this is software or related documentation that is delivered to the U.S. Government or anyone licensing it on behalf of the U.S. Government, then the following notice is applicable: U.S. GOVERNMENT END USERS: Oracle programs (including any operating system, integrated software, any programs embedded, installed or activated on delivered hardware, and modifications of such programs) and Oracle computer documentation or other Oracle data delivered to or accessed by U.S. Government end users are "commercial computer software" or "commercial computer software documentation" pursuant to the applicable Federal Acquisition Regulation and agency-specific supplemental regulations. As such, the use, reproduction, duplication, release, display, disclosure, modification, preparation of derivative works, and/or adaptation of i) Oracle programs (including any operating system, integrated software, any programs embedded, installed or activated on delivered hardware, and modifications of such programs), ii) Oracle computer documentation and/or iii) other Oracle data, is subject to the rights and limitations specified in the license contained in the applicable contract.
[Show full text]
Filesystem Considerations for Embedded Devices ELC2015 03/25/15
Filesystem considerations for embedded devices ELC2015 03/25/15 Tristan Lelong Senior embedded software engineer Filesystem considerations ABSTRACT The goal of this presentation is to answer a question asked by several customers: which filesystem should you use within your embedded design’s eMMC/SDCard? These storage devices use a standard block interface, compatible with traditional filesystems, but constraints are not those of desktop PC environments. EXT2/3/4, BTRFS, F2FS are the first of many solutions which come to mind, but how do they all compare? Typical queries include performance, longevity, tools availability, support, and power loss robustness. This presentation will not dive into implementation details but will instead summarize provided answers with the help of various figures and meaningful test results. 2 TABLE OF CONTENTS 1. Introduction 2. Block devices 3. Available filesystems 4. Performances 5. Tools 6. Reliability 7. Conclusion Filesystem considerations ABOUT THE AUTHOR • Tristan Lelong • Embedded software engineer @ Adeneo Embedded • French, living in the Pacific northwest • Embedded software, free software, and Linux kernel enthusiast. 4 Introduction Filesystem considerations Introduction INTRODUCTION More and more embedded designs rely on smart memory chips rather than bare NAND or NOR. This presentation will start by describing: • Some context to help understand the differences between NAND and MMC • Some typical requirements found in embedded devices designs • Potential filesystems to use on MMC devices 6 Filesystem considerations Introduction INTRODUCTION Focus will then move to block filesystems. How they are supported, what feature do they advertise. To help understand how they compare, we will present some benchmarks and comparisons regarding: • Tools • Reliability • Performances 7 Block devices Filesystem considerations Block devices MMC, EMMC, SD CARD Vocabulary: • MMC: MultiMediaCard is a memory card unveiled in 1997 by SanDisk and Siemens based on NAND flash memory.
[Show full text]
The Linux Kernel Past, Present and Future - the Linux Way
The Linux kernel Past, Present and Future - the Linux way Muli Ben-Yehuda [email protected] IBM Haifa Research Labs The Linux Kernel - HRL 2003 – p.1/21 The Linux Kernel linux is a free (speech and beer) UNIX like operating system, developed by thousands of volunteers around the world - including yours truly started as a hobby, grew and matured with the years countless man years went into it, some paid, some for fun one of the leading server operating systems today . and one of the leading embedded operating systems poised to take over the desktop? Hebrew support? The Linux Kernel - HRL 2003 – p.2/21 The beginning From: [email protected] (Linus Benedict Torvalds) Date: 25 Aug 91 20:57:08 GMT Hello everybody out there using minix - I'm doing a (free) operating system (just a hobby, won't be big and professional like gnu) for 386(486) AT clones. This has been brewing since april, and is starting to get ready. I'd like any feedback on things people like/dislike in minix, as my OS resembles it somewhat (same physical layout of the file-system (due to practical reasons) among other things). I've currently ported bash(1.08) and gcc(1.40), and things seem to work.This implies that I'll get something practical within a few months, andI'd like to know what features most people would want. Any suggestions are welcome, but I won't promise I'll implement them :-) Linus ([email protected]) PS. Yes - it's free of any minix code, and it has a multi-threaded fs.
[Show full text]
System Wide Tracing and Profiling in Linux
SYSTEM WIDE TRACING AND PROFILING IN LINUX Nadav Amit Agenda • System counters inspection • Profiling with Linux perf tool • Tracing using ftrace Disclaimer • Introductory level presentation • We are not going to cover many tools • We are not going to get deep into the implementation of the tools • I am not an expert on many of the issues Collect Statistics • First step in analyzing the system behavior • Option 1: Resource statistics tools • iostat, vmstat, netstat, ifstat • dstat Examples: dstat --vm --aio • dstat • dstat --udp --tcp --socket • dstat --vm --aio dstat --udp --tcp --socket Watch system behavior online • Option 2: Sample the counter • top • Use –H switch for thread specific • Use ‘f’ to choose additional fields: page faults, last used processor • Use ‘1’ to turn off cumulative mode • iotop • Remember to run as sudoer top Inspect Raw Counters • Option 3: Go to the raw counters • General • /proc/stat • /proc/meminfo • /proc/interrupts • Process specific • /proc/[pid]/statm – process memory • /proc/[pid]/stat – process execution times • /proc/[pid]/status – human readable • Device specific • /sys/block/[dev]/stat • /proc/dev/net • Hardware • smartctl /proc/interrupts /sys/block/[dev]/stat Name units description ---- ----- ----------- read I/Os requests number of read I/Os processed read merges requests number of read I/Os merged with in-queue I/O read sectors sectors number of sectors read read ticks milliseconds total wait time for read requests write I/Os requests number of write I/Os processed write merges requests number
[Show full text]
Referência Debian I
Referência Debian i Referência Debian Osamu Aoki Referência Debian ii Copyright © 2013-2021 Osamu Aoki Esta Referência Debian (versão 2.85) (2021-09-17 09:11:56 UTC) pretende fornecer uma visão geral do sistema Debian como um guia do utilizador pós-instalação. Cobre muitos aspetos da administração do sistema através de exemplos shell-command para não programadores. Referência Debian iii COLLABORATORS TITLE : Referência Debian ACTION NAME DATE SIGNATURE WRITTEN BY Osamu Aoki 17 de setembro de 2021 REVISION HISTORY NUMBER DATE DESCRIPTION NAME Referência Debian iv Conteúdo 1 Manuais de GNU/Linux 1 1.1 Básico da consola ................................................... 1 1.1.1 A linha de comandos da shell ........................................ 1 1.1.2 The shell prompt under GUI ......................................... 2 1.1.3 A conta root .................................................. 2 1.1.4 A linha de comandos shell do root ...................................... 3 1.1.5 GUI de ferramentas de administração do sistema .............................. 3 1.1.6 Consolas virtuais ............................................... 3 1.1.7 Como abandonar a linha de comandos .................................... 3 1.1.8 Como desligar o sistema ........................................... 4 1.1.9 Recuperar uma consola sã .......................................... 4 1.1.10 Sugestões de pacotes adicionais para o novato ................................ 4 1.1.11 Uma conta de utilizador extra ........................................ 5 1.1.12 Configuração
[Show full text]
Red Hat Enterprise Linux for Real Time 7 Tuning Guide
Red Hat Enterprise Linux for Real Time 7 Tuning Guide Advanced tuning procedures for Red Hat Enterprise Linux for Real Time Radek Bíba David Ryan Cheryn Tan Lana Brindley Alison Young Red Hat Enterprise Linux for Real Time 7 Tuning Guide Advanced tuning procedures for Red Hat Enterprise Linux for Real Time Radek Bíba Red Hat Customer Content Services [email protected] David Ryan Red Hat Customer Content Services [email protected] Cheryn Tan Red Hat Customer Content Services Lana Brindley Red Hat Customer Content Services Alison Young Red Hat Customer Content Services Legal Notice Copyright © 2015 Red Hat, Inc. This document is licensed by Red Hat under the Creative Commons Attribution-ShareAlike 3.0 Unported License. If you distribute this document, or a modified version of it, you must provide attribution to Red Hat, Inc. and provide a link to the original. If the document is modified, all Red Hat trademarks must be removed. Red Hat, as the licensor of this document, waives the right to enforce, and agrees not to assert, Section 4d of CC-BY-SA to the fullest extent permitted by applicable law. Red Hat, Red Hat Enterprise Linux, the Shadowman logo, JBoss, MetaMatrix, Fedora, the Infinity Logo, and RHCE are trademarks of Red Hat, Inc., registered in the United States and other countries. Linux ® is the registered trademark of Linus Torvalds in the United States and other countries. Java ® is a registered trademark of Oracle and/or its affiliates. XFS ® is a trademark of Silicon Graphics International Corp. or its subsidiaries in the United States and/or other countries.
[Show full text]
Ted Ts'o on Linux File Systems
Ted Ts’o on Linux File Systems An Interview RIK FARROW Rik Farrow is the Editor of ;login:. ran into Ted Ts’o during a tutorial luncheon at LISA ’12, and that later [email protected] sparked an email discussion. I started by asking Ted questions that had I puzzled me about the early history of ext2 having to do with the perfor- mance of ext2 compared to the BSD Fast File System (FFS). I had met Rob Kolstad, then president of BSDi, because of my interest in the AT&T lawsuit against the University of California and BSDi. BSDi was being sued for, among other things, Theodore Ts’o is the first having a phone number that could be spelled 800-ITS-UNIX. I thought that it was important North American Linux for the future of open source operating systems that AT&T lose that lawsuit. Kernel Developer, having That said, when I compared the performance of early versions of Linux to the current version started working with Linux of BSDi, I found that they were closely matched, with one glaring exception. Unpacking tar in September 1991. He also archives using Linux (likely .9) was blazingly fast compared to BSDi. I asked Rob, and he served as the tech lead for the MIT Kerberos explained that the issue had to do with synchronous writes, finally clearing up a mystery for me. V5 development team, and was the architect at IBM in charge of bringing real-time Linux Now I had a chance to ask Ted about the story from the Linux side, as well as other questions in support of real-time Java to the US Navy.
[Show full text]
Linux Bootloaders on System Z Current and Future Implementations
Linux Bootloaders on System z Current and Future Implementations Session 16432 Marcus Kraft Product Manager mkraft@suse,com Agenda • Boot Process on System Z ‒ Initial Program Load (IPL) ‒ zIPL and limitations ‒ Grub 2 function ‒ Grub 2 s390 implementation • Btrfs Intergration • Demo 2 IPL Process Initial Program Load (IPL) • Formalized with System 360 ‒ Set the dials, push the button • Initiated from CP console for z/VM or via HMC • 1 PSW and 2 CCW loaded from ‒ That's enough to load a Command Program 4 Linux Initial Program Load (zIPL) • Linux bootloader for System z → zipl • Configured with /etc/zipl.conf • Bootloader is written to disk using /sbin/zipl 5 zIPL Configuration file /etc/zipl.conf [defaultboot] defaultmenu = menu [SLES11_SP3] image = /boot/image-3.0.76-0.11-default target = /boot/zipl ramdisk = /boot/initrd-3.0.76-0.11-default,0x2000000 parameters = "root=/dev/dasda2 TERM=dumb resume=/dev/dasda1" :menu default = 1 prompt = 1 target = /boot/zipl timeout = 10 1 = SLES11_SP3 6 zIPL Operation • Multi-stage operation ‒ Stage 0 – 24 bytes – 2 CCWs to load and TIC to Stage1 ‒ Stage 1 – 104 byte channel program to load Stage 1b ‒ Stage 1b – 1k channel program to load Stage 2 ‒ Stage 2 – 8k maximum menu and kernel loader ‒ Stage 3 – Kernel parameter parser and execution 7 ZIPL Limitations - Devices • Limited device capabilities ‒ Restricted to specific Channel I/O device types ‒ DASD ‒ SCSI ‒ Tape ‒ Boot from the Network? Nope. ‒ Device-mapper tends to be finicky 8 zIPL Limitations – File Systems • Limited file system support ‒
[Show full text]