Connectx Programmers' Reference Manual
Total Page:16
File Type:pdf, Size:1020Kb
Mellanox FlexBoot User Manual Rev 1.4 www.mellanox.com Rev 1.4 NOTE: THIS HARDWARE, SOFTWARE OR TEST SUITE PRODUCT (“PRODUCT(S)”) AND ITS RELATED DOCUMENTATION ARE PROVIDED BY MELLANOX TECHNOLOGIES “AS-IS” WITH ALL FAULTS OF ANY KIND AND SOLELY FOR THE PURPOSE OF AIDING THE CUSTOMER IN TESTING APPLICATIONS THAT USE THE PRODUCTS IN DESIGNATED SOLUTIONS. THE CUSTOMER'S MANUFACTURING TEST ENVIRONMENT HAS NOT MET THE STANDARDS SET BY MELLANOX TECHNOLOGIES TO FULLY QUALIFY THE PRODUCTO(S) AND/OR THE SYSTEM USING IT. THEREFORE, MELLANOX TECHNOLOGIES CANNOT AND DOES NOT GUARANTEE OR WARRANT THAT THE PRODUCTS WILL OPERATE WITH THE HIGHEST QUALITY. ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT ARE DISCLAIMED. IN NO EVENT SHALL MELLANOX BE LIABLE TO CUSTOMER OR ANY THIRD PARTIES FOR ANY DIRECT, INDIRECT, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES OF ANY KIND (INCLUDING, BUT NOT LIMITED TO, PAYMENT FOR PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY FROM THE USE OF THE PRODUCT(S) AND RELATED DOCUMENTATION EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. Mellanox Technologies Mellanox Technologies, Ltd. 350 Oakmead Parkway Suite 100 Beit Mellanox Sunnyvale, CA 94085 PO Box 586 Yokneam 20692 U.S.A. Israel www.mellanox.com www.mellanox.com Tel: (408) 970-3400 Tel: +972 (0)74 723 7200 Fax: (408) 970-3403 Fax: +972 (0)4 959 3245 © Copyright 2013. Mellanox Technologies. All Rights Reserved. Mellanox®, Mellanox logo, BridgeX®, ConnectX®, CORE-Direct®, InfiniBridge®, InfiniHost®, InfiniScale®, MLNX-OS®, PhyX®, SwitchX®, UFM®, Virtual Protocol Interconnect® and Voltaire® are registered trademarks of Mellanox Technologies, Ltd. Connect-IB™, ExtendX™, FabricIT™, Mellanox Open Ethernet™, Mellanox Virtual Modular Switch™, MetroX™, MetroDX™, ScalableHPC™, Unbreakable-Link™ are trademarks of Mellanox Technologies, Ltd. All other trademarks are property of their respective owners. 2 Mellanox Technologies Document Number: 3218 Rev 1.4 Table of Contents Table of Contents . 1 Document Revision History . 2 Related Documents . .2 Chapter 1 Mellanox FlexBoot . 3 1.1 Overview . 3 1.2 FlexBoot Package . 3 1.3 Reference Documents and Downloads . 3 Chapter 2 Burning the Expansion ROM Image . 4 2.1 Burning the Image on ConnectX®-2/ ConnectX®-3 Devices . 4 2.1.1 Prerequisites . 4 2.1.2 Image Burning Procedure . 4 2.2 Removing the Expansion ROM Image . 4 Chapter 3 Preparing the DHCP Server in Linux . 5 3.1 Case I: InfiniBand Ports. 5 3.1.1 Installing DHCP . 5 3.1.2 Configuring the DHCP Server . 5 3.1.3 Running the DHCP Server . 6 3.1.4 Running the DHCP Client (Optional). 7 3.2 Case II: Ethernet Ports . 7 3.2.1 Extracting the MAC Address – Method I . 7 3.2.2 Extracting the MAC Address – Method II . 8 3.2.3 Placing MAC Addresses in /etc/dhcpd.conf . 8 Chapter 4 Subnet Manager – OpenSM . 9 Chapter 5 BIOS Configuration. 10 Chapter 6 Operation . 11 6.1 Prerequisites. 11 6.2 Starting Boot . 11 Chapter 7 Diskless Machines . 12 7.1 Case I: InfiniBand Ports. 12 7.1.1 Example: Adding an IB Driver to initrd (Linux) . 12 7.2 Case II: Ethernet Ports . 15 7.2.1 Example: Adding an Ethernet Driver to initrd (Linux). 16 Chapter 8 iSCSI Boot . 18 8.1 Configuring an iSCSI Target in Linux Environment . 18 8.1.1 Prerequisites . 18 8.1.2 Configuring the DHCP Server to Boot From an iSCSI Target. 18 Mellanox Technologies 1 Rev 1.4 Document Revision History Table 1 - Document Revision History Revision Date Description Rev 1.4 20-August-2013 • Removed the following sections: • Supported Mellanox Adapter Devices and Firmware • Tested Platforms • Command Line Interface (CLI) • Preparing the DHCP Server in Linux • TFTP Server •WinPE • Updated the following section: • “FlexBoot Package” on page 3 • “Starting Boot” on page 11 Rev 1.3 21-July-2011 • Added ConnectX®-3 • Removed the following sections: • iSCSI Boot Example of SLES 10 SP2 OS • For InfiniHost III Family Devices (PCI Device IDs: 25204, 25218) Rev 1.2.1 16-January-2011 Added section ““Removing the Expansion ROM Image” on page 4 Rev 1.2 15-September-2010 Added module “ipoib_helper.ko” in the section “Case I: InfiniBand Ports” on page 5 Rev 1.1 15-February-2010 Dropped an extra byte of “00” in the dhcp-client-identifier Rev 1.0 21-December-2009 Initial Release Related Documents For further information, please refer to the following documents: • Linux PXE User Guide (www.mellanox.com > Products > Adapter IB/VPI SW > Flex- Boot) • Windows PXE User Guide (www.mellanox.com > Products > Adapter IB/VPI SW > FlexBoot) 2 Mellanox Technologies Rev 1.4 1 Mellanox FlexBoot 1.1 Overview Mellanox FlexBoot is a multiprotocol remote boot technology. FlexBoot supports remote Boot over InfiniBand (BoIB) and over Ethernet. Using Mellanox Virtual Protocol Interconnect (VPI) technologies available in ConnectX® adapt- ers, FlexBoot gives IT Managers’ the choice to boot from a remote storage target (iSCSI target) or a LAN target (Ethernet Remote Boot Server) using a single ROM image on Mellanox Con- nectX products. FlexBoot is based on the open source project iPXE available at http://www.ipxe.org. FlexBoot first initializes the adapter device, senses the port protocol – Ethernet or InfiniBand, and brings up the port. Then it connects to a DHCP server to obtain its assigned IP address and network parameters, and also to obtain the source location of the kernel/OS to boot from. The DHCP server instructs FlexBoot to access the kernel/OS through a TFTP server, an iSCSI target, or some other service. For an InfiniBand port, Mellanox FlexBoot implements a network driver with IP over IB acting as the transport layer. IP over IB is part of the Mellanox OFED for Linux software package (see www.mellanox.com > Products > InfiniBand/VPI SW/Drivers ). The binary code is exported by the device as an expansion ROM image. 1.2 FlexBoot Package The FlexBoot package is provided as a tarball (.tgz extension). Uncompress it using the com- mand “tar zxf <package file name>”. The tarball contains PXE binary files (with the *.mrom extension) for the supported adapter devices. See the release notes file FlexBoot- <flexboot_version>_release_notes.txt for details. The pacakge includes the following files: • dhcpd.conf – sample DHCP configuration file • dhcp.patch – patch file for DHCP v3.1.3 1.3 Reference Documents and Downloads • The Mellanox Firmware Tools (MFT) package and documentation can be downloaded from www.mellanox.com > Products > InfiniBand/VPI Drivers > Firmware Tools. • The MLNX OFED software stack and documentation can be downloaded from www.mellanox.com > Products > InfiniBand Software/Drivers > Linux SW/Drivers. Mellanox Technologies 3 Rev 1.4 Burning the Expansion ROM Image 2 Burning the Expansion ROM Image 2.1 Burning the Image on ConnectX®-2/ ConnectX®-3 Devices This section is valid for ConnectX®-2 devices with firmware versions 2.9.1000 or later and ConnectX®-3 firmware versions 2.30.3000 or later. 2.1.1 Prerequisites 1. Expansion ROM Image The expansion ROM images are provided as part of the Mellanox FlexBoot package and are listed in the release notes file FlexBoot-<flexboot_version>_release_notes.txt. 2. Firmware Burning Tools You need to install the Mellanox Firmware Tools (MFT) package (version 3.0.0 or later) in order to burn the PXE ROM image. To download MFT, see Firmware Tools under www.mellanox.com > Products > InfiniBand/VPI Drivers > Firmware Tools. 2.1.2 Image Burning Procedure To burn the composite image, perform the following steps: Step 1. Obtain the MST device name. Run: # mst start # mst status The device name will be of the form: mt<dev_id>_pci{_cr0|conf0}.1 Step 2. Create and burn the composite image. Run: flint -dev <mst device name> brom <expansion ROM image> Example on Linux: flint -dev /dev/mst/mt26428_pci_cr0 brom ConnectX_26428_ROM-X_X_XXX.mrom Example on Windows: flint -dev mt26428_pci_cr0 brom ConnectX_26428_ROM-X_X_XXX.mrom 2.2 Removing the Expansion ROM Image • Remove the expansion ROM image. Run: flint -dev <mst device name> drom When removing the expansion ROM image, you also remove Flexboot from the boot device list. 1. Depending on the OS, the device name may be superceded with a prefix. 4 Mellanox Technologies Rev 1.4 3 Preparing the DHCP Server in Linux When the boot session starts, the PXE firmware attempts to bring up an adapter network link (port). If it succeeds to bring up a connected link, the PXE firmware communicates with the DHCP server. The DHCP server assigns an IP address to the PXE client and provides it with the location of the boot program. 3.1 Case I: InfiniBand Ports 3.1.1 Installing DHCP FlexBoot requires that the DHCP server runs on a machine which supports IP over IB. 1. Prior to installing DHCP, make sure that Mellanox OFED for Linux is already installed on your DHCP server – see www.mellanox.com. 2. Download and install DHCP v3.1.3 from www.isc.org. A special patch for DHCP is required for supporting IPoIB. The patch file “dhcp.patch” is available under the docs/ directory. 3. To download and install other supported DHCP server versions, please download Mellanox OFED from www.mellanox.com > Products > InfiniBand Software/Drivers > Linux SW/ Drivers and refer to docs/dhcp/README. Standard DHCP fields holding MAC addresses are not large enough to contain an IPoIB hard- ware address. To overcome this problem, DHCP over InfiniBand messages convey a client iden- tifier field used to identify the DHCP session.