Qlogic HCA and Qlogic OFED Software User Guide Version
Total Page:16
File Type:pdf, Size:1020Kb
QLogic HCA and QLogic OFED Software Users Guide QLogic OFED Version 1.4 Draft IB6054601-00 G.02 Preliminary QLogic HCA and QLogic OFED Software Users Guide QLogic OFED Version 1.4 S Information furnished in this manual is believed to be accurate and reliable. However, QLogic Corporation assumes no responsibility for its use, nor for any infringements of patents or other rights of third parties which may result from its use. QLogic Corporation reserves the right to change product specifications at any time without notice. Applications described in this document for any of these products are for illustrative purposes only. QLogic Corporation makes no representation nor warranty that such applications are suitable for the specified use without further testing or modification. QLogic Corporation assumes no responsibility for any errors that may appear in this document. No part of this document may be copied nor reproduced by any means, nor translated nor transmitted to any magnetic medium without the express written consent of QLogic Corporation. In accordance with the terms of their valid QLogic agreements, customers are permitted to make electronic and paper copies of this document for their own exclusive use. The QHT7040, QHT7140, QLE7140, QLE7240, and QLE7280 QLogic Host Channel Adapters are covered by the following patent: 7308535. Document Revision History Rev. 1.0, 8/20/2005 Rev. 1.1, 11/15/05 Rev. 1.2, 02/15/06 Rev. 1.3 Beta 1, 4/15/06 Rev. 1.3, 6/15/06 Rev. 2.0 Beta, 9/25/06, QLogic Rev IB6054601 A Rev. 2.0 Beta 2, 10/15/06, QLogic Rev IB6054601 B Rev. 2.0, 11/30/06, QLogic Rev IB6054601 C Rev. 2.0, 3/23/07, QLogic Rev IB6054601 D Rev. 2.1, 8/24/07, QLogic Rev IB6054601 E Rev. 2.2, 5/27/08, QLogic Rev IB6054601 F Rev. 2.2, 9/5/08, QLogic Rev IB6054601 G Rev. QLogic OFED 1.4, 4/7/09, QLogic Rev IB6054601 G.02 Changes Sections Affected Product name changed from InfiniPathDraft to QLogic All OFED. Version number is set to 1.4. Instances of InfiniPath changed where appropriate; some file- names and output messages keep old name. Updated Contact Information “Contact Information” on page 1-5 Added more information on VNIC Interface “OpenSM” on page 4-7 Remove sections on configuring ipath_ether. Was “Configuring the ipath_ether Network Inter- ipath_ether now removed. face” on page 3-7. Updated the supported distributions information. “Supported Distributions and Kernels” on page 2-3 Page ii Preliminary IB6054601-00 G.02 QLogic HCA and QLogic OFED Software Users Guide S QLogic OFED Version 1.4 Delete reference to ipath_ether “InfiniPath and OpenFabrics Driver Overview” on page 4-5, “Manual Shutdown or Restart May Hang if NFS in Use” on page D-7, Table F-7 on page F-18, “Glossary” on page Glossary-1, “Managing the InfiniPath Driver” on page 4-18 Delete error entry for ipath_ether in Troubleshoot- Was ifup on ipath_ether on SLES 10 Reports ing "unknown device" on page C-9 Deleted NOTE about MPICH_ROOT replacing Table 5-7 on page 5-18 INFINIPATH_ROOT. No longer needed. Change What’s New in this Release to Features. Appendix 2 Deleted old Features section Updated compiler support information “Compiler Support” on page 2-4 Updated information on software components. “Software Components” on page 2-5 Removed paragraph about this release adding “Features” on page 2-1 support for QLE7240/7280, since this is outdated. Changed InfiniPath to QLogic OFED where appro- priate. Updated Installed Layout section. “Installed Layout” on page 4-2 Updated InfiniPath and OpenFabrics Driver Over- “InfiniPath and OpenFabrics Driver Overview” on view section. page 4-5 Updated Configuring the IPoIB Network Interface “Configuring the IPoIB Network Interface” on section. page 4-6 Updated VNIC interface information “Configuring and Administering the VNIC Inter- face” on page 4-10 Updated SRP section “SRP” on page 4-8 Added new content for configuring Intel MPI. “Setup” on page 6-3 Updated information for starting and stopping the “Managing the InfiniPath Driver” on page 4-18 InfiniPath software. Added NOTE about how to stop driver/modulesDraft manually. Added NOTE about using ibhosts to create a “Console I/O in MPI Programs” on page 5-17, nodefile or hostsfile. “ipath_checkout” on page F-7 Moved Performance and Management Tips sec- “Performance Settings and Management Tips” on tion to the end of Cluster Administration. Reor- page 4-21 dered the subsections in this section. IB6054601-00 G.02 Preliminary Page iii QLogic HCA and QLogic OFED Software Users Guide QLogic OFED Version 1.4 S Removed references to Fedora 6; not supported in “Use Wrapper Scripts for Compiling and Linking” this release. on page 5-6, “SDP Module Not Loading” on page D-8, “Cross-Compilation Issues” on page D-14 Added more information on processor affinity. “Erratic Performance” on page D-11 Reorganized methods 1 and 2. sysctl parameters don’t help TCP/SDP perfor- “Use sysctl to Configure Kernel Parameters” on mance in all systems. Dropped reference to page 5-21 "TCP/SDP" in this section. Replace PathScale compilers with GNU compilers “Getting Started with MPI” on page 5-3, “As noted as default compilers. in “Create the mpihosts File” on page 5-3, an mpi- hosts file (also called a machines file, nodefile, or hostsfile) has been created in your current working directory. This file names the nodes on which the node programs may run.” on page 5-13, “Use Wrapper Scripts for Compiling and Linking” on page 5-6 Change reference to PathScale 3.0 release to 3.1 “Use Wrapper Scripts for Compiling and Linking” on page 5-6 Move “mpirun options summary” to new Appendix A Appendix A. Added/changed these options to mpirun options “mpirun Options Summary” on page A-1 summary. Added -H, -i, -job-info, -no-syslog, -statsmode Updated -np, -rcfile, psc-debug-level, -debug options Added setup checklist. “Step-by-Step Cluster Setup and MPI Usage Checklists” on page 3-1 Renamed Compiling and Linking section. “Use Wrapper Scripts for Compiling and Linking” Drafton page 5-6 Combined sections on configuring ssh. “Configuring ssh and sshd Using shosts.equiv” on page 4-25, “Configuring for ssh Using ssh-agent” on page 4-27 Main section on MPI File I/O removed; use only “Introduction” on page 5-1 subsections Linux File I/O in MPI Programs, and “Linux File I/O in MPI Programs” on page 5-2 MPI-IO with ROMIO to new Introduction in the beginning of the MPI section. “MPI-IO with ROMIO” on page 5-2 Page iv Preliminary IB6054601-00 G.02 QLogic HCA and QLogic OFED Software Users Guide S QLogic OFED Version 1.4 Clarified descriptions of: “mpirun Tunable Options” on page 5-22 -long-len, -L [default: 64000] -long-len-shmem, -s [default: 16000] Split MPI chapter into two parts: “Using QLogic “Using QLogic MPI” on page 5-1, MPI” and “Using Other MPIs” “Using Other MPIs” on page 6-1 More introductory material about MPI. “Using QLogic MPI” on page 5-1, Added introductory remarks about mpirun. “Using mpirun” on page 5-15, New sections for the other MPIs “Open MPI” on page 6-3 “HP-MPI” on page 6-7 “Platform (Scali) MPI” on page 6-8 “Intel MPI” on page 6-10 “Improving Performance of Other MPIs Over IB Verbs” on page 6-12 Updated Other Supported MPI Implementations Table 6-1 on page 6-1 table Deleted NOTE about using mpi-selector; infor- Was below Table 6-1 on page 6-1 mation is now contained in table. Added new issue to Troubleshooting “ibsrpdm Command Hangs When Two HCAs are Installed but Only Unit 1 is Connected to the Switch” on page D-8 Added new issue to Troubleshooting “Number of Processes Exceeds ulimit for Num- ber of Open Files” on page D-20 Added Glossary entry for Verbs. “Verbs” on page Glossary-6 Added Glossary entry for PAT. “PAT” on page Glossary-4 InfiniPath Cluster Administration renamed to Infini- “InfiniPath Cluster Setup and Administration” on Path Cluster Setup and Administration. page 4-1 Location of kernel modules changedDraft to: “Installed Layout” on page 4-2 /lib/modules/$(uname -r)/updates/kernel/drivers/infiniband /hw/ipath Also deleted last paragraph after the pathname above. (Information about replacing kernel mod- ules now obsolete). Also added that documentation can be down- loaded from the web site; it is not in /usr/share/doc/infinipath. IB6054601-00 G.02 Preliminary Page v QLogic HCA and QLogic OFED Software Users Guide QLogic OFED Version 1.4 S Deleted reference to MTRR BIOS setting; obso- “BIOS Settings” on page 4-5 and “Performance lete. Settings and Management Tips” on page 4-21 HP-MPI and Scali now run over Verbs. “Other MPIs” on page 5-2 ssh has limit on number of concurrent connections. “Configuring for ssh” on page 4-25 Added new Troubleshooting issue related to “Outdated ipath_ether Configuration Setup Gener- removed network interface ipath_ether ates Error” on page D-8 New section in Troubleshooting. “General Error Messages” on page D-25 New troubleshooting issue. “Problem with Shell Special Characters and Wrap- per Scripts” on page D-17 Renamed Performance and Management Tips “Performance Settings and Management Tips” on section. page 4-21 Renamed and moved Checking Software Status “Checking Cluster and Software Status” on section. page 4-29 Added default values, if used, to table of environ- Table 5-7 on page 5-18 ment variables. Clarified mpihosts file format “mpihosts File Details” on page 5-13 Update version numbers on other MPIs.