R Cray Programming Environment User's Guide S–2529–116 © 2004–2014 Cray Inc. All Rights Reserved. This document or parts thereof may not be reproduced in any form unless permitted by contract or by written permission of Cray Inc. The gnulicinfo(7) man page contains the Open Source Software licenses (the "Licenses"). Your use of this software release constitutes your acceptance of the License terms and conditions. U.S. GOVERNMENT RESTRICTED RIGHTS NOTICE The Computer Software is delivered as "Commercial Computer Software" as defined in DFARS 48 CFR 252.227-7014. All Computer Software and Computer Software Documentation acquired by or for the U.S. Government is provided with Restricted Rights. Use, duplication or disclosure by the U.S. Government is subject to the restrictions described in FAR 48 CFR 52.227-14 or DFARS 48 CFR 252.227-7014, as applicable. Technical Data acquired by or for the U.S. Government, if any, is provided with Limited Rights. Use, duplication or disclosure by the U.S. Government is subject to the restrictions described in FAR 48 CFR 52.227-14 or DFARS 48 CFR 252.227-7013, as applicable. The following are trademarks of Cray Inc. and are registered in the United States and other countries: Cray and design, Sonexion, Urika, and YarcData. The following are trademarks of Cray Inc.: ACE, Apprentice2, Chapel, Cluster Connect, CrayDoc, CrayPat, CrayPort, ECOPhlex, LibSci, NodeKARE, Threadstorm. The following system family marks, and associated model number marks, are trademarks of Cray Inc.: CS, CX, XC, XE, XK, XMT, and XT. The registered trademark Linux is used pursuant to a sublicense from LMI, the exclusive licensee of Linus Torvalds, owner of the mark on a worldwide basis. Other trademarks used in this document are the property of their respective owners. AMD, Opteron, and AMD Opteron are trademarks of Advanced Micro Devices, Inc. Apple and OS X are trademarks of Apple Inc. Intel, Gemini, SeaStar, SeaStar2, SeaStar2+ Aries, and Intel Xeon Phi are trademarks of Intel Corporation in the United States and/or other countries. Java is a trademark of Oracle and/or its affiliates. LSF and Platform LSF are trademarks of Platform Computing Corporation. Lustre is a trademark of Xyratex and/or its affiliates. Moab is a trademark of Adaptive Computing Enterprises, Inc. MySQL is a trademark of Oracle and/or its affiliates. NVIDIA, CUDA, Kepler, Tesla, and OpenACC are trademarks of NVIDIA Corporation. OpenMP is a trademark of OpenMP Architecture Review Board. PBS and PBS Professional are trademarks of Altair Engineering, Inc. and are protected under U.S. and international law and treaties. PETSc is a trademark of Copyright (C) 1995-2004 University of Chicago. PGI is a trademark of The Portland Group Compiler Technology, STMicroelectronics, Inc. RSA and SecurID are trademarks of RSA Security Inc. TotalView is a trademark of Rogue Wave Software, Inc. VM is a trademark of International Business Machines Corporation. Windows is a trademark of Microsoft Corporation. UNIX, the “X device,” X Window System, and X/Open are trademarks of The Open Group. RECORD OF REVISION S–2529–116 Published June 2014 Supports Cray XC series systems running Cray Linux Environment (CLE) release 5.1 or later and Cray XE and Cray XK systems running Cray Linux Environment (CLE) release 4.2 or later. Supports Intel® Xeon® Phi™ in autonomous and offload modes. S–2529–114 Published March 2014 Supports Cray XC30 and Cray XC30-AC systems running Cray Linux Environment (CLE) release 5.0 or later and Cray XE and Cray XK systems running Cray Linux Environment (CLE) release 3.1 or later. Supports Intel® Xeon® Phi™ in autonomous mode only. S–2529–111 Published December 2013 Supports Cray XC30 and Cray XC30-AC systems running Cray Linux Environment (CLE) release 5.0 or later and Cray XE and Cray XK systems running Cray Linux Environment (CLE) release 3.1 or later. S–2529–107 Published July 2013 Supports Cray XC30 and Cray XC30-AC systems running Cray Linux Environment (CLE) release 5.0 or later and Cray XE and Cray XK systems running Cray Linux Environment (CLE) release 3.1 or later. S–2529–103 Published March 2013 Supports Cray XC30 and Cray XC30-AC systems running Cray Linux Environment (CLE) release 5.0 or later and Cray XE and Cray XK systems running Cray Linux Environment (CLE) release 3.1 or later. Changes to this Document Cray Programming Environment User's Guide S–2529–116 This guide replaces the Cray Application Developer's Environment User's Guide and supports version 6.32 (and later) of the Cray Application Development Environment and version 1.16 (and later) of the Cray Developer Toolkit. Revised information • The Intel Xeon Phi targeting instructions have been revised and expanded. See Targeting for Intel Xeon Phi on page 31. Contents Page Introduction [1] 13 1.1 What You Must Know About Your System . 13 1.1.1 Processor Type . 13 1.1.2 Compute Units and CPUs . 14 1.1.3 CPU Numbering . 15 1.1.4 Which Network ASIC? . 15 1.1.5 Which GPU or Coprocessor? . 16 1.1.6 Which Operating System? . 16 1.1.7 What Is a Compute Node? . 17 1.1.8 Which File System? . 18 1.1.9 Which Batch System? . 18 1.2 Logging In . 18 1.2.1 UNIX or Linux Users . 18 1.2.2 Windows Users . 19 1.2.3 Apple Users . 22 1.3 Navigating the File Systems . 23 Using Modules [2] 25 2.1 What Is Loaded Now? . 26 2.2 What Is Available? . 26 2.3 Loading and Unloading Modulefiles . 28 2.4 Swapping Compiler Modulefiles . 28 2.5 Swapping Other Programming Environment Components . 29 2.6 Using Targeting Modules . 30 2.6.1 Targeting for a Cray System . 30 2.6.1.1 Compiling Without the Cray Networking Libraries . 31 2.6.2 Targeting for a Standalone Linux workstation, CDL, or Service Node . 31 2.6.3 Targeting for an Accelerator . 31 2.6.4 Targeting for Intel Xeon Phi . 31 2.6.4.1 Offload Mode . 32 S–2529–116 7 Cray Programming Environment User's Guide Page 2.6.4.2 Autonomous Mode . 32 2.6.4.3 Known Limitations . 33 2.7 Module Help . 33 2.8 For More Information . 34 Batch Systems and Program Execution [3] 35 3.1 Interactive Mode . 36 3.1.1 Notes . 37 3.2 Batch Mode . 37 3.3 Using aprun ......................... 39 3.3.1 Special Considerations for Intel Xeon Phi . 41 Using Compilers [4] 43 4.1 About Compiler Drivers . 43 4.1.1 Bypassing the Compiler Drivers . 44 4.2 About C/C++ Data Types . 44 4.3 About the Cray Compiling Environment (CCE) . 45 4.3.1 Known Limitations . 45 4.4 About PGI Compilers . 46 4.4.1 Known Limitations . 46 4.5 About Intel Compilers . 47 4.5.1 Known Limitations . 47 4.6 About GNU Compilers . 47 4.6.1 Known Limitations . 48 4.7 About the Chapel Parallel Programming Language . 48 4.8 About Cross-compilers . 49 Dynamic Linking [5] 51 5.1 Implementation . 51 5.2 Linking Defaults . 51 5.3 Modify Linking Behavior to Use Non-default Libraries . 52 Libraries [6] 55 6.1 Cray Scientific and Math Libraries (CSML) . 55 6.1.1 Basic CSML Components . 55 6.1.2 BLAS and LAPACK . 56 6.1.2.1 Notes . 57 6.1.3 BLACS and ScaLAPACK . 58 6.1.3.1 Notes . 59 8 S–2529–116 Contents Page 6.1.4 Iterative Refinement Toolkit (IRT) . 59 6.1.5 Fourier Transformations . 60 6.1.5.1 FFTW . 60 6.1.5.2 ACML . 61 6.1.6 PETSc . 61 6.1.6.1 Notes . 63 6.1.7 Trilinos . 63 6.1.8 Cray LibSci_ACC . 64 6.2MPT........................... 65 6.2.1 Using MPI and SHMEM Modules . 66 6.2.2 MPI Usage Notes . 67 6.2.3 SHMEM Usage Notes . 67 6.2.4 GPU-to-GPU Communications . 69 6.3 Hugepages . 71 6.3.1 When to Use Hugepages . 71 6.3.2 When to Avoid Using Hugepages . 71 6.3.3 Cray XC30 Usage . 72 6.3.4 Cray XE and Cray XK Usage . 72 6.3.5 Cray XT Usage . 73 6.3.6 Running Independent Software Vendor (ISV) Applications . 73 6.3.7 Known Issues . 74 Debugging Code [7] 75 7.1 Cray Debugger Support Tools . 75 7.1.1 Using CCDB . 76 7.1.2 Using LGDB . 76 7.1.3 Using Abnormal Termination Processing (ATP) . 78 7.1.4 Using STAT . 79 7.2 Using Cray Fast-track Debugging . 80 7.2.1 Supported Compilers and Debuggers . 81 7.3 About Core Files . 81 7.4 Using DDT . 81 7.4.1 Known Limitations . 82 7.5 Using TotalView . 83 7.5.1 Known Limitations . 84 Optimizing Code [8] 85 8.1 Improving I/O . 85 S–2529–116 9 Cray Programming Environment User's Guide Page 8.1.1 Using iobuf ....................... 85 8.1.2 Improving MPI I/O . 87 8.2 Using Compiler Optimizations . 87 8.2.1 Cray Compiling Environment (CCE) . 87 8.3 Using the Cray Performance Measurement and Analysis Tools . 88 8.3.1 About CrayPat-lite . 90 8.3.2 About CrayPat . 90 8.3.2.1 Instrumenting the Program . 91 8.3.2.2 Collecting Data . 91 8.3.2.3 Analyzing Data . 91 8.3.2.4 For More Information . 92 8.3.3 About Cray Apprentice2 . 92 8.3.4 About Reveal . 93 8.3.5 About PAPI . 94 Appendix A glibc Functions 95 Figures Figure 1.