User's Guide on Software at Uow HPC Cluster

User’s Guide on Software at UoW HPC Cluster

Information Technology Services University of Wollongong

( Last updated on October 10, 2011)

Contents

1 Requests for Software4

2 Environment Modules4

3 Software List4

4 How to Use7 4.1 Parallel Programming Libraries/Tools...... 7 4.1.1 LAM/MPI...... 7 4.1.2 MPICH...... 7 4.1.3 MPICH2...... 7 4.1.4 OpenMPI...... 7 4.2 Compilers...... 8 4.2.1 GNU Compiler Collection (GCC)...... 8 4.2.2 Intel Fortran Compiler...... 8 4.2.3 Intel C Compiler...... 8 4.2.4 CMake...... 9 4.3 Scripting Languages...... 9 4.3.1 Python...... 9 4.3.2 Perl...... 9 4.3.3 Java...... 9 4.4 Editors...... 9 4.4.1 Emacs...... 9 4.4.2 Nano...... 10 4.5 Math Libraries...... 10 CONTENTS CONTENTS

4.5.1 FFTW...... 10 4.5.2 FFTW3...... 10 4.5.3 Intel Math Kernel Library (IMKL)...... 10 4.5.4 Basic Linear Algebra Subroutines (BLAS)...... 11 4.5.5 Linear Algebra PACKage (LAPACK)...... 11 4.5.6 Basic Linear Algebra Communication Subprograms (BLACS).. 12 4.5.7 Scalable LAPACK (ScaLAPACK)...... 12 4.5.8 Automatically Tuned Linear Algebra Software (ATLAS)..... 13 4.5.9 AMD Core Math Library (ACML)...... 13 4.5.10 GotoBLAS2...... 14 4.6 Debuggers, Proﬁlers and Simulators...... 14 4.6.1 Valgrind...... 14 4.6.2 Intel Debugger...... 14 4.7 Graphics...... 15 4.7.1 GNUPlot...... 15 4.8 Statistics and Mathematics Environments...... 15 4.8.1 R...... 15 4.9 Computational Chemistry...... 15 4.9.1 Quantum ESPRESSO...... 15 4.9.2 GROMACS...... 16 4.9.3 GAUSSIAN...... 17 4.9.4 VMD...... 18 4.9.5 NAMD...... 18 4.9.6 ABINIT...... 19 4.9.7 CPMD...... 20 4.9.8 XCrySDen...... 21 4.9.9 PSI3...... 21 4.9.10 TINKER...... 22 4.9.11 MOPAC 2009...... 23 4.9.12 DOCK...... 23 4.9.13 NWChem...... 24 4.9.14 GAMESS...... 25 4.9.15 OpenBabel...... 26 4.10 Informatics...... 26 4.10.1 GhostScript...... 26 4.10.2 ImageMagick...... 26 4.11 Engineering...... 26 CONTENTS CONTENTS

4.11.1 MATLAB...... 26 4.11.2 ANSYS...... 31 4.11.3 ABAQUS...... 31 4.11.4 Ansoft Maxwell...... 31 4.11.5 LAMMPS...... 31 4.11.6 Materials Studio...... 32 4.12 Biology...... 32 4.12.1 MrBayes...... 32 4.12.2 ATSAS...... 33 3 SOFTWARE LIST

1 Requests for Software

If you require software that is not installed on the UoW HPC cluster please send an email to the ITS admin team at hpc [email protected]. Please note: As there is currently no central budget for software licenses, any costs incurred for the requested software must be covered by user’s research group or the department of the university that the user is employed by. Please see the Section3 (Software List) for the details of the available software.

2 Environment Modules

Package Environment Modules provide a great way to easily customize your shell environment to the requirements of whatever software you wish to use. They are available on the whole system. Always put the following line in your job script to active the environment modules: source /etc/profile.d/00-modules.sh

3 Software List

Some software installed on the UoW HPC cluster is licensed and paid for by various departments and research groups on campus. The * under the ‘Prerequisite’ column indicates software that requires some discussion with the HPC admin to be able to access. Read the ‘How to Use’ section of this guide for more details. Please Note: All software installed on the UoW HPC cluster has passed the standard test suite (if existing). However, it is a user’s responsibility to check the correctness and validity of the software installed on the cluster prior to publishing any result by using them. Table 1: List of software

Name Version Site Prerequisite 1. Parallel Programming Libraries/Tools (Sec. 4.1) LAMMPI 7.1.4 System MPICH 1.2.7 System MPICH2 1.3.1 System OpenMPI 1.3; 1.4.2; 1.4.3 System

2. Compilers (Sec. 4.2) GCC 4.1.2;4.5.2 System Intel Fortran Compiler 10.1.023; 11.1.046 System Intel C/C++ Compiler 10.1.023 System Continued on next page

4 3 SOFTWARE LIST

Table 1 – continued from previous page Name Version Site Prerequisite CMake 2.8.3; 2.8.5 System

3. Scripting Languages (Sec. 4.3) Python 2.4.3 System Perl 5.12.2 System Java 1.6.0 17 Head-node

4. Editors (Sec. 4.4) Emacs 23.2.1 System Nano 2.2.6 System

5. Math Libraries (Sec. 4.5) FFTW 2.1.5 System FFTW3 3.2.2 System Intel MKL 10.2 System BLAS — System LAPACK 3.2.2 System BLACS — System SCALAPACK 1.8.0 System ATLAS 3.9.32 System ACML 4.4.0 System GotoBLAS2 1.13 System

6. Debuggers,Proﬁlers and Simulators (Sec. 4.6) Valgrind 3.2.1 Head-node Intel Debugger 11.1.046 Head-node

7. Graphics (Sec. 4.7) GNUPlot 4.0 Head-node

8. Statistics and Mathematics Environments (Sec. 4.8) R 2.12.0 System

9. Computational Chemistry (Sec. 4.9) Quantum Espresso 4.2.1; 4.3 System GROMACS 4.5.1 System GAUSSIAN g03e01; g09a02 System * VMD 1.8.7; 1.9 System * NAMD 2.7b4; 2.7; 2.8 System * ABINIT 6.4.1 System CPMD 3.13.2; 3.15.1 System * Continued on next page

5 3 SOFTWARE LIST

Table 1 – continued from previous page Name Version Site Prerequisite XCrySDen 1.5.21 System PSI3 3.4 System TINKER 5.0.19 System MOPAC2009 11.053 System DOCK 6.5 System * NWChem 6.0 System GAMESS 1Oct2010 System * OpenBabel 2.3.0 System

10. Informatics (Sec. 4.10) GhostScript 8.15 System ImageMagick 6.6.4-10 System

11. Engineering (Sec. 4.11) MATLAB R2010b; R2011a System ANSYS 12,13 System * ABAQUS 6.91 System * Ansoft Maxwell 14.0 System * LAMMPS 05Nov10 System Materials Studio 5.5 System *

12. Biology (Sec. 4.12) MrBayes 3.1.2 System ATSAS 2.4.1-4 System

6 4 HOW TO USE

4 How to Use

4.1 Parallel Programming Libraries/Tools

4.1.1 LAM/MPI

To use LAM/MPI version 7.1.4 in the current shell, do %module load mpi/lam-7.1.4

NOTE: ‘%’ represents command line prompt all through this paper.

4.1.2 MPICH

To use MPICH version 1.2.7 (gcc build) in the current shell, do %module load mpich/1.2.7-gcc

4.1.3 MPICH2

To use MPICH2 version 1.3.1 (gcc build) in the current shell, do %module load mpi/mpich2-1.3.1-gcc

To use Intel Compiler build of MPICH2 %module load mpich2/1.3.1-itl

4.1.4 OpenMPI

To use OpenMPI version 1.3 in the current shell, do %module load mpi/openmpi-1.3

To use OpenMPI version 1.4.2 in the current shell, do %module load mpi/openmpi-1.4.2

To use OpenMPI version 1.4.3 in the current shell, do %module load mpi/openmpi-1.4.3

OpenMPI 1.4.3 is RECOMMENDED as your default MPI environment. NOTE: MPI environment can be set for future shells by using environment switcher package. Please refer to ‘UOW HPC Cluster User Guide’ for further detail.

7 4.2 Compilers 4 HOW TO USE

4.2 Compilers

4.2.1 GNU Compiler Collection (GCC)

GCC 4.1.2 is installed in system path which can be used directly. To use GCC version 4.5.2, do %module load gcc/4.5.2 in the command line or in your job script.

4.2.2 Intel Fortran Compiler

The Intel Fortran Compiler is called ifort. To use Intel Fortran Compiler version 11.1.046, do %module load intel-fc/11.1.046 in the command line or add it in your job script. NOTE: This version is incompatible with intel-cc/10.1.023. To use Intel Fortran Compiler version 10.1.023, do %module load intel-fc/10.1.023

This version is compatible with intel-cc/10.2.023. To see all other versions of the compiler available do %module avail intel-fc

Note that the debugger idb is now the graphical debugger. To use the non-graphical interface use idbc command.

4.2.3 Intel C Compiler

The Intel C/C++ Compiler is called as icc and the C++ compiler is icpc. To use Intel C/C++ Compiler version 10.1.023, do %module load intel-cc/10.1.023

NOTE: This version is incompatible with intel-fc/11.1.046. To see all installed versions do %module avail intel-cc

8 4.3 Scripting Languages 4 HOW TO USE

4.2.4 CMake

The cross-platform, open-source build system CMake is a family of tools designed to build, test and package software. CMake is used to control the software compilation process using simple platform and compiler independent configuration files. CMake generates native makefiles and workspaces that can be used in the compiler environment of your choice. To use CMake, type module load cmake/2.8.3 or module load cmake/2.8.5 in the command line.

4.3 Scripting Languages

4.3.1 Python

Python is installed system wide (which means user can submit python based applications to the queue system). To use python, do %python

4.3.2 Perl

To use Perl, do module load perl/5.12.2 in your job script.

4.3.3 Java

Java is installed at the head node only.

4.4 Editors

4.4.1 Emacs

Emacs is mainly used as the text editor for interactive PBS jobs. To use emacs in your interactive PBS job shell, do %module load emacs/23.2.1 in the shell command line.

9 4.5 Math Libraries 4 HOW TO USE

4.4.2 Nano

Nano is mainly used as the text editor for interactive PBS jobs. To use nano in your interactive PBS job shell, do %module load nano/2.2.6 in the shell command line.

4.5 Math Libraries

4.5.1 FFTW

FFTW is a C subroutine library for computing the discrete Fourier transform (DFT) in one or more dimensions, of arbitrary input size, and of both real and complex data (as well as of even/odd data, i.e. the discrete cosine/sine transforms or DCT/DST). To use FFTW, do %module load fftw/2.1.5

This will add the FFTW library path into environment variables LD LIBRARY PATH, LD RUN PATH and LIBRARY PATH and set the FFT include path to CPATH and FPATH.

4.5.2 FFTW3

To use FFTW3 do %module load fftw/3.2.2

This will add the FFTW3 library path into environment variables LD LIBRARY PATH, LD RUN PATH and LIBRARY PATH and set the FFT include path to CPATH and FPATH.

4.5.3 Intel Math Kernel Library (IMKL)

IMKL is Intel’s math library optimised for the Intel compiler. It includes the BLAS and LAPACK routines. The IMKL library also includes FFT routines, some sparse solvers and a vector statistical library which provides random number generators. To use IMKL version 10.2, do %module load intel-mkl/10.2

The path of IMKL are preset as MKL_ROOT, MKL_BASE, INTEL_MKL_ROOT or INTEL_MKL_BASE

Set the correct variables in your Makeﬁle to reﬂect these. To learn more on how to use IMKL, please visit IMKL site.

10 4.5 Math Libraries 4 HOW TO USE

4.5.4 Basic Linear Algebra Subroutines (BLAS)

The Basic Linear Algebra Subroutines (BLAS) are fundamental to many linear algebra software packages. The 3 levels of BLAS corresponding to the following operations respectively: Level 1 BLAS: vector*vector Level 2 BLAS: vector*matrix Level 3 BLAS: matrix*matrix The BLAS routines can be found in several packages such as Intel MKL, SCSL, Atlas and Libgoto. This is a GCC version. To use BLAS static library, include /hpc/software/package/BLAS/blas_LINUX.a as your compiling ﬂag. To use BLAS shared library, include -L/hpc/software/packages/LAPACK/3.2.2_shared/lib -lblas as your compiling ﬂag.

4.5.5 Linear Algebra PACKage (LAPACK)

Linear Algebra PACKage (LAPACK) is a collection of routines for solving systems of simultaneous linear equations, least-squares solutions of linear systems of equations, eigenvalue problems, and singular value problems. The associated matrix factorizations (LU, Cholesky, QR, SVD, Schur, generalized Schur) are also provided, as are related computations such as reordering of the Schur factorizations and estimating condition numbers. Dense and banded matrices are handled, but not general sparse matrices. In all areas, similar functionality is provided for real and complex matrices, in both single and double precision. This is a GCC version. To use LAPACK static library, include both /hpc/software/packages/LAPACK/3.2.2/lapack_LINUX.a and /hpc/software/package/BLAS/blas_LINUX.a as your compiling ﬂags. To use BLAS shared library, include -L/hpc/software/packages/LAPACK/3.2.2_shared/lib -llapack -lblas as your compiling ﬂag.

11 4.5 Math Libraries 4 HOW TO USE

4.5.6 Basic Linear Algebra Communication Subprograms (BLACS)

The BLACS project is an ongoing investigation whose purpose is to create a linear algebra oriented message passing interface that may be implemented efﬁciently and uniformly across a large range of distributed memory platforms. This is an OpenMPI version. To use the MPI BLACS static library, include /hpc/software/packages/BLACS/LIB/blacsF77init_MPI-LINUX-0.a /hpc/software/packages/BLACS/LIB/blacs_MPI-LINUX-0.a /hpc/software/packages/BLACS/LIB/blacsCinit_MPI-LINUX-0.a as your compiling ﬂag within the OpenMPI environment loaded by module load mpi/openmpi-1.4.3

4.5.7 Scalable LAPACK (ScaLAPACK)

TheScaLAPACK library includes a subset of LAPACK routines redesigned for distributed memory MIMD parallel computers. It is currently written in a SPMD (Single Program Multiple Data) style using explicit message passing for interprocessor communication. It assumes matrices are laid out in a two-dimensional block cyclic decomposition. It is designed for heterogeneous computing and is portable on any computer that supports MPI (or PVM). Like LAPACK, the ScaLAPACK routines are based on block-partitioned algorithms in order to minimize the frequency of data movement between different levels of the memory hierarchy. (For such machines, the memory hierarchy includes the off-processor memory of other processors, in addition to the hierarchy of registers, cache, and local memory on each processor.) The fundamental building blocks of the ScaLAPACK library are distributed memory versions (PBLAS) of the Level 1, 2 and 3 BLAS, and a set of Basic Linear Algebra Communication Subprograms (BLACS) for communication tasks that arise frequently in parallel linear algebra computations. In the ScaLAPACK routines, all interprocessor communication occurs within the PBLAS and the BLACS. One of the design goals of ScaLAPACK was to have the ScaLAPACK routines resemble their LAPACK equivalents as much as possible. This is a OpenMPI version. To use this SCALAPACK static library, include /hpc/software/packages/SCALAPACK/1.8.0/libscalapack.a /hpc/software/packages/BLACS/LIB/blacsF77init_MPI-LINUX-0.a /hpc/software/packages/BLACS/LIB/blacs_MPI-LINUX-0.a /hpc/software/packages/BLACS/LIB/blacsCinit_MPI-LINUX-0.a as your compiling ﬂag within the OpenMPI environment loaded by module load mpi/openmpi-1.4.3

12 4.5 Math Libraries 4 HOW TO USE

4.5.8 Automatically Tuned Linear Algebra Software (ATLAS)

The ATLAS project is an ongoing research effort focusing on applying empirical techniques in order to provide portable performance. At present, it provides C and Fortran77 interfaces to a portably efﬁcient BLAS implementation, as well as a few routines from LAPACK. The ATLAS installed at gur provides a full LAPACK package. ATLAS support gfortran compiler. To use ATLAS static library, include /hpc/software/packages/ATLAS/3.9.32/lib/libatlas.a /hpc/software/packages/ATLAS/3.9.32/lib/libcblas.a /hpc/software/packages/ATLAS/3.9.32/lib/libf77blas.a /hpc/software/packages/ATLAS/3.9.32/lib/liblapack.a /hpc/software/packages/ATLAS/3.9.32/lib/libptcblas.a /hpc/software/packages/ATLAS/3.9.32/lib/libptf77blas.a as your compiling ﬂags.

4.5.9 AMD Core Math Library (ACML)

ACML provides a free set of thoroughly optimized and threaded math routines for HPC, scientiﬁc, engineering and related compute-intensive applications. ACML consists of the following main components:

• A full implementation of Level 1, 2 and 3 BLAS and LAPACK, with key routines optimized for high performance on AMD Opteron processors.

• A comprehensive suite of FFTs in single-, double-, single and double-complex data types.

• Fast scalar, vector, and array math transcendental library routines optimized for high performance on AMD Opteron processors.

• Random Number Generators in both single- and double-precision.

ACML support both gfortran and Intel compiler. For gfortran version, use /hpc/software/packages/ACML/4.4.0/gfortran64/lib /hpc/software/packages/ACML/4.4.0/gfortran64_mp/lib

For Intel Compiler version, use /hpc/software/packages/ACML/4.4.0/ifort64/lib /hpc/software/packages/ACML/4.4.0/ifort64_mp/lib

13 4.6 Debuggers, Proﬁlers and Simulators 4 HOW TO USE

4.5.10 GotoBLAS2

GotoBLAS2 is released by the Texas Advanced Computing Center (TACC). It uses new algorithms and memory techniques for optimal performance of the BLAS routines and includes the following features:

• Conﬁgurations for a variety of hardware platform. • Incorporation of features of many ISAs (Instruction Set Architecture) • Implementation of NUMA controls to assure best process afﬁnity and memory policy. • Dynamic detection of multiple architecture components, which can be included in a single binary (for binary distributions)

GotoBLBS2 can be used with gfortran. To use GotoBLAS2, include /hpc/software/packages/GotoBLAS2/libgoto2.a /hpc/software/packages/GotoBLAS2/libgoto2_barcelonap-r1.13.a as the static library, or, link following shared libraries /hpc/software/packages/GotoBLAS2/libgoto2.so /hpc/software/packages/GotoBLAS2/libgoto2_barcelonap-r1.13.so

4.6 Debuggers, Proﬁlers and Simulators

4.6.1 Valgrind

Valgrind is an instrumentation framework for building dynamic analysis tools. There are Valgrind tools that can automatically detect many memory management and threading bugs, and proﬁle your programs in detail. You can also use Valgrind to build new tools. To use valgrind on the head node, do %valgrind

4.6.2 Intel Debugger

To use Intel Debugger, in the command line of the head node, ﬁrstly do %module load intel-fc/11.1.046 And then, do %idb for GUI environment or %idbc as the command line interface.

14 4.7 Graphics 4 HOW TO USE

4.7 Graphics

4.7.1 GNUPlot

GNUPlot is a portable command-line driven graphing utility for linux, OS/2, Windows, OSX, VMS, and many other platforms. It was originally created to allow scientists and students to visualize mathematical functions and data interactively, but has grown to support many non-interactive uses such as web scripting. Gnuplot supports many types of plots in either 2D and 3D. It can draw using lines, points, boxes, contours, vector fields, surfaces, and various associated text. It also supports various specialized plot types. Gnuplot supports many different types of output: interactive screen terminals (with mouse and hotkey input), direct output to pen plotters or modern printers, and output to many file formats (eps, fig, jpeg, LaTeX, metafont, pbm, pdf, png, postscript, svg, ...). Gnuplot is easily extensible to include new output modes. Recent additions include an interactive terminal based on wxWidgets and the creation of mousable graphs for web display using the HTML5 canvas element. GNUPlot is installed on the head node only. To use GNUPlot, do %gnuplot

4.8 Statistics and Mathematics Environments

4.8.1 R

R is language and environment for statistical computing and graphics. It provides a wide variety of statistical and graphical techniques (linear and nonlinear modeling, statistical tests, time series analysis, classification, clustering, ...). R is designed as a true computer language with control-flow constructions for iteration and alternation, and it allows users to add additional functionality by defining new functions. For computationally-intensive tasks, C, C++, and Fortran code can be linked and called at run time. Advanced users can write C or Java code to manipulate R objects directly. To use R, add module load R/2.12.0 in your job script.

4.9 Computational Chemistry

4.9.1 Quantum ESPRESSO

Quantum ESPRESSO is an integrated suite of computer codes for electronic-structure calculations and materials modeling at the nanoscale. It is based on density-functional theory, plane waves, and pseudopotentials (both norm-conserving and ultrasoft). To use Quantum ESPRESSO, add

15 4.9 Computational Chemistry 4 HOW TO USE module load quantum_espresso/4.2.1 or module load quantum_espresso/4.3 in your job script. An example job script is similar to the following ﬁle jobscript: #!/bin/sh #PBS -q workq #PBS -l cput=30:00:00 #PBS -l pvmem=1GB #PBS -l nodes=4 #PBS -j eo #PBS -e test.err #PBS -m abe

NP=‘wc -l < $PBS_NODEFILE‘ #ALWAYS KEEP THESE LINES IN echo "Adding the modules environments for torque..." source /etc/profile.d/00-modules.sh #ADDING MODULES HERE module load quantum_espresso/4.3 #END OF ALL MODULES cd $PBS_O_WORKDIR mpirun -machinefile $PBS_NODEFILE -np $NP pw.x < electric0.in >& electric0.out

Submit the job to PBS by using %qsub jobscript

4.9.2 GROMACS

GROMACS is a versatile package to perform molecular dynamics, i.e. simulate the Newtonian equations of motion for systems with hundreds to millions of particles. It is primarily designed for biochemical molecules like proteins, lipids and nucleic acids that have a lot of complicated bonded interactions, but since GROMACS is extremely fast at calculating the nonbonded interactions (that usually dominate simulations) many groups are also using it for research on non-biological systems, e.g. polymers. To use GROMACS in the double precision add module load gromacs/4.5.1 in your job script. An example job scrpit on submitting parallel GROMACS jobs to the PBS is given below: #!/bin/bash #PBS -l walltime=00:10:00,pvmem=800MB

16 4.9 Computational Chemistry 4 HOW TO USE

#PBS -l nodes=4 #PBS -j eo #PBS -e test.err #PBS -m abe #PBS -N gromacs test

NP=‘wc -l < $PBS_NODEFILE‘ source /etc/profile.d/00-modules.sh module load gromacs/4.5.1 cd $PBS_O_WORKDIR grompp_d -f min_stpst.mdp -c methane.gro -p methane.top -o min_stpst.tpr mpirun -machinefile $PBS_NODEFILE -np $NP mdrun_d_mpi -s min_stpst.tpr -c methane.gro -g min_stpst.log -e min_stpst.ene The above example job script requests 4 cpus, and is submitted to run within 10 minutes of wall clock time, and 800MB of virtual memory. Submit the job to PBS by using %qsub jobscript The above example test can be found at /hpc/tmp/examples/gromacs-4.5.1 The GROMACS executables are available with the sufﬁx ‘ d’ as it is a double precision version. Please note the extra sufﬁx ‘ mpi’ for the MPI-aware utility mdrun d mpi.

4.9.3 GAUSSIAN

GAUSSIAN contains series of electronic structure programs, used by chemists, chemical engineers, biochemists, physicists and other scientists worldwide. Starting from the fundamental laws of quantum mechanics, GAUSSIAN 09 predicts the energies, molecular structures, vibrational frequencies and molecular properties of molecules and reactions in a wide variety of chemical environments. GAUSSIAN is commercial licensed software. Please contact ITS for further information. An example job script of Gaussian is shown as below: #!/bin/sh #PBS -q workq #PBS -N test #PBS -l cput=100:00:00 #PBS -l pvmem=1GB #PBS -j eo #PBS -e test.err

17 4.9 Computational Chemistry 4 HOW TO USE

#PBS -m ae #PBS -l nodes=1:ppn=4

#ALWAYS KEEP THESE LINES IN echo "Adding the modules environments for torque..." source /etc/profile.d/00-modules.sh // this is necessary #ADDING MODULES HERE module load gaussian/g09a02 #END OF ALL MODULES # You can check which CPUs are allocated in test.err cat $PBS_NODEFILE cd $PBS_O_WORKDIR g09 < test001.com >& test001.log

The above example job can be found at /hpc/tmp/examples/gaussian/g09a02: NOTE: Gaussian is parallelized using OpenMP which can only utilize CPUs within a single node. Always request CPUs within one node by using #PBS -l nodes=1:ppn=N with 1≤N≤8.

4.9.4 VMD

VMD is a molecular visualization program for displaying, animating, and analyzing large biomolecular systems using 3-D graphics and built-in scripting. VMD is licensed software. Please make registration prior to requesting access to it. To run VMD in the text mode, add module load vmd/1.8.7 or module load vmd/1.9 in your job script. To run VMD in the Graphic mode, ﬁrst submit an interactive job to the queue system by %qsub -I -X -q short

Then load the ‘VMD’ module and run ‘vmd’ command to start. %module load vmd/1.9 %vmd

Remember to type ‘exit’ after working to quit the interactive job.

4.9.5 NAMD

NAMD is a parallel, object-oriented molecular dynamics code designed for high-performance simulation of large biomolecular systems.

18 4.9 Computational Chemistry 4 HOW TO USE

User needs to make registration prior to accessing it. Please contact ITS for further information. NAMD example job script: #!/bin/bash #PBS -l nodes=8 #PDB -q workq #PBS -l cput=30:00:00 #PBS -j eo #PBS -e test.err #PBS -N NAMDrun source /etc/profile.d/00-modules.sh module load namd/2.7 cd $PBS_O_WORKDIR namdrun alanin.namd > test.log

An example test can be found at /hpc/tmp/examples/namd-2.7 To use NAMD 2.8, put module load namd/2.8 in the job script instead.

4.9.6 ABINIT

ABINIT is a package whose main program allows one to ﬁnd the total energy, charge density and electronic structure of systems made of electrons and nuclei (molecules and periodic solids) within Density Functional Theory (DFT), using pseudopotentials and a planewave basis. ABINIT also includes options to optimize the geometry according to the DFT forces and stresses, or to perform molecular dynamics simulations using these forces, or to generate dynamical matrices, Born effective charges, and dielectric tensors. Excited states can be computed within the Time-Dependent Density Functional Theory (for molecules), or within Many-Body Perturbation Theory (the GW approximation). In addition to the main ABINIT code, different utility programs are provided. To use ABINIT, add module load abinit/6.4.1 in your job script. An example job script is similar to the following jobscript ﬁle: #!/bin/sh #PBS -q workq #PBS -l cput=30:00:00 #PBS -l pvmem=1GB #PBS -j eo #PBS -e test.err #PBS -l nodes=2:ppn=2

19 4.9 Computational Chemistry 4 HOW TO USE

NP=‘wc -l < $PBS_NODEFILE‘ source /etc/profile.d/00-modules.sh module load abinit/6.4.1 cd $PBS_O_WORKDIR mpirun -machinefile $PBS_NODEFILE -np $NP abinit < tH1.file > log

The example ﬁle can be found at /hpc/tmp/examples/abinit-6.4.1

Submit the job to PBS by using %qsub jobscript

4.9.7 CPMD

The CPMD code is a plane wave/pseudopotential implementation of Density Functional Theory, particularly designed for ab-initio molecular dynamics. Its ﬁrst version was developed by Jurg Hutter at IBM Zurich Research Laboratory starting from the original Car-Parrinello codes. During the years many people from diverse organizations contributed to the development of the code and of its pseudopotential library:contains series of electronic structure programs, used by chemists, chemical engineers, biochemists, physicists and other scientists worldwide. CPMD is licensed software. Please make the registration before requesting access to it. Adding module load cpmd/3.13.2 in your job script to use 3.13.2 or adding module load cpmd/3.15.1 in your job script to use CPMD 3.15.1. An example job script on using cpmd-3.15.1 is given as below: #!/bin/sh #PBS -q workq #PBS -l cput=30:00:00 #PBS -l pvmem=1GB #PBS -j eo #PBS -e test.err #PBS -m n #PBS -l nodes=2:ppn=2

NP=‘wc -l <$PBS_NODEFILE‘ source /etc/profile.d/00-modules.sh module load cpmd/3.15.1 cd $PBS_O_WORKDIR

20 4.9 Computational Chemistry 4 HOW TO USE mpirun -machinefile $PBS_NODEFILE -np $NP cpmd.x inpit >& output

The example ﬁle can be found at /hpc/tmp/examples/cpmd-3.15.1

Submit the job to PBS by using %qsub jobscript

4.9.8 XCrySDen

XCrySDen is a crystalline and molecular structure visualisation program, which aims at display of isosurfaces and contours, which can be superimposed on crystalline structures and interactively rotated and manipulated. It can run on most UNIX platforms, without any special hardware requirements. To use XCrySDen, please submit an interactive job to ’short’ queue and then load the module ’xcrysden’, i.e. %qsub -I -X -q short % module load xcrysden/1.5.21 % xcrysden

Remember typing ’exit’ after ﬁninshing all work to quit the interactive job.

4.9.9 PSI3

The PSI3 suite of quantum chemical programs is designed for efficient, high-accuracy calculations of properties of small to medium-sized molecules. The package’s current capabilities include a variety of Hartree-Fock, coupled cluster, complete-active-space self-consistent-field, and multi-reference configuration interaction models. Molecular point-group symmetry is utilized throughout to maximize efficiency. To use PSI3, add module load psi3/3.4 in your job script. An example job script is similar to the following jobscript file: #!/bin/sh #PBS -q workq #PBS -l cput=30:00:00 #PBS -l pvmem=1GB #PBS -j eo #PBS -e test.err #PBS -m n #PBS -l nodes=1

NP=‘wc -l <$PBS_NODEFILE‘ source /etc/profile.d/00-modules.sh

21 4.9 Computational Chemistry 4 HOW TO USE module load psi3/3.4 cd $PBS_O_WORKDIR psi3 < inpput

The example ﬁle can be found at /hpc/tmp/examples/psi3

Submit the job to PBS by using %qsub jobscript

4.9.10 TINKER

The TINKER molecular modeling software is a complete and general package for molecular mechanics and dynamics, with some special features for biopolymers. TINKER has the ability to use any of several common parameter sets, such as Amber (ff94, ff96, ff98, ff99, ff99SB), CHARMM (19, 22, 22CMAP), Allinger MM (MM2-1991 and MM3-2000), OPLS (OPLS-UA, OPLS-AA), Merck Molecular Force Field (MMFF), Liam Dang’s polarizable potentials, and their own AMOEBA polarizable atomic multipole force field. Parameter sets for other standard force fields such as GROMOS, UFF, ENCAD and MM4 are under consideration for future releases. To use TINKER, add module load tinker/5.0.19 in your job script. An example job script is similar to the following jobscript file: #!/bin/sh #PBS -q workq #PBS -l cput=30:00:00 #PBS -l pvmem=1GB #PBS -j eo #PBS -e test.err #PBS -m n #PBS -l nodes=1

NP=‘wc -l <$PBS_NODEFILE‘ source /etc/profile.d/00-modules.sh module load tinker/5.0.19 cd $PBS_O_WORKDIR ./anion.run >& output

The example ﬁle can be found at /hpc/tmp/examples/tinker

Submit the job to PBS by using

22 4.9 Computational Chemistry 4 HOW TO USE

%qsub jobscript

NOTE: The potential ﬁle path is set to ‘/hpc/software/packages/tinker/5.0.19/params/’

4.9.11 MOPAC 2009

MOPAC 2009 is a semiempirical quantum chemistry software package for the prediction of chemical properties and modeling of chemical reactions. It is used by chemists and biochemists for both research and teaching. To use MOPAC 2009, add module load mopac2009/11.053 in your job script. An example job script is similar to the following jobscript ﬁle: #!/bin/sh #PBS -q workq #PBS -N mopacjob #PBS -l cput=01:00:00 #PBS -l pvmem=100MB #PBS -j eo #PBS -e test.err #PBS -m abe #PBS -l nodes=1

NP=‘wc -l < $PBS_NODEFILE‘ source /etc/profile.d/00-modules.sh module load mopac2009/11.053 cd $PBS_O_WORKDIR MOPAC2009.exe test.mop

The example ﬁle can be found at /hpc/tmp/examples/mopac2009

Submit the job to PBS by using %qsub jobscript

4.9.12 DOCK

DOCK addresses the problem of ”docking” molecules to each other. In general, ”docking” is the identiﬁcation of the low-energy binding modes of a small molecule, or ligand, within the active site of a macromolecule, or receptor, whose structure is known. A compound that interacts strongly with, or binds, a receptor associated with a disease may inhibit its function and thus act as a drug. Solving the docking problem computationally

23 4.9 Computational Chemistry 4 HOW TO USE requires an accurate representation of the molecular energetics as well as an efﬁcient algorithm to search the potential binding modes. An example job script of DOCK is similar to the following jobscript ﬁle: #PBS -l cput=0:30:00 #PBS -l pvmem=400MB #PBS -N dock job #PBS -l nodes=4 #PBS -q workq #PBS -m abe source /etc/profile.d/00-modules.sh module load dock/6.5 cat $PBS_NODEFILE cd $PBS_O_WORKDIR mpiexec dock6.mpi -i mpi.dockin -o mpi.dockout

An example test job can be found at /hpc/tmp/examples/dock6

DOCK is licensed software. Please make registration here ﬁrstly before requesting access to DOCK deployed at the cluster.

4.9.13 NWChem

NWChem aims to provide its users with computational chemistry tools that are scalable both in their ability to treat large scientiﬁc computational chemistry problems efﬁciently, and in their use of available parallel computing resources from high-performance parallel supercomputers to conventional workstation clusters. NWChem software can handle

• Biomolecules, nanostructures, and solid-state

• From quantum to classical, and all combinations

• Gaussian basis functions or plane-waves

• Scaling from one to thousands of processors

• Properties and relativity

To use NWChem, just adding module load nwchem/6.0 in your job script. An example job script on running NWChem calculations is shown as below:

24 4.9 Computational Chemistry 4 HOW TO USE

#!/bin/sh #PBS -q workq #PBS -N nwchemjob #PBS -l cput=100:00:00 #PBS -l pvmem=600MB #PBS -m abe #PBS -l nodes=8

NP=‘wc -l < $PBS_NODEFILE‘ source /etc/profile.d/00-modules.sh module load nwchem/6.0 cd $PBS_O_WORKDIR mpirun -machinefile $PBS_NODEFILE -np $NP nwchem 3carbo_dft.nw >& 3carbo_dft.8.out

The example test can be found in /hpc/tmp/examples/nwchem-6.0/qmd.

4.9.14 GAMESS

GAMESS is a program for ab initio molecular quantum chemistry. Brieﬂy, GAMESS can compute SCF wavefunctions ranging from RHF, ROHF, UHF, GVB, and MCSCF. Correlation corrections to these SCF wavefunctions include Conﬁguration Interaction, second order perturbation Theory, and Coupled-Cluster approaches, as well as the Density Functional Theory approximation. Please make registration on GAMESS website before accessing to GAMESS package deployed at the cluster. To use GAMESS, adding gamess/01-Oct-2010_R1 in your job script. An example job of running GAMESS calculations can be found at /hpc/tmp/examples/gamess and the job script is shown as below: #!/bin/sh #PBS -q workq #PBS -N gamess #PBS -l cput=01:00:00 #PBS -l pvmem=500MB #PBS -m abe #PBS -l nodes=4

NP=‘wc -l < $PBS_NODEFILE‘ VER=gft source /etc/profile.d/00-modules.sh module load gamess/01-Oct-2010_R1

25 4.10 Informatics 4 HOW TO USE

cd $PBS_O_WORKDIR rungms exam01.inp $VER $NP >& exam01.log

4.9.15 OpenBabel

Open Babel is a chemical toolbox designed to speak the many languages of chemical data. It’s an open, collaborative project allowing anyone to search, convert, analyze, or store data from molecular modeling, chemistry, solid-state materials, biochemistry, or related areas. To use Openbabel v2.3, adding module load openbabel/2.3.0 in your job script.

4.10 Informatics

4.10.1 GhostScript

To use GhostScript, add module load ghostscript/8.15 in your job script.

4.10.2 ImageMagick

ImageMagick is a software suite to create, edit, and compose bitmap images. It can read, convert and write images in a variety of formats (over 100) including DPX, EXR, GIF, JPEG, JPEG-2000, PDF, PhotoCD, PNG, Postscript, SVG, and TIFF. Use ImageMagick to translate, ﬂip, mirror, rotate, scale, shear and transform images, adjust image colors, apply various special effects, or draw text, lines, polygons, ellipses and Bezier´ curves. To use ImageMagick, add module load imagemagick/6.6.4 in your job script. NOTE: The GhostScript module has been automatically loaded when loading module ImageMagick.

4.11 Engineering

4.11.1 MATLAB

MATLAB is a high-level technical computing language and interactive environment for algorithm development, data visualization, data analysis, and numeric computation.

26 4.11 Engineering 4 HOW TO USE

MATLAB R2010b and R2011a installed at the HPC cluster support a wide range of toolboxes as listed below:

Bioinformatics Toolbox Parallel Computing Toolbox Communications Blockset Partial Differential Equation Toolbox Communications System Toolbox RF Toolbox Computer Vision System Toolbox Real-Time Windows Target Control System Toolbox Robust Control Toolbox Curve Fitting Toolbox Signal Processing Toolbox DSP System Toolbox SimElectronics Data Acquisition Toolbox SimEvents Econometrics Toolbox SimMechanics Embedded Coder SimPowerSystems Filter Design Toolbox SimRF Financial Toolbox Simscape Fixed-Income Toolbox Simulink Fixed-Point Toolbox Simulink 3D Animation Fuzzy Logic Toolbox Simulink Coder Gauges Blockset Simulink Control Design Global Optimization Toolbox Simulink Design Optimization Image Acquisition Toolbox Simulink PLC Coder Image Processing Toolbox Spreadsheet Link EX Instrument Control Toolbox Stateflow MATLAB Builder NE Statistics Toolbox MATLAB Coder Symbolic Math Toolbox MATLAB Compiler System Identification Toolbox Mapping Toolbox SystemTest Neural Network Toolbox Wavelet Toolbox Optimization Toolbox xPC Target

How to submit MATLAB job depends on differnet parallel models in use:

• Implicit Multithreading MATLAB installed on the cluster support multithreaded computation as default. As mentioned in the MATLAB documentation, multithreading in MATLAB speeds up the following functions:

– Element Wise Functions and Expressions:

∗ Trigonometric: ACOS(x), ACOSH(x), ASIN(x), ASINH(x), ATAN(x), ATAND(x), ATANH(x), COS(x), COSH(x), SIN(x), SINH(x), TAN(x), TANH(x) ∗ Exponential: EXP(x), POW2(x), SQRT(x) ∗ Operators: x.yˆ ∗ Trigonometric: HYPOT(x,y), TAND(x)

27 4.11 Engineering 4 HOW TO USE

∗ Complex: ABS(x) ∗ Rounding and remainder: UNWRAP(x), CEIL(x), FIX(x), FLOOR(x), MOD(x,N), ROUND(x) ∗ Basic and array operations: LOGICAL(X), ISINF(X), ISNAN(X), INT8(X), INT16(X), INT32(X) – Linear Algebra Functions:

∗ Operators: X*Y(Matrix Multiply), XNˆ(Matrix Power) ∗ Reduction Operations :MAX and MIN (Three Input), PROD, SUM ∗ Matrix Analysis:DET(X), RCOND(X), HESS(X), EXPM(X) ∗ Linear Equations: INV(X), LSCOV(X,x), LINSOLVE(X,Y), A(backslash) ∗ Matrix Factorizations: LU(X), QR(X) for sparse matrix inputs ¯ ∗ Other Operations: FFT and IFFT of multiple columns of data, FFTN, IFFTN, SORT, BSXFUN, GAMMA, GAMMALN, ERF, ERFC, ERFCX, ERFINV, ERFCINV, FILTER Please check the list of multithreaded functions on Matlab. Thus it is quite likely that your MATLAB scripts are already multithreaded if above functions invoked. The problem will arise if users assume their MATLAB script is not multithreaded and request a single cpu for their job, but if that job actually uses multiple threads then it will adversely affect other jobs on the cluster. Therefore, please make sure that your MATLAB script only uses the amount of resources requested in the job script. We have two options to submit Matlab jobs: – Use a single core You should restrict MATLAB to one thread with the command-line ﬂag ‘-singleCompThread’ and reserve only one core (nodes=1:ppn=1) in the job script if you are not sure your MATLAB code can use multithreading. An example job script is given below: #!/bin/sh #PBS -q workq #PBS -N test #PBS -l cput=48:00:00 #PBS -l pvmem=1GB #PBS -j eo #PBS -e test.err #PBS -m abe #PBS -l nodes=1:ppn=1

source /etc/profile.d/00-modules.sh module load matlab/r2011a

cd $PBS_O_WORKDIR matlab -singleCompThread -nosplash -nodesktop -r MATLAB_SCRIPT > output

28 4.11 Engineering 4 HOW TO USE

– Use the whole computing node You can request the whole compute node (nodes-=1:ppn=8) and let MATLAB use all available cores if you are sure your MATLAB code invoke many multi-threaded functions and gain dramatic performance enhancement. An example job script is given below: #!/bin/sh #PBS -q workq #PBS -N test #PBS -l cput=48:00:00 #PBS -l pvmem=1GB #PBS -j eo #PBS -e test.err #PBS -m abe #PBS -l nodes=1:ppn=8

source /etc/profile.d/00-modules.sh module load matlab/r2011a

cd $PBS_O_WORKDIR matlab -nosplash -nodesktop -r MATLAB_SCRIPT > output

Before requesting the whole node for your Matlab job, please profile the performance of your multithreaded MATLAB code by comparing the elapsed time using both single core mode and the whole node mode. If the multithreading performance benefit is trivial or slight, you should work in an embarassingly parallel way, i.e. submit multiple jobs with each job working in single core mode. Please note that requesting a whole node might suffer from several hours or days waitting unitl the resource available, espcially when the cluster is busy. If you are not clear on which mode to use, please always use single core mode as above with -singleCompThread flag.

• Parallel Computing Toolbox Parallel Computing Toolbox (PCT) enable users to parallelize their MATLAB code in a general way like MPI but do not need to invoke any external parallel libraries. For details on using PCT, please refer to its manual. By using the Parallel Computing Toolbox, MATLAB could support parallel computing with up to 8 workers/labs within a single node. User must request CPU cores within a single node to run MATLAB parallel jobs. The following MATLAB example job script can be found in /hpc/tmp/examples/matlab/r2011a: #!/bin/sh #PBS -q workq #PBS -N test #PBS -l cput=48:00:00

29 4.11 Engineering 4 HOW TO USE

#PBS -l pvmem=1GB #PBS -j eo #PBS -e test.err #PBS -m abe #PBS -l nodes=1:ppn=2

export NWORKERS=$(cat $PBS_NODEFILE | wc -l) source /etc/profile.d/00-modules.sh module load matlab/r2011a

cd $PBS_O_WORKDIR matlab -singleCompThread -nosplash -nodesktop -r MATLAB_SCRIPT > output

Please put additional two lines in your MATLAB code as below to ensure you are creating consistent number of workers with the requested number of CPU cores. matlabpool(’open’, ’local’, getenv(’NWORKERS’)); YOUR MATLAB CODE; matlabpool close;

In addition, some key functions in several MathWorks products as below offer built-in ability to take advantage of PCT without requiring any extra coding:

• Bioinformatics Toolbox

• Communications System Toolbox

• Embedded Coder

• Global Optimization Toolbox

• Model-Based Calibration Toolbox

• Optimization Toolbox

• Simulink

• SimulinkCoder

• SimulinkControl Design

• SimulinkDesign Optimization

• Statistics Toolbox

• System Test

Check the latest list and their manual for further detail.

30 4.11 Engineering 4 HOW TO USE

4.11.2 ANSYS

ANSYS Mechanical and ANSYS Multiphysics software are non exportable analysis tools incorporating pre-processing (geometry creation, meshing), solver and post-processing modules in a graphical user interface. These are general-purpose ﬁnite element modeling packages for numerically solving mechanical problems, including static/dynamic structural analysis (both linear and non-linear), heat transfer and ﬂuid problems, as well as acoustic and electro-magnetic problems. ANSYS is a commercial licensed software. Please contact ITS for further information.

4.11.3 ABAQUS

ABAQUS suite of engineering analysis software packages is used to simulate the physical response of structures and solid bodies to load, temperature, contact,impact, and other environmental conditions. ABAQUS is a commercial licensed software. Please contact ITS for further information.

4.11.4 Ansoft Maxwell

Maxwell is the premier electromagnetic field simulation software for engineers tasked with designing and analyzing 3-D and 2-D electromagnetic and electromechanical devices such as motors, actuators, transformers, sensors and coils. Maxwell uses the accurate finite element method to solve static, frequency-domain and time-varying electromagnetic and electric fields. A key benefit of Maxwell is its automated solution process where users are only required to specify geometry, material properties and the desired output. From this point, Maxwell will automatically generate an appropriate, efficient and accurate mesh for solving the problem. This proven automatic adaptive meshing process removes complexity from the analysis process and allows engineers to benefit from a highly-efficient, easy-to-use design flow. Maxwell is a commercial licensed software. Please contact ITS for further information.

4.11.5 LAMMPS

LAMMPS stands for Large-scale Atomic/Molecular Massively Parallel Simulator. LAMMPS has potentials for soft materials (biomolecules, polymers) and solid-state materials (metals, semiconductors) and coarse-grain systems. It can be used to model atoms or, more generically, as a parallel particle simulator at the mesoscale or continuum levels. To use LAMMPS, add ‘module load lammps/05Nov10’ in the job script. An example job script is similar to the following ‘jobscript’ ﬁle: #!/bin/sh #PBS -N lammps job #PBS -q workq #PBS -l cput=30:00:00

31 4.12 Biology 4 HOW TO USE

#PBS -l pvmem=1GB #PBS -l nodes=4 #PBS -j eo #PBS -e test.err #PBS -m abe NP=‘wc -l < $PBS_NODEFILE‘ source /etc/profile.d/00-modules.sh module load lammps/05Nov10 cd $PBS_O_WORKDIR mpirun -prefix /hpc/software/openmpi/1.4.2 -x PATH -machinefile $PBS_NODEFILE -np $NP lmp_openmpi < input >& output

Submit the job to PBS by using %qsub jobscript

4.11.6 Materials Studio

Materials Studio is a comprehensive materials modeling and simulation application designed for scientists in chemicals and materials R&D as well as pharmaceuticals development. At present, we have 3 modules installed on the cluster: amorphouscell, discover and compass. Materials Studio is a commercial licensed software. Please contact ITS for further information.

4.12 Biology

4.12.1 MrBayes

MrBayes is a program for the Bayesian estimation of phylogeny. Bayesian inference of phylogeny is based upon a quantity called the posterior probability distribution of trees, which is the probability of a tree conditioned on the observations. The conditioning is accomplished using Bayes’s theorem. The posterior probability distribution of trees is impossible to calculate analytically; instead, MrBayes uses a simulation technique called Markov chain Monte Carlo (or MCMC) to approximate the posterior probabilities of trees. An example job script on using MrBayes is similar to the following ‘jobscript’ ﬁle: #!/bin/sh #PBS -q workq #PBS -N mrbayes jobs #PBS -l cput=01:00:00 #PBS -l pvmem=1GB #PBS -j eo #PBS -e test.err #PBS -m abe

32 4.12 Biology 4 HOW TO USE

#PBS -l nodes=4

NP=‘wc -l < $PBS_NODEFILE‘ source /etc/profile.d/00-modules.sh module load mrbayes/3.1.2 cd $PBS_O_WORKDIR mpirun -machinefile $PBS_NODEFILE -np $NP mb bglobin.nex >& bglobin.log

Submit the job to PBS by using %qsub jobscript

4.12.2 ATSAS

ATSAS is a program suite for small-angle scattering data analysis from biological macromolecules. To use ATSAS, add ‘module load atsas/2.4.1-4’ in your job script.