Chemistry Packages at CHPC Anita M. Orendt Center for High Performance Computing [email protected] Fall 2014 Purpose of Presentation • Identify the computational chemistry software and related tools currently available at CHPC • Present brief overview of these packages • Present how to access packages on CHPC • Information on usage of Gaussian09 http://www.chpc.utah.edu Ember Lonepeak /scratch/lonepeak/serial 141 nodes/1692 cores 16 nodes/256 cores Infiniband and GigE Large memory, GigE General 70 nodes/852 cores 12 GPU nodes (6 General) Kingspeak 199 nodes/4008 cores Ash cluster (417 nodes) Infiniband and GigE Switch General 44 nodes/832 cores Apex Arch Great Wall PFS Telluride cluster (93 nodes) /scratch/ibrix meteoXX/atmosXX nodes NFS Home NFS Administrative Directories & /scratch/kingspeak/serial Group Nodes Directories 10/1/14 http://www.chpc.utah.edu Slide 3 Brief Overview CHPC Resources • Computational Clusters – kingspeak, ember – general allocation and owner-guest on owner nodes – lonepeak – no allocation – ash and telluride – can run as smithp-guest and cheatham-guest, respectively • Home directory – NFS mounted on all clusters – /uufs/chpc.utah.edu/common/home/<uNID> – generally not backed up (there are exceptions) • Scratch systems – /scratch/ibrix/chpc_gen – kingspeak, ember, ash, telluride – 56TB – /scratch/kingspeak/serial – kingspeak, ember, ash – 175TB – /scratch/lonepeak/serial – lonepeak – 33 TB – /scratch/local on compute nodes • Applications – /uufs/chpc.utah.edu/sys/pkg (use for lonepeak, linux desktops if chpc admined) – /uufs/$UUFSCELL/sys/pkg where $UUFSCELL = kingspeak.peaks, ember.arches, ash.peaks, telluride.arches http://www.chpc.utah.edu Getting Started at CHPC • Account application – an online process – https://www.chpc.utah.edu/apps/profile/account_request.php • Username is your unid with password administrated by campus • Interactive nodes – two CHPC owned nodes per cluster (cluster.chpc.utah.edu) with round- robin access to divide load (on ash, use ash-guest.chpc.utah.edu) • CHPC environment scripts – in account when created – www.chpc.utah.edu/docs/manuals/getting_started/codes/chpc.tcshrc – www.chpc.utah.edu/docs/manuals/getting_started/codes/chpc.bashrc • Getting started guide – www.chpc.utah.edu/docs/manuals/getting_started • Problem reporting system – http://jira.chpc.utah.edu or email to [email protected] http://www.chpc.utah.edu Interactive Node Usage • Interactive nodes for prepping/testing of input files, analyzing results, compilations, debugging, data transfer, etc – no running of jobs – 15 min MAX cpu – no jobs of ANY time length that negatively impact ability of other users to get work done (e.g., heavy cpu, memory usage and/or i/o) • https://wiki.chpc.utah.edu/display/policy/2.1.1+Cluster +Interactive+Node+Policy http://www.chpc.utah.edu Batch System • All use of compute nodes go through a batch system using Torque (PBS) and Maui (Moab) for scheduling #PBS -S /bin/csh #PBS – A account #PBS –l walltime=24:00:00,nodes=2:ppn=16 • qsub –I to get interactive use of a compute node (-X if X11 forwarding needed) • Walltime limits – 72 hours (long qos by request) on all clusters except ash which is 24 hours http://www.chpc.utah.edu Security Policies • No clear text passwords - use ssh and scp • Do not share your account under any circumstances • Don’t leave your terminal unattended while logged into your account • Do not introduce classified or sensitive work onto CHPC systems except on protected environment • Do not try to break passwords, tamper with files, look into anyone else’s directory, etc. – your privileges do not extend beyond your own directory • Do not distribute or copy privileged data or software • Report suspicions to CHPC ([email protected]) • Please see http://www.chpc.utah.edu/docs/policies/security.html for more details http://www.chpc.utah.edu Access to Interactive Nodes • From Windows machine: – Need ssh client • PuTTY: http://www.chiark.greenend.org.uk/~sgtatham/putty/ • Xshell 4: http://www.netsarang.com/products/xsh_overview.html – For Xwindowing – need tool to display graphical interfaces such as Gaussview, VMD, AutoDockTools • Xming (use Mesa version) - http://www.straightrunning.com/XmingNotes – FastX another option which replaces both of the above (and can be used on linux and Mac desktops also) • https://wiki.chpc.utah.edu/display/DOCS/FastX http://www.chpc.utah.edu Default login scripts • CHPC maintains default login scripts that will set up necessary environment for batch commands and many of the programs to work – http://www.chpc.utah.edu/docs/manuals/getting_started/code/chpc.tcshrc – http://www.chpc.utah.edu/docs/manuals/getting_started/code/chpc.bashrc • Located in your home directory as .tcshrc or .bashrc • Can comment out setups for packages not used • Default ones provided have chemistry package setups commented out – need to remove # at start of line • Can customize by creating .aliases file that is sourced at end of the CHPC script – do not customize .tcshrc/.bashrc file as you will need to occasionally get a new version http://www.chpc.utah.edu Software - General Information • Available on CHPC web pages/wiki – Can get to from www.chpc.utah.edu → Software documentation → More (for complete list) http://www.chpc.utah.edu/docs/manuals/software – Available for widely used packages – Has information on licensing restrictions, example batch scripts, where to get more information on a specific package – Also has useful information on running of jobs (scaling, issues, etc) for some applications http://www.chpc.utah.edu Computational Chemistry Packages • Molecular Mechanics/Dynamics • Semi Empirical • Electronic Structure Calculations – HF, DFT, MPn, CI, CC, multireference methods etc • Solid state/Materials Modeling packages – Plane Wave Pseudopotential based packages Calculation type & level of theory used depends on a number of factors – the information you are after, system being studied, size of system, computational resources. Often consider accuracy versus computational cost http://www.chpc.utah.edu Molecular Mechanics/Dynamics • Amber – http://ambermd.org • Gromacs – http://www.gromacs.org/ • NAMD – http://www.ks.uiuc.edu/Research/namd/ • LAMMPS – http://lammps.sandia.gov/ • Charmm – http://www.charmm.org/ – licensed by research group Note – Gaussian and NWChem also have some MM capabilities http://www.chpc.utah.edu MM/MD packages • Versions: – Amber14 – Gromacs 4.6.3 (4.6.5 in chp.utah.edu) – Lammps – package is updated often so installs are different versions; see date on installation. • Basic information on getting started: – https://wiki.chpc.utah.edu/display/DOCS/Amber – https://wiki.chpc.utah.edu/display/DOCS/Gromacs – https://wiki.chpc.utah.edu/display/DOCS/LAMMPS • Different parallel builds available for each cluster to make use of the different interconnects http://www.chpc.utah.edu Semi-Emprical Packages • These are based on the Hartree-Fock formalism of electronic structure methods, but make many approximations and obtain some parameters from empirical data • MOPAC2012 is the only semi-empirical only package installed at CHPC • Installation at /uufs/chpc.utah.edu/sys/pkg/mopac • How to run at https://wiki.chpc.utah.edu/display/DOCS/MOPAC • Gaussian, GAMESS also has some capabilities http://www.chpc.utah.edu Quantum (Mainly) Packages • Gaussian09 • NWChem There are other packages not currently installed available free of charge, e.g., • GAMESS - http://www.msg.ameslab.gov/gamess/ (general purpose, properties) • Psi4 – http://www.psicode.org/ (general purpose, properties) • CFOUR – http://www.cfour.de/ (coupled cluster) • SIESTA – http://departments.icmab.es/leem/siesta/ (DFT, linear scaling) http://www.chpc.utah.edu Gaussian09 • Commercial electronic structure package – http://www.gaussian.com for information and User’s Guide • Version D01 of G09 installed – /uufs/chpc.utah.edu/sys/pkg/gaussian09/EM64T – Also have legacy EM64T, AMD64, and legacy AMD64 builds • For information on accessing the CHPC installation • http://www.chpc.utah.edu/docs/manuals/software/g09.html • More later http://www.chpc.utah.edu NWChem • Package developed at PNNL to work on massively parallel systems • http://www.nwchem-sw.org • Goal: Computational chemistry solutions that are scalable with respect to both chemical system size and MPP hardware size • Has quantum mechanics, molecular mechanics/dynamics, and quantum molecular dynamics, plane waves for periodic systems • Version 6.3 ( NOT with Python support) – /uufs/chpc.utah.edu/sys/pkg/nwchem/nwchem-6.3/bin/LINUX64 – /uufs/ember.arches/sys/pkg/nwchem/nwchem-6.3/bin/LINUX64 – /uufs/kingspeak.peaks/sys/pkg/nwchem/nwchem-6.3/bin/LINUX64 – /uufs/chpc.utah.edu/sys/pkg/nwchem/nwchem-6.3/bin/LINUX64 • To run: – source appropriate etc/nwchem.csh • More information and example batch script at – https://wiki.chpc.utah.edu/display/DOCS/NWChem http://www.chpc.utah.edu Soid State/ Materials Packages Plane wave codes for the study of systems under periodic boundary conditions (PBC) • NWChem has some functionality and more is being added • VASP – http://cms.mpi.univie.ac.at/vasp/ – Ab initio QM/MD – Licensed per research group (have 1 group with an old version) • Quantum Espresso - http://www.quantum-espresso.org/ – Understands crystal space groups – Has GIPAW module to do NMR calculations – J-ICE and XCrysDen to view • CPMD - http://www.cpmd.org/ • BigDFT - http://inac.cea.fr/L_Sim/BigDFT/ http://www.chpc.utah.edu Support Packages • Molecular Viewers • Gaussview • Molden • VMD • Chimera • Babel (Openbabel)