Introduction to Unix/

Ken-ichi Nomura, Ph.D. [email protected] Center for High-Performance Computing and Communications Outline 1. HPCC Overview 2. Basic Commands and Utilities 3. 4. Directory, Software and Account Issues 5. Introduction to Parallel Computing – Compilers (, Fortran, MPI) – Portable Batch System HPCC Overview

HPCC clusters: - HP SL230 (128GB memory), HP SL250 HPCC (64 GB memory, dual NVIDIA K20m GPUs), Dual Xeon 8-core 2.4 GHz, on Infiniband Myrinet 56.6-Gbit FDR Infiniband network - 2,225-node, 4-core, 6-core, and 12-core dual-processor cluster contains Dell, Oracle Sun, HP, and IBM compute nodes on a 10-Gbit Myrinet network hpc-login1 - 4 large-memory nodes with 1 TB of hpc-login2 RAM and 4x10-core Intel Xeon processors. - 2 head nodes, hpc-login1.usc.edu and hpc-login2.usc.edu Internet HPCC Overview HPCC website: http://hpcc.usc.edu/

- About - Research - User Support - New User Guide - Frequently Asked Questions - Contact Us HPCC Website: HPCC Policies

http://hpcc.usc.edu/support/accounts/hpcc-policies/

5 HPCC Website: Account Application

http://hpcc.usc.edu/support/accounts/applying-for-a-hpcc-account/

6 Machine Room Outline 1. HPCC Overview 2. Basic Commands and Utilities 3. Text Editor 4. Directory, Software and Account Issues 5. Introduction to Parallel Computing – Compilers (C, Fortran, MPI) – Portable Batch System What is Linux? Graphical User Interface

Operating System (OS) Command Line Interface Basic Commands

ls List file and/or directory names pwd Print working (current) directory mkdir/rmdir Create/remove directory cd Change directory chmod Change file or directory permission cp/mv Copy/move file or directory rm Remove file or directory cat/more/less Display file contents man Display online manual Basic Commands: Example

[~]$ ls docs hpc-23 Makefile util workshop [~]$ ls –a … [~]$ pwd /home/rcf-12/knomura [~]$ mkdir tmp [~]$ cd tmp/ [tmp]$ cp ~/.bash_profile dot_bash_profile [tmp]$ cat dot_bash_profile [tmp]$ ls -l dot_bash_profile [tmp]$ chmod u-r dot_bash_profile [tmp]$ cat dot_bash_profile cat: dot_bash_profile: Permission denied [tmp]$ chmod u+r dot_bash_profile Basic Commands: Example [tmp]$ cp dot_bash_profile DOT_bash_profile [tmp]$ ls dot_bash_profile DOT_bash_profile [tmp]$ ls -l *_bash_profile -rw-r--r-- 1 knomura m-cacs 208 Oct 3 11:43 dot_bash_profile -rw-r--r-- 1 knomura m-cacs 208 Oct 3 11:47 DOT_bash_profile [tmp]$ mv dot_bash_profile Dot_Bash_Profile [tmp]$ ls –la … [~]$ cd .. [~]$ ls [~]$ pwd [~]$ rm -rv tmp/ removed `tmp/DOT_bash_profile' removed `tmp/Dot_Bash_Profile' removed directory: `tmp' Shell ( and tcsh) • Shell is a command processor by which users tell what to do to system. • Command-line Interface (CLI) or Character user interface (CUI) • To check which shell you are using, type echo $0 • Auto logout after 20 minuets of inactivity (for HIPPA compliant) Bash Config Files • Configuration files to setup user environment, for examples command prompt, path, alias so on. Sometimes called “dot file” • .bash_profile and .bashrc are stored in each user’s home directory • When bash is invoked as a login shell, it first reads /etc/profile, if that file exists, then looks for .bash_profile and .profile • When bash is invoked as a non-login shell, only read .profile Csh & Tcsh Config Files

• .login & .cshrc are in each user’s home directory • When tcsh is invoked as a login shell, it reads first .tcshrc or, if .tcshrc is not found, .cshrc, then .history, then .login, and finally .cshdirs • When tcsh is invoked as a non-login shell, only read /etc/csh.cshrc and .cshrc Permission & Ownership

- File and directory have ownership and permission - Three types of permission, readable, writeable and executable - Each permission is given to three groups, owner, group and others

[~]$ ls -l .bash_profile -rw-r--r-- 1 knomura m-cacs 208 Sep 26 14:11 .bash_profile

r readable, w writable, x executable u user (owner), g group, o others, a all

Permission & Ownership chmod Change file/directory permission chown Change file/directory owner Example) chmod a+w file Add W permission to all users chmod o-x file Remove E permission from others chmod 750 file Add R/W/E permission to myself, R/E permission to group no permission to others chown usr:grp file Change ownership of a file/directory r(4) readable, w(2) writable, x(1) executable u(I) user (owner), g(II) group, o(III) others, a all Task 1: Change Permission and Run clock.sh

#!/bin/bash 1. Add executable permission to clock.sh while true ; do 2. Run it echo -n " " date +"%r" 3. Terminate the script by Ctrl+c sleep 1 done [~]$ ./clock.sh 02:21:02 PM 02:21:03 PM 02:21:04 PM … Useful Commands env display environmental variables echo/export display/set an environmental variable grep print lines matching a pattern head/tail show first/last several lines sort sort text ps/top display currently running jobs kill/killall terminate a process (not for PBS job) tar archive/unarchive files bc calculator Useful Commands: Example

[~]$ env Archive multiple files/directories into [~]$ echo $PATH one file, then compress it. [~]$ export PATH=$PATH:~/bin [~]$ tar cvfz out.tar.gz file1 file2 dir1 [~]$ env | grep PATH Unzip and extract from an archived file. [~]$ env > env.txt [~]$ tar xvfz output.tar.gz [~]$ cat env.txt Display archived file contents [~]$ head (or tail) env.txt [~]$ tar tvfz output.tar.gz [~]$ grep -n knomura env.txt [~]$ sort env.txt | grep -n knomura > env2.txt [~]$ ps -u knomura [~]$ ps -ef | grep knomura [~]$ top [~]$ kill 123456 Special Characters/Commands ~ home directory . current directory .. parent directory * wild card > redirect output >> redirect output (append) <<, <<< here document/string | pipe `command` back quotes, replace `…` with command output Special Characters/Commands

tab Complete name of command, file or directory clear/ctrl+l Clear terminal ctrl+a Go to the beginning of line ctrl+e Go to the end of line history Display command history éê Display command history ! Rerun a command in history Outline 1. HPCC Overview 2. Basic Commands and Utilities 3. Text Editor 4. Directory, Software and Account Issues 5. Introduction to Parallel Computing – Compilers (C, Fortran, MPI) – Portable Batch System Text Editor: GNU nano

http://www.nano-editor.org/ Text Editor: GNU nano

Arrow-keys Move cursor Enter Change line CTRL+a Move to the beginning of line CTRL+e Move to the end of line CTRL+v Move forward one page CTRL+y Move backward one page Text Editor: GNU nano

CTRL+o Save file CTRL+w Search text CTRL+d Delete a character CTRL+k Remove a line CTRL+u Paste buffer CTRL+x Save data and exit Task 2: bash scripting

1 #!/bin/bash 2 pi.sh 3 # function definition 4 function pi { 5 6 # max number of RNG. use this for normalization. 7 RANDMAX=32767 8 9 # reset the number dot within a circle 10 in=0 11 for ((n=0; n<${1}; n++)); do 12 x=${RANDOM} 13 y=${RANDOM} 14 z=`echo "(${x}^2 + ${y}^2)/(${RANDMAX}*${RANDMAX})" | bc` 15 16 [ $z -lt 1 ] && let in+=1; 17 done 18 } Task 2: pi.sh

20 # read total number of trial, number of procs 21 echo -n "type total number of trial: " 22 read total 23 24 # call the PI estimator 25 pi ${total} 26 27 echo "total trial=${total} PI = `echo ${in}/${total}*4 | bc -l`"

1. Fix pi.sh with text editor 2. Try 10, 50, 100, 1000 for the total number of trail to see how the result improves Outline 1. HPCC Overview 2. Basic Commands and Utilities 3. Text Editor 4. Directory, Software and Account Issues 5. Introduction to Parallel Computing – Compilers (C, Fortran, MPI) – Portable Batch System System Directories

/bin, /sbin executable files (commands) /etc system config files /home users home directories /lib, /lib64, /usr/lib64 library files /proc process information /tmp temporary space /usr/usc software repository Directory and Quota

• Home directory /home/rcf-xx/username Each user has 1 GB of disk quota and 100,000 files of file quota

• Project directory /home/rcf-proj/projectid Quota on project directory varies depending on each project. If you need more space, send a request from HPCC website Monitoring Your Quota: myquota myquota shows the quota on your home & project directories. ------• Program may crash due to Disk Quota for knomura ID 55302 the quota, being unable to Used Soft Hard write files in either home Files 8146 100000 101000 directory or project Bytes 839.56M 4.30G 4.30G ------directory Disk Quota for /home/rcf-proj/hpcc ID 419 • Pay attention to the file Used Soft Hard quota (max number of Files 400792 1000000 1100000 files) also. Bytes 236.94G 400.00G 402.00G ------Account, Balance & core-hour • To be able to submit your jobs to the HPCC cluster, your account needs some balance (core-hour) • Whenever your job finishes (successfully or unsuccessfully), your account is charged by the number of processors used times the wallclock time the job spent Example) If you request 2 nodes with 4 processors per nodes for 2 hours (-l nodes=2:ppn=4,walltime=2:00:00), the total charge is 2x4x2 = 16 core-hour. Monitoring Your Balance: mybalance mybalance shows the current balance of your account [~]$ mybalance -h Balance Name ------11106267.27 lc_an 20000.00 lc_kn1 - Job will not start if your balance is not enough. - Submit a request from HPCC website for additional core-hour. - Make sure the ppn value matches with the number of processors actually used. (Don’t use ppn=12 for single- core job!) Software Repository: /usr/usc

Many applications are available. Compilers: , intel, pgi Numerical Libraries: mpich, openmpi, cuda, fftw, petsc Molecular Simulation: NAMD, gromacs, amber Quantum Chemistry: gaussian, schrodinger Numerical Environment: matlab, R, python

hpc-login1.usc.edu for 32-bit applications hpc-login2.usc.edu for 64-bit applications Software Repository: /usr/usc How to use the installed software on HPCC: - Find application and version you want to use - Source setup script. There are two setup scripts, one for bash the other for tcsh Example) Matlab R2013a

[~]$ ls /usr/usc/matlab/ 2009a 2009b 2010b 2011a 2013a 2013b default Scripts [~]$ source /usr/usc/matlab/2013a/setup.sh [~]$ matlab -nodisplay < M A T L A B (R) > Copyright 1984-2013 The MathWorks, Inc. R2013a (8.1.0.604) 64-bit (glnxa64) February 15, 2013 >> System Software vs Repo Software

• Some software and libraries (e.g. gcc, python, fftw) come with OS • Although command name is the same, the system software and repo software are often different (versions, libraries, developers). Make sure that you use what you want to use • which command shows the absolute path of a command

[~]$ which python /usr/bin/python [~]$ source /usr/usc/python/enthought/default/setup.sh [~]$ which python /usr/usc/python/enthought/default/bin/python Outline 1. HPCC Overview 2. Basic Commands and Utilities 3. Text Editor 4. Directory, Software and Account Issues 5. Introduction to Parallel Computing – Compilers (C, Fortran, MPI) – Portable Batch System C & Fortran Compiler HPCC supports GNU/Intel/PGI compilers [~]$ source /usr/usc/gnu/gcc/default/setup.sh [~]$ source /usr/usc/intel/default/setup.sh [~]$ source /usr/usc/pgi/default/setup.sh

Compiler names) GNU: gcc, gfortran Intel: icc, ifort PGI: pgcc, pgfortran Message Passing Interface (MPI) Library and Compiler • Standard for high-performance parallel computing library. • Most of high performance computing (HPC) applications are MPI plus X, (X=OpenMP, Pthread, CUDA, MPI.. ) • There are a number of MPI implementations. MPICH & OpenMPI are installed on HPCC. • Use wrapper scripts (mpicc, mpicxx, mpif77, mpif90) to compile MPI codes. Message Passing Interface (MPI) Library and Compiler

To call the MPI compilers, [~] $ source /usr/usc/openmpi/1.6.4/setup.sh [~] $ mpicc mpihello.c You will have a.out

ldd command shows list of dynamically linked libraries [~]$ ldd a.out

Task 3: Compile MPI Hello World

#include #include mpihello.c

int main(int argc, char **argv) { int *buf, i, rank, nints, len; char hostname[256];

MPI_Init(&argc,&argv); MPI_Comm_rank(MPI_COMM_WORLD, &rank); gethostname(hostname,255); printf("Hello world! I am process number: %d on host %s\n", rank, hostname);

MPI_Finalize(); return 0; } Task 3: Compile MPI Hello World

1. To compile an MPI code, source mpi library setup script first. [~]$ source /usr/usc/openmpi/default/setup.sh

2. Use mpicc to compile MPI code (use mpif90 for Fortran) [~]$ mpicc mpihello.c

3. You should have an executable file (a.out) Portable Batch System: qsub qsub - submit a job to cluster. -I interactive job -d starting directory -l resource list nodes number of nodes ppn processor per core mem amount of memory gpus number of GPUs walltime time to run your job -T allcores request all CPU cores Portable Batch System: PBS Script PBS script is a text file which contains 1. what computing resources are necessary for your job 2. list of commands to be executed #!/bin/sh #PBS -l nodes=1:ppn=12,walltime=00:30:00 #PBS -d . #PBS -N test pwd hostname echo Hello World!

To submit a job, type qsub yourscript.sh You can overwrite PBS options from command line [~]$ qsub yourscript.sh -l nodes=4:ppn=8,mem=1gb,walltime=12:00:00 Portable Batch System: qstat & qdel

qstat - show status of your jobs: -a all jobs are displayed -u username display status of specific user’s job -f jobid display full status of a specific job

qdel - delete a job: qdel jobid delete a PBS job [~]$ qstat -u knomura Job ID Username Queue Jobname SessID NDS TSK Memory Time S Time ------5286386.hpc-pbs. knomura main STDIN 33471 1 2 -- 02:00 C 00:53 5287425.hpc-pbs. knomura main STDIN -- 8 8 -- 00:30 R --

[~]$ qstat -f 5287425 Job Id: 5287425.hpc-pbs.hpcc.usc.edu Job_Name = STDIN job_state = R queue = main ctime = Mon Sep 30 16:21:26 2013 Error_Path = /home/rcf-12/knomura/STDIN.e5287425 exec_host = hpc1930/0+hpc1936/0+hpc1959/0+hpc1962/0+ hpc2023/0+hpc2024/0+hpc2025/0+hpc2026/0 Output_Path = /home/rcf-12/knomura/STDIN.o5287425 Resource_List.walltime = 00:30:00 Variable_List = PBS_O_QUEUE=default, PBS_O_HOME=/home/rcf-12/knomura,PBS_O_LANG=en_US.UTF-8, PBS_O_LOGNAME=knomura, etime = Mon Sep 30 16:21:40 2013 submit_args = -I -d . -l nodes=8 start_time = Mon Sep 30 16:22:42 2013 Walltime.Remaining = 1779 init_work_dir = /home/rcf-12/knomura ..... 47 Task 4: Run MPI Hello World Code

1. Fix run.sh 2. Submit a PBS job to the HPCC cluster. [~]$ qsub run.sh 3. Change the number of nodes and processor per nodes, then submit again 4. Monitor your job using qstat. #!/bin/bash #PBS -l nodes=4:ppn=4 run.sh [~]$ qstat -u yourname #PBS -l walltime=00:30:00 #PBS -N mpihello #PBS -d .

source /usr/usc/openmpi/1.6.4/setup.sh mpirun -np 16 ./a.out Thank you!