Intermediate Shell Scripting.Key

Intermediate Shell Scripting

11/11/2015 Dr.)Yongjun)Choi) Research)Specialist,) Institute)for)CyberAEnabled)Research) Evaluation Form

• Please don’t forget to complete your evaluation form before you leave. How this workshop works

• We are going to cover some intermediate topics. – Lots of hands on examples • Exercises are denoted by the following icon in this presents: Green and Red Sticky

• Use the sticky notes provided to help me help you. – No Sticky = I am working – Green = I am done and ready to move on (yea!) – Red = I am stuck and need more time and/or some help examples

• login • connect to dev-intel10 • module load powertools • getexample shell_scripting Agenda

• Review – Hello, ${world} – basic commands – grep/sed/find – Flow control • Advanced shell scripting – ex6 – ex7 Hello, ${world}

hellow_world.sh

!/bin/bash msg1=world! echo “Hello, ${msg1}”

• It does NOT work. Please fix it. Hello, ${world}

hellow_world2.sh

#!/bin/bash echo Hello, %{1}!

• It does NOT work. Please fix it. Hello, ${world}

hellow_world3.sh #!/bin/bash echo ‘msg=“Hello World!”’ > hello echo “echo $msg” >> hello chmod 000 hello ./hello Command substitution

• $(command) • `command`

dir.sh

#!/bin/bash #!/bin/bash dir=`ls -l` dir=`ls -l` echo $dir echo “$dir” Arithmetic Expansion

• $((expression))

#!/bin/bash echo $((1+3+4))

#!/bin/bash #!/bin/bash echo $((ls)) echo $( (ls) ) Redirection and pipelines

• use > to redirect stdout to a file: ls > file • Use < to specify that we: grep searchterm < file • Use >> to append stdout to a file – date > date.out – cal >> date.out • Use 2> to redirect standard error to a file – commandA 2>1; commandA 2>&1 • Use pipelines to chain commands – ls -l |grep searchword | sort -r grep

• grep pattern file • grep pattern < file • grep -i pattern file • grep -v pattern file • eg grep HPCC sample.txt grep -i hpcc sample.txt grep -v hpcc sample.txt sed

• sed (short for “steam editor”) is used to filter and transform text. • eg echo day |sed s/day/night echo My night is more beautiful than your day |sed s/day/ night/ Change HPCC to hpcc in sample.txt sed -i s/HPCC/hpcc/g sample.txt if…then…elif…else…fi

• The generic form if list then list [elif list then list]… [else list] fi • The parts enclosed within ([) and (]) are optional. if…then…elif…else…fi

dev-intel10:warmingUp$ sh if_elif.sh 1 The first choice is nice dev-intel10:warmingUp$ sh if_elif.sh 2 The second choice is just as nice dev-intel10:warmingUp$ sh if_elif.sh 3 The third choice is excellent dev-intel10:warmingUp$ sh if_elif.sh 4 I see you were wise enough not to choose You win dev-intel10:warmingUp$ do…while

• The generic form While list do list done do…while

dev-intel10:warmingUp$ sh do_while.sh 5 5 seconds till snack time! 4 seconds till snack time! 3 seconds till snack time! 2 seconds till snack time! 1 seconds till snack time! snack time!!, YEAH!! dev-intel10:warmingUp$ for

• The generic form For variable in word … do list done for

fruitlist="Apple Pear Tomato Peach Grape"

dev-intel10:warmingUp$ sh for.sh I like Apples I like Pears I like Tomatoes I like Peaches I like Grapes dev-intel10:warmingUp$ case

• The generic form case word in pattern list ;; … esac case

dev-intel10:warmingUp$ sh case.sh 1 First Choice dev-intel10:warmingUp$ sh case.sh 2 Second Choice dev-intel10:warmingUp$ sh case.sh 3 Other Choice dev-intel10:warmingUp$ sh case.sh 1234 Other Choice dev-intel10:warmingUp$ Functions

• The generic form name( ) { commands } Functions

dev-intel10:warmingUp$ sh function1.sh 1 2 10 1 3 5 7 9 dev-intel10:warmingUp$ Expected results

dev-intel10:warmingUp$ sh function1.sh 1 2 10 1 3 5 7 9 dev-intel10:warmingUp$ Intermediate shell scripting

• Preparation – getexample shell_scripting_examples – cd shell_script_examples/ex6 • We will develop a shell script which manipulates files & directories, and submits a queue script. ex6 directory structure

ex6

common.sh ex6* origin ex6.cpp input.dat output.dat unWantedFile.tar.gz Anatomy of ex6.cpp

ex6.cpp

#include #include input.dat using namespace std; char inputString[1000]; 0. double param[3]; 3. int main (int argc, char **argv) { 7.

ifstream inFile(argv[1]); int ncount = 0; //counter

while(!inFile.eof()) { //read parameters inFile>>param[ncount]; ncount++; output.dat } inFile.close(); 1. 4. ofstream outFile("output.dat"); //write parameters 8. for (int icount = 0; icount < ncount-1; icount++) outFile<<“ “<

return 0; } Our goal is…

• Write a shell script which: – imports a shell script which builds ex6 executable at origin. – creates six subdirectories (0, 1, 2, 3, 4, & 5) under data directory (data/0, data/1,…, data/5) – copies all files except unWantedFile.tar.gz from origin to each subdirectory – manipulates the input parameters in input.dat on each subdirectory (0, 1, 2, 3, 4, & 5) – submits ex6.qsub at each directory and redirect all output including error message to standard output – clears out unnecessary files such as ex6 at each directory Our goal

ex6

origin data common.sh ex6 common.sh ex6.cpp ex6 0 1 2 3 4 5 input.dat ex6.cpp output.dat input.dat unWantedFile.tar.gz Base script template

#!/bin/sh DDIRS=“0 1 2 3 4 5" #directory names in which data will be saved. SIMDIRS='data' #parent directory of data directories FROM_DIR='./origin/' #location of original files

• We will use this template. • Please create this shell script and save it as ex6.sh under ex6 directory. We will expand ex6.sh for the rest of the workshop Our goal is…

• Build ex6 executable at origin. – You can use other shell scripts in a script to import functions/ commands – Use ./origin/common.sh in ex6.sh to build ex6 executable

common.sh

g++ -O3 -o ex6 ex6.cpp Sample script

#!/bin/sh DDIRS=“0 1 2 3 4 5" #directory names in which data will be saved. SIMDIRS='data' #parent directory of data directories DDIRS FROM_DIR='./origin/' #location of original files

cd $FROM_DIR ./common.sh cd .. Our goal is…

• Create six subdirectories (0, 1, 2, 3, 4, & 5) under data directory (data/0, data/1,…, data/5). – Do not do this manually. Please expand ex6.sh

data

0 1 2 3 4 5 Sample script

#!/bin/sh DDIRS=“0 1 2 3 4 5" #directory names in which data will be saved. SIMDIRS='data' #parent directory of data directories FROM_DIR='./origin/' #location of original files

for dd in $DDIRS; do SIM_DIR=$SIMDIRS/$dd mkdir -p $SIM_DIR done Our goal is…

copy all files except unWantedFile.tar.gz from origin to each subdirectory

origin data

common.sh ex6 0 1 2 3 4 5 ex6.cpp input.dat unWantedFile.tar.gz common.sh ex6 ex6.cpp input.dat Sample script

#!/bin/sh DDIRS="0 1 2 3 4 5" #directory names in which data will be saved. SIMDIRS='data' #parent directory of data directories FROM_DIR='./origin/' #location of original files

IFILES=`ls $FROM_DIR`

for dd in $DDIRS; do SIM_DIR=$SIMDIRS/$dd mkdir -p $SIM_DIR

for inif in $IFILES; do if [ $inif != "unWantedFile.tar.gz" ]; then cp $FROM_DIR/$inif $SIM_DIR/ fi done done Our goal is…

• Replace input parameters – the first input parameter (0) in input.dat to each name of subdirectory (0, 1, 2, 3, 4, 5)

data/ origin/input.dat 0/input.dat … 3/input.dat … 5/input.dat

0. 0. 3. 5. 3. 3. 3. 3. 7 7. 7. 7. Sample script

#!/bin/sh for inif in $IFILES; do DDIRS="0 1 2 3 4 5" #directory names in if [ $inif != "unWantedFile.tar.gz" ]; then which data will be saved. cp $FROM_DIR/$inif $SIM_DIR/ SIMDIRS='data' #parent directory of fi data directories done FROM_DIR='./origin/' #location of original files SSTR="0" SARP="s/$SSTR/$dd/g" IFILES=`ls $FROM_DIR` sed -i $SARP $SIM_DIR/input.dat for dd in $DDIRS; do SIM_DIR=$SIMDIRS/$dd done mkdir -p $SIM_DIR Step 5: More manipulation

• Change walltime – from 1min to num+1 mins in data/num/ex6.qsub • Change jobname – from 0.jobname to num.jobname

– num = 0, 1, 2, 3, 4, & 5 Sample script

#!/bin/sh DDIRS="0 1 2 3 4 5" #directory names in which data will be saved. SSTR="0" SIMDIRS='data' #parent directory of data SARP="s/$SSTR/$dd/g" directories sed -i $SARP $SIM_DIR/input.dat FROM_DIR='./origin/' #location of original files

IFILES=`ls $FROM_DIR` SSTR1="00:01:00" SSTR2="00:$dd:00" for dd in $DDIRS; do SIM_DIR=$SIMDIRS/$dd SARP="s/$SSTR1/$SSTR2/g" mkdir -p $SIM_DIR sed -i $SARP $SIM_DIR/ex6.qsub

done for inif in $IFILES; do if [ $inif != "unWantedFile.tar.gz" ]; then cp $FROM_DIR/$inif $SIM_DIR/ fi done

Our goal is…

• Submit ex6.qsub at each directory and redirect all output including error message to standard output • Check if your shell script works as you expect. One more example

• If you submit a queue script with walltime is less than four hours, it is easier to get resource. • What should I do if I need to run my job more than four hours, and want to use these short job nodes? – Sit in front of the screen, wait until your job is finished, resubmit the script, and repeat this line – Have a student to work for you – Use BLCR (Berkley Lab Check point Restart) or – write a shell script Sample script: ex7.qsub

#!/bin/bash --login #PBS -l nodes=1:ppn=1,walltime=00:01:00,mem=200mb #PBS -j oe

cd ${PBS_O_WORKDIR}

# Start job (mpiexec -np 1 ex7 input.dat)&

#get the process id mypid=$!

#wait for the process to complete wait ${mypid}

#variables for flow control istep=1234; nstep=1243

if [ "$istep" -lt "$nstep" ]; then cp output.dat output_${istep}.dat cp output.dat input.dat sed -i "s/${istep}/$(($istep+1))/g" ex7.qsub else exit fi qsub ex7.qsub