Intermediate Shell Scripting 11/11/2015 Dr.)Yongjun)Choi) Research)Specialist,) Institute)for)CyberAEnabled)Research) Evaluation Form • Please don’t forget to complete your evaluation form before you leave. How this workshop works • We are going to cover some intermediate topics. – Lots of hands on examples • Exercises are denoted by the following icon in this presents: Green and Red Sticky • Use the sticky notes provided to help me help you. – No Sticky = I am working – Green = I am done and ready to move on (yea!) – Red = I am stuck and need more time and/or some help examples • login • connect to dev-intel10 • module load powertools • getexample shell_scripting Agenda • Review – Hello, ${world} – basic commands – grep/sed/find – Flow control • Advanced shell scripting – ex6 – ex7 Hello, ${world} hellow_world.sh !/bin/bash msg1=world! echo “Hello, ${msg1}” • It does NOT work. Please fix it. Hello, ${world} hellow_world2.sh #!/bin/bash echo Hello, %{1}! • It does NOT work. Please fix it. Hello, ${world} hellow_world3.sh #!/bin/bash echo ‘msg=“Hello World!”’ > hello echo “echo $msg” >> hello chmod 000 hello ./hello Command substitution • $(command) • `command` dir.sh #!/bin/bash #!/bin/bash dir=`ls -l` dir=`ls -l` echo $dir echo “$dir” Arithmetic Expansion • $((expression)) #!/bin/bash echo $((1+3+4)) #!/bin/bash #!/bin/bash echo $((ls)) echo $( (ls) ) Redirection and pipelines • use > to redirect stdout to a file: ls > file • Use < to specify that we: grep searchterm < file • Use >> to append stdout to a file – date > date.out – cal >> date.out • Use 2> to redirect standard error to a file – commandA 2>1; commandA 2>&1 • Use pipelines to chain commands – ls -l |grep searchword | sort -r grep • grep pattern file • grep pattern < file • grep -i pattern file • grep -v pattern file • eg grep HPCC sample.txt grep -i hpcc sample.txt grep -v hpcc sample.txt sed • sed (short for “steam editor”) is used to filter and transform text. • eg echo day |sed s/day/night echo My night is more beautiful than your day |sed s/day/ night/ Change HPCC to hpcc in sample.txt sed -i s/HPCC/hpcc/g sample.txt if…then…elif…else…fi • The generic form if list then list [elif list then list]… [else list] fi • The parts enclosed within ([) and (]) are optional. if…then…elif…else…fi dev-intel10:warmingUp$ sh if_elif.sh 1 The first choice is nice dev-intel10:warmingUp$ sh if_elif.sh 2 The second choice is just as nice dev-intel10:warmingUp$ sh if_elif.sh 3 The third choice is excellent dev-intel10:warmingUp$ sh if_elif.sh 4 I see you were wise enough not to choose You win dev-intel10:warmingUp$ do…while • The generic form While list do list done do…while dev-intel10:warmingUp$ sh do_while.sh 5 5 seconds till snack time! 4 seconds till snack time! 3 seconds till snack time! 2 seconds till snack time! 1 seconds till snack time! snack time!!, YEAH!! dev-intel10:warmingUp$ for • The generic form For variable in word … do list done for fruitlist="Apple Pear Tomato Peach Grape" dev-intel10:warmingUp$ sh for.sh I like Apples I like Pears I like Tomatoes I like Peaches I like Grapes dev-intel10:warmingUp$ case • The generic form case word in pattern list ;; … esac case dev-intel10:warmingUp$ sh case.sh 1 First Choice dev-intel10:warmingUp$ sh case.sh 2 Second Choice dev-intel10:warmingUp$ sh case.sh 3 Other Choice dev-intel10:warmingUp$ sh case.sh 1234 Other Choice dev-intel10:warmingUp$ Functions • The generic form name( ) { commands } Functions dev-intel10:warmingUp$ sh function1.sh 1 2 10 1 3 5 7 9 dev-intel10:warmingUp$ Expected results dev-intel10:warmingUp$ sh function1.sh 1 2 10 1 3 5 7 9 dev-intel10:warmingUp$ Intermediate shell scripting • Preparation – getexample shell_scripting_examples – cd shell_script_examples/ex6 • We will develop a shell script which manipulates files & directories, and submits a queue script. ex6 directory structure ex6 common.sh ex6* origin ex6.cpp input.dat output.dat unWantedFile.tar.gz Anatomy of ex6.cpp ex6.cpp #include <fstream> #include <iostream> input.dat using namespace std; char inputString[1000]; 0. double param[3]; 3. int main (int argc, char **argv) { 7. ifstream inFile(argv[1]); int ncount = 0; //counter while(!inFile.eof()) { //read parameters inFile>>param[ncount]; ncount++; output.dat } inFile.close(); 1. 4. ofstream outFile("output.dat"); //write parameters 8. for (int icount = 0; icount < ncount-1; icount++) outFile<<“ “<<param[icount]+1.0<<std::endl; return 0; } Our goal is… • Write a shell script which: – imports a shell script which builds ex6 executable at origin. – creates six subdirectories (0, 1, 2, 3, 4, & 5) under data directory (data/0, data/1,…, data/5) – copies all files except unWantedFile.tar.gz from origin to each subdirectory – manipulates the input parameters in input.dat on each subdirectory (0, 1, 2, 3, 4, & 5) – submits ex6.qsub at each directory and redirect all output including error message to standard output – clears out unnecessary files such as ex6 at each directory Our goal ex6 origin data common.sh ex6 common.sh ex6.cpp ex6 0 1 2 3 4 5 input.dat ex6.cpp output.dat input.dat unWantedFile.tar.gz Base script template #!/bin/sh DDIRS=“0 1 2 3 4 5" #directory names in which data will be saved. SIMDIRS='data' #parent directory of data directories FROM_DIR='./origin/' #location of original files • We will use this template. • Please create this shell script and save it as ex6.sh under ex6 directory. We will expand ex6.sh for the rest of the workshop Our goal is… • Write a shell script which: – imports a shell script which builds ex6 executable at origin. – creates six subdirectories (0, 1, 2, 3, 4, & 5) under data directory (data/0, data/1,…, data/5) – copies all files except unWantedFile.tar.gz from origin to each subdirectory – manipulates the input parameters in input.dat on each subdirectory (0, 1, 2, 3, 4, & 5) – submits ex6.qsub at each directory and redirect all output including error message to standard output – clears out unnecessary files such as ex6 at each directory Step 1: build executable • Build ex6 executable at origin. – You can use other shell scripts in a script to import functions/ commands – Use ./origin/common.sh in ex6.sh to build ex6 executable common.sh g++ -O3 -o ex6 ex6.cpp Sample script #!/bin/sh DDIRS=“0 1 2 3 4 5" #directory names in which data will be saved. SIMDIRS='data' #parent directory of data directories DDIRS FROM_DIR='./origin/' #location of original files cd $FROM_DIR ./common.sh cd .. Our goal is… • Write a shell script which: – imports a shell script which builds ex6 executable at origin. – creates six subdirectories (0, 1, 2, 3, 4, & 5) under data directory (data/0, data/1,…, data/5) – copies all files except unWantedFile.tar.gz from origin to each subdirectory – manipulates the input parameters in input.dat on each subdirectory (0, 1, 2, 3, 4, & 5) – submits ex6.qsub at each directory and redirect all output including error message to standard output – clears out unnecessary files such as ex6 at each directory Step 2: Create subdirectories • Create six subdirectories (0, 1, 2, 3, 4, & 5) under data directory (data/0, data/1,…, data/5). – Do not do this manually. Please expand ex6.sh data 0 1 2 3 4 5 Sample script #!/bin/sh DDIRS=“0 1 2 3 4 5" #directory names in which data will be saved. SIMDIRS='data' #parent directory of data directories FROM_DIR='./origin/' #location of original files for dd in $DDIRS; do SIM_DIR=$SIMDIRS/$dd mkdir -p $SIM_DIR done Our goal is… • Write a shell script which: – imports a shell script which builds ex6 executable at origin. – creates six subdirectories (0, 1, 2, 3, 4, & 5) under data directory (data/0, data/1,…, data/5) – copies all files except unWantedFile.tar.gz from origin to each subdirectory – manipulates the input parameters in input.dat on each subdirectory (0, 1, 2, 3, 4, & 5) – submits ex6.qsub at each directory and redirect all output including error message to standard output – clears out unnecessary files such as ex6 at each directory Step 3: Copy files to subdirectories copy all files except unWantedFile.tar.gz from origin to each subdirectory origin data common.sh ex6 0 1 2 3 4 5 ex6.cpp input.dat unWantedFile.tar.gz common.sh ex6 ex6.cpp input.dat Sample script #!/bin/sh DDIRS="0 1 2 3 4 5" #directory names in which data will be saved. SIMDIRS='data' #parent directory of data directories FROM_DIR='./origin/' #location of original files IFILES=`ls $FROM_DIR` for dd in $DDIRS; do SIM_DIR=$SIMDIRS/$dd mkdir -p $SIM_DIR for inif in $IFILES; do if [ $inif != "unWantedFile.tar.gz" ]; then cp $FROM_DIR/$inif $SIM_DIR/ fi done done Our goal is… • Write a shell script which: – imports a shell script which builds ex6 executable at origin. – creates six subdirectories (0, 1, 2, 3, 4, & 5) under data directory (data/0, data/1,…, data/5) – copies all files except unWantedFile.tar.gz from origin to each subdirectory – manipulates the input parameters in input.dat on each subdirectory (0, 1, 2, 3, 4, & 5) – submits ex6.qsub at each directory and redirect all output including error message to standard output – clears out unnecessary files such as ex6 at each directory Step 4: manipulate input.dat • Replace input parameters – the first input parameter (0) in input.dat to each name of subdirectory (0, 1, 2, 3, 4, 5) data/ origin/input.dat 0/input.dat … 3/input.dat … 5/input.dat 0. 0. 3. 5. 3.