Shell Programming

Erwin Earley ([email protected]) Sr. Professional Services Consultant @erwinephp @RougeWaveInc @Zend

• What is a Shell?

• Working with the shell

• Programming Constructs

© 2018 Rogue Wave Software, Inc. All Rights Reserved. 2 What is a Shell? • The command line used on “Unix™” systems (as well as Unix-like systems) • Like CL it can be used interactively, or run as a program • Like CL most commands are actually programs that get called – There are some “built in” commands • Unlike CL there are a number of varieties of shell – sh bourne shell – csh c shell – ksh korn shell – bash bourne again shell – qsh Q shell • There are some similarities and some differences • Most of the discussion here is not operating system specific – Will work on AIX, Linux, QSH in OS/400, other nasty Unix variants, etc

• All system configuration operations can be done through the shell – often more quickly then through a GUI

• It may be the only environment available in the case of a system crash

• Shell scripts can automate most routine tasks such as backups, scheduled emails, etc.

• GUI can be used for a great amount of admin activities – However, the shell tends to be a comfort zone providing ability to fix things in case something goes wrong

• A number of shells are available each providing function/usability customized to a particular type of user:

• Popular shells include: – BASH (Bourne Again Shell) – PDKsh (Public Domain Korn Shell) – csh (C shell) – mc (Midnight Commander) – QSHELL (PASE shell) – ksh (Korn shell, default on AIX)

• Difference tends to be in scripting capabilities and user interface – Items such as command recall and file name completion are typically different

• Login Shell: Shell executed when user logs in to the system. – Typically performs general setup – initializes the terminal, sets some variables, etc. • Non-login shell: Either subshells (started from the login shell), shells started by the GUI desktop, or disconnected shells started by a command – Remember that shell is simply another command on the system • Login shells unlike Non-login shells read a series of configuration files:

On login On startup if "/etc/profile”" source it if "~/.bashrc" exists, source it if "~/.bash_profile" exists, source it else if "~/.bash_login" exists, source it if "~/.profile" exists, source it

On exit if "~/.bash_logout" exists, source it

• bash stands for Bourne Again Shell Available in PASE, more – Developed by Brian Fox in 1987 on that coming up! – One of the most popular shells available in Linux

• Bash incorporates features of the Korn and C shell (ksh and csh)

• Bash configuration files: /bin/bash – Bash executable /etc/profile – System wide initialization file for login shells ~/.bash_profile – Personal initialization file for login shells ~/.bashrc – Personal per-interactive-shell startup file ~/.bash_logout – Login shell clean file that executes when shell exits

© 2018 Rogue Wave Software, Inc. All Rights Reserved. 7 A little more on BASH • Default Linux shell – This can be changed in a variety of ways • /etc/profile – login shell • $HOME/.profile • Magic string within the script (more on this in a moment)

• Very powerful as a command line shell – Recall previous commands – Command and file completion with the key

• Many programming features – Loops and conditionals

• The shell is an environment where commands can be entered and the Operations system can respond to them

• A key concept to the environment is environment variables – There are a large number of environment variables – HISTFILE: points to file containing the shell history, defaults to ~/.bash_history – HISTFILESIZE: how man last commands you wish to have in history – HOME: points to your home directory – PATH: set of directories to search when trying to execute a command – PS1: Prompt variable NOTE: All of these environment variables – USER: username are available when running bash in the PASE environment.

Symbol Meaning > Output redirection >> Output redirection (append) < Input redirection * File substitution wildcard; zero or more characters ? File substitution wildcard; one character [ ] File substitution wildcard; any character between brackets `cmd` Command substitution $(cmd) Command substitution | The pipe (connect output of command on right to input on command on left) ; Command Sequence || OR conditional execution && AND conditional execution

Symbol Meaning ( ) Group commands & Run command in the background # Comment $ Expand the value of a variable \ Prevent or escape interpretation of the next character << Input redirection " $val " Literal with variable substitution ' $val ' Literal without variable substitution

• Arrow Up & Down: Scroll through recent commands used

• &&: command is only executed if preceding command was successful: command1 && command2

• alias: sets a command alias or prints defined aliases alias wrklnk=ls

• bg[jobid]: Resumes the suspended job in the background

• cd: changes current directory to directory indicated cd /home

• echo: Outputs the arguments echo “hello world”

• find [path][expression]: searches the directory indicating looking for files that match expression: find / -name passwd –print

• pwd – Prints the absolute pathname of the current working directory

• unalias – Removes an alias

• history – displays command history with line numbers

• umask – User file creation mask

• logout: exits the shell environment

• exit [n]: exits shell environment with exit status n

• Just enter the command

– $ man bash

• Manual page for bash is about 5,000 lines!!

NOTE: the manual pages are not yet available in the PASE environment.

• The bash shell is available for PASE – It is part of the RPM pile • Step 1: Install the RPM pile bootstrap • Step 2: Install bash yum install bash • Step 3: Create a home directory: PATH=/QOpenSys/pkgs/bin:$PATH mkdir /home/ export PATH • Step 4: Create a .profile • Step 5: Install nano bash yum install nano

• A version of a shell for System i • Has many of the same features as bash • RTFM

This is an excellent and comprehensive book!

#!/bin/bash NOTE: on IBM i the path to bash is echo Hello World /QOpenSys/pkgs/bin/bash However, symbolic links have been created to /bin/bash and /usr/bin/bash This means that many existing scripts can be brought over from Linux environments and run unchanged in PASE under the bash shell

• This is called a shell script • The first line "#!/bin/bash" is a magic string • When a file is loaded if the first two characters are “#!” (called shebang) then the remainder of the line is used as the program to run the file • "#!/home/erwin/my_nice_program" is a valid shell script • echo is a built-in command that just displays the rest of the line

• Use a text editor –Traditional Unix editors include vi and emacs –More user friendly editors include “nano” • On IBM i –Install and use nano from a PASE session

earley@testbed:~$ ls –l test.sh -rw-r—r– 1 earley earley 32 2006-09-11 08:54 test.sh earley@testbed:~$ test.sh -bash: test.sh: command not found earley@testbed:~$

• We have a "test.sh" program in our directory • Lets try and run it • Doesn’t work! Why not!?!?

earley@testbed:~$ echo $PATH /home/earley/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/ usr/bin:/sbin:/bin:/usr/bin/X11 earley@testbed:~$ . test.sh Hello World earley@testbed:~$

• The program is not in the “path” – The current directory is not in the path • Can run a shell script by typing “.” – Means load the file into the current shell

earley@testbed:~$ mv test.sh ~/bin/ earley@testbed:~$ echo $PATH /home/earley/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/ usr/bin:/sbin:/bin:/usr/bin/X11 earley@testbed:~$ test.sh -bash: /home/earley/bin/test.sh: Permission denied earley@testbed:~$

• Move the shell script into a directory in the path • Try and execute it again • Now we get “Permission denied”! Why!?!?

earley@testbed:~$ ls –l ~/bin/test.sh -rw-r--r-- 1 earley earley 32 2006-09-11 08:54 /home/earley/bin/test.sh earley@testbed:~$ chmod u+x ~/bin/test.sh earley@testbed:~$ ls –l ~/bin/test.sh -rwx--r-- 1 earley earley 32 2006-09-11 08:54 /home/earley/bin/test.sh earley@testbed:~$ test.sh Hello World earley@testbed:~$

• The file is not marked with "executable" permissions • "chmod u+x …" says "make it executable by the user who owns it" • NOW it works

Can also get into qsh via 'STRQSH'

Keep in mind that you could also simply ssh into your IBM i system (assuming the ssh daemon is running) and run your scripts

• If you get Command not found the path to the shell is probably wrong • Another error occurs if you move a file from a Windows machine to Linux and the line endings get messed up – If this happens, you can use the following command: tr –d ‘\r’ < erwin.sh > erwin2.sh we will understand that line more later

• *nix programs start with three open files – Input (called stdin) (#0) STDIN (0) STDOUT (1) program – Output (called stdout) (#1) STDERR (2) – Error output (called stderr) (#2) • Our shell program uses the "echo" command to write to stdout • We can redirect the output to go to a file by using ">"

ls –l > output.txt

This will take the output of the "ls –l" command and write it into a file called "output.txt"

• You can also specify which output goes to a file

ls –l 1> output.txt this is the same as before.

• Redirecting only error output

grep fred * 2> grep.err

this will redirect only the error output to the file grep.err

• Use the "<" operator to redirect input – Equivalent to typing at the keyboard

• For example

sed "s/erwin/Erwin/g" < my.txt

This runs the command "sed" (an editor) changing "erwin" to "Erwin" and taking its input from a file called "my.txt"

• Input and output

sed "s/erwin/Erwin/g" < my.txt > my2.txt

• Unix is built around the idea of lots of little programs that all do one thing well – lslists files – sed an editor – grep a searcher – cat a file outputer All of these are available in the IFS and can be run from an IBM – find a file locator i shell (i.e., qsh, ssh session) – sort sorts files – tr translates characters – ps list processes – seq print a sequence of numbers – … a thousand others Currently 230 available in PASE

• Help for all these is in the “man” (for “manual”) command Not currently available in – e.g. man sed PASE • Shell programs generally involve stringing lots of these together – Which leads us to “pipes”

• Standard output (STDOUT) of one command/program is used as the standard input (STDIN) for the next command/program

STDOUT (1) STDIN (0)

ps –x | grep java • List processes, search for any involving java • Only the 'STDOUT' from the last command is actually output to the screen (unless re-directed) – NOTE: Any output to STDERR by the commands will be output to the screen – again unless re-directed.

NOTE: pipes are not limited to two commands… any number of pipes can be used to build a pipeline: cmd1 | cmd2 | cmd3 | …. | cmdx

• Bash (both interactively and in programs) has variables • Variables do not need to be declared, and don’t have types • Variable assignment occurs via a reference to the variable followed by an equal sign and the value to assign. NOTE: There is no spacing: STR="Hello World!" • Variable reference occurs by preceding the variable with an dollar sign: echo $STR

#!/bin/bash STR="Hello World!" echo $STR

Variable Description/Usage $1 The first argument/parameter to a program/command, $2 is the second argument, etc $* All of the arguments to a program/command represented as a single string. $@ All of the arguments to a program/command represented as a set of strings. $# The number of arguments/parameters that the program/command was called with. Useful for testing that the program/command was called correctly as well as looping through the arguments/parameters $? The exit status of the last program/command. $$ The process id of the current process.

• Command substitution uses the output of a command to replace the command itself. – The shell performs the expansion by executing the command and replacing the command substitution with the standard output of the command. • $(command) can be used like a variable, but executes the command – FILES=$(ls) – Sets the variable $FILES to the output of the "ls" command

• The 'let' built-in command is used to evaluate arithmetic expressions: • Consider the following

COUNTER=0 COUNTER=$COUNTER+1 echo $COUNTER # This prints “0+1”

COUNTER=0 let COUNTER=COUNTER+1 echo $COUNTER # This prints 1

NOTE: more then just simple addition can be performed with the 'let' command.

• Another choice for doing math is $(( expression ))

counter=1 counter=$((counter+1)) echo $counter //output will be 2

#!/bin/bash Shell script gets a echo –n "Enter Your Name: " name from user input read NAME and echoes it to the echo "Your name is $NAME" screen exit 0

NOTE: The '-n' option to the echo command indicates that the should be surprised – this results in the prompt for input to be immediately after the echoed output

• Provides a menu based on a word list • Enables the user to enter a number instead of a word – The user enters the sequence number corresponding to the word

#!/bin/bash 1) Steak select var in steak chicken pork 2) Chicken do 3) Pork echo "I like eating $var" #? 2 done I like eating chicken

NOTE: The prompt for the user input can be controlled with the PS3 environment variable: PS3="select entry: " Will replace "#?" With "select entry: "

• Standard “if/then/else” • Note the weird syntax involving [ … ] and ; – The [ ] indicates a test • In fact there is a 'test' command that can be used in place of [ ] – Space following the [ and preceding the ] are required – The ';' is used to separate two commands • In this case it allows us to include the 'then' on the same line as the 'if'

#!/bin/bash if [ "foo" = "foo" ] ; then echo expression evaluated as true else echo expression evaluated as false fi

Not a typo

• [ ] is a synonym for the "test" command – "man test" gives the options • [ -z "$x" ] the length of the string is zero • [ STRING1 = STRING2 ] the strings are equal • [ STRING1 != STRING2 ] the strings are not equal • [ INTEGER1 -eq INTEGER2 ] INTEGER1 is equal to INTEGER2 • [ INTEGER1 -ge INTEGER2 ] INTEGER1 is greater than or equal to INTEGER2 • [ INTEGER1 -gt INTEGER2 ] INTEGER1 is greater than INTEGER2 • [ INTEGER1 -le INTEGER2 ] INTEGER1 is less than or equal to INTEGER2 • [ INTEGER1 -lt INTEGER2 ] INTEGER1 is less than INTEGER2 • [ INTEGER1 -ne INTEGER2 ] INTEGER1 is not equal to INTEGER2

• [ -d FILE ] FILE exists and is a directory • [ -e FILE ] FILE exists • [ -f FILE ] FILE exists and is a regular file

• The previous example makes more sense with a variable:

#!/bin/bash if [ "$myvar" = "foo" ]; then echo expression evaluated as true else echo expression evaluated as false fi

#!/bin/bash #Name: Copy File echo "Enter a Filename to copy:" read SRC echo "Enter the filename to copy to:" Simple file copy read DEST if cp $SRC $DEST 2>/dev/null script with a then echo "Task Complete" if/then/else loop else echo "Copy Failed!" > &2 exit 1 fi

exit 0

• for: very different from other programming languages. Loops over a series of 'words' within a string. • while: loops while a condition is true • until: almost identical to the while loop, except that the code is executed while the control expression evaluates to false. • case: Alternative to if/then/else statements

#!/bin/bash for i in $( ls ); do echo item: $i done

#!/bin/bash for i in `seq 1 10`; do echo $i done

Note: The use of accent marks in the above example is not a typo. The shell will execute the command and replace it with the output. The statement could have been represented as: for i in "1 2 3 4 5 6 7 8 9 10" ; do

#!/bin/bash These two scripts will NOT COUNTER=0 output the same sequence while [ $COUNTER -lt 10 ]; do of numbers – what's the echo The counter is $COUNTER difference? let COUNTER=$COUNTER+1 done

The case statement can be used to replace nested if/then/else statements case EXRESSIOON in pattern1) statements ;; pattern2) statements ;; ... esac Not a typo

• NOTES: – EXPRESSION is expanded and checks are made against each pattern (in order) – When a match is found all associated statements are executed (execution terminates with ;;) – After first match, case terminates – exit status is status of last command executed – If no match, exit status of case is zero

#!/bin/bash echo -n "Do you agree with this? [yes or no]: " read yno case $yno in [yY] | [yY][Ee][Ss] ) echo "Agreed" ;; [nN] | [n|N][O|o] ) echo "Not agreed, you can't proceed with the installation"; exit 1 ;; *) echo "Invalid input" ;; esac

• A shell function is similar to a shell script in that it: – Stores a series of commands for later execution – Functions are stored in memory – The shell executes a shell function in the same shell that called it • Functions can be defined: – In a user's .profile – In a script – On the command line • The built-in function 'unset' can be used to remove a function • Functions must be defined before they can be referenced – Functions are usually placed at the begging of the script

#!/bin/bash function quit { exit } function hello { echo Hello! } hello quit echo foo Never gets executed, why?

Arguments use the same $1, $2 syntax as the main shell script

#!/bin/bash function quit { exit } function mysay { Hello echo $1 Output } World mysay Hello mysay World quit echo foo

• Can redirect input from inside a shell script use the "<<" operator • Treats any text until the TAG as input to the command

#!/bin/bash mail –s "Test Mail" $1 << END_OF_MAIL Hello $1,

This is some spam. Thanks for reading END_OF_MAIL

• Shell scripts return a status value

exit 0 … exit -1 • You can also test the status of a command you call using the special $? variable or by using the command in an if statement directly #!/bin/bash cp $1 /tmp/backup/ if [ $? == 0 ] then echo "Copy of $1 succeeded" else echo "Copy of $1 failed" fi

#!/bin/bash if cp $1 /tmp/backup/ then echo "Copy of $1 succeeded" else echo "Copy of $1 failed" fi

#!/bin/bash if [ -z "$1" ]; then echo usage: $0 directory exit fi

SRCD=$1 TGTD="/var/backups/" OF=home-$(date +%Y%m%d).tgz tar -cZf $TGTD$OF $SRCD

• Currently in beta, but…. Check out: – https://www.ibm.com/developerworks/community/wikis/home?lang=en#! /wiki/IBM%20i%20Technology%20Updates/page/RPMs%20- %20Getting%20Started

• qsh –Read the book • bash –man bash –http://bashprogramming.com • Other useful links –http://www.linuxcommand.org/lc3_writing_she ll_scripts.php