Shells What Shells Do
• Unix shells have the same basic mode of operation: • Shells are command interpreters loop • they allow interactive users to execute the commands. if (interactive) print a prompt • typically a command causes another program to be run read a line of user input • shells may have a graphical (point-and-click) interface apply transformations to line • e.g. Windows or Mac desktop split line into words using whitespace • much easier for naive users use first word in line as command name • much less powerful & not covered in this course execute command, passing other words as arguments • command-line shells are programmable, powerful tools for expert users end loop • bash is the most popular used shell for unix-like systems • shells can also be run with commands in a file • other significant unix-like shells include : dash, zsh, busybox • shells are programming languages • we will cover the core features provided by all shells • shells have design decisions to suit interactive use • essentially the POSIX standard shell features • e.g. variables don’t have to be initialized or declared • these decisions not ideal for programming in Shell • in other words there have to be design compromises 1 2
Processing a Shell Input Line echo - print arguments to stdout
• a series of transformations are applied to Shell input lines • echo prints its arguments to stdout • mainly used in scripts, but also useful when exploring shell behaviour 1. variable expansion, e.g. $HOME → /home/z1234567 • echo is often builtin to shells for efficiency, but also provided by /bin/echo 2. command expansion e.g. $(whoami) → z1234567 • see also /usr/bin/printf - not POSIX but widely available 3. arithmetic, e.g. $((6 * 7)) → 42 • Two useful echo options:
4. word splitting - line is broken up on white-space unless inside quotes -n do not output a trailing newline 5. pathname globbing, e.g. *.c → main.c i.c -e enable interpretation of backslash escapes
6. I/O redirection e.g. 3 4 echo in Python: echo in C: int main(int argc, char *argv[]) { for (inti=1; i < argc; i++) { import sys if (i >1){ def main(): fputc('', stdout); """ } print arguments to stdout fputs(argv[i], stdout); """ } print(''.join(sys.argv[1:])) fputc('\n', stdout); source code for echo.py return 0; } source code for echo.c 5 6 Shell Variables $(command) - command expansion: • shell variables are untyped - consider them as strings • $(command) is evaluated by running command • note that 1 is equivalent to “1” • stdout is captured from command • shell variables are not declared • $(command) is replaced with the entire captured stdout • shell variables do not need initialization • ‘command‘ (backticks) is equivalent to $(command) • initial value is the empty string • backticks is original syntax, so widely used • one scope - no local variables • nesting of backticks is problematic • except sub-shells & functions (sort-of) For example: • changes to variables in sub-shells have no effect outside sub-shell • components of pipeline executed in sub-shell $ now=$(date) $ echo $now • $name replaced with value of variable name Sun 23 Jun 1912 02:31:00 GMT • name=value assigns value to variable name $ • note: no spaces around = 7 8 '' - Single Quotes "" - Double Quotes • double quotes "" group the characters within into a single word • variables and commands are expanded inside double-quotes • single quotes '' group the characters within into a single word • backslash can be used to escape $ “ "" ” \ • no characters interpreted specially inside single quotes • other characters not interpreted specially inside double quotes • a single quote can not occur within single quotes • you can put a single quote between double-quotes • you can put a double quote between single-quotes For example: For example: $ answer=42 $ echo '*** !@#$%^&*(){}[]:;"<>?,./` ***' $ echo "The answer is $answer." *** !@#$%^&*(){}[]:;"<>?,./` *** The answer is 42. $ echo 'this is "normal"' $ echo 'The answer is $answer.' this is "normal" The answer is $answer. $ echo "time's up" time's up 9 10 << - here documents Arithmetic • $((expression)) is evaluated as an arithmetic expression • < 11 12 word splitting *?[]! - pathname globbing • coders not understanding how shells split words is a frequent source of bugs • *?[]! characters cause a word to be matched against pathnames • confusingly similar to regexes - but much less powerful # inspect how shell splits lines into program arguments (argv) • * matches 0 or more of any character - equivalent to regex .* import sys • ? matches any one characters - equivalent to regex . print(f'sys.argv ={sys.argv} ') • [characters] matches 1 of characters - same as regex [] source code for print_argv.py • [!characters] matches 1 character not in characters - same as regex [ˆ] • if no pathname matches the word is unchanged $ v='' • aside: globbing also available in Python, Perl, C & other languages $ ./print_argv.py $v $ echo *.[ch] sys.argv = ['./print_argv.py'] functions.c functions.h i.h main.c $ ./print_argv.py"$v" $ ./print_argv.py *.[ch] sys.argv = ['./print_argv.py', ''] ['./print_argv.py', 'functions.c', 'functions.h', 'i.h', 'main.c'] $ w=' xx yyy zzzz ' $ ./print_argv.py '*.[ch]' $ ./print_argv.py $w ['./print_argv.py', '*.[ch]'] $ ./print_argv.py "*.[ch]" sys.argv = ['./print_argv.py', 'xx', 'yyy', 'zzzz'] ['./print_argv.py', '*.[ch]'] $ ./print_argv.py"$w" $ ./print_argv.py *.zzzzz sys.argv = ['./print_argv.py', ' xx yyy zzzz '] ['./print_argv.py', '*.zzzzz'] 13 14 I/O Redirection Pipelines • stdin, stdout & stderr for a command can be directed to/from files • command1 | command2 | command3 | ... • stdout of commandn-1 connected to stdin of commandn < infile connect stdin to the file infile > outfile send stdout to the file outfile • beware changes to variables in pipeline are lost >> outfile append stdout to the file outfile • some non-filter style Unix programs given a filename - read from stdin 2> outfile send stderr to the file outfile • allows them to be used in a pipeline 2>> outfile append stderr to the file outfile > outfile 2>&1 send stderr+stdout to outfile 1>&2 send stdout to stderr (handy for error messages) • beware: > truncates file before executing command. • always have backups! 15 16 searching PATH for the program danger of having . in your PATH • if . is not last in PATH then programs in the current directory may be unexpectedly run • first word on line specifies command to be run • this can also happen inside run shell scripts or other programs you run • if first word is not the full (absolute) pathname of a file the colon-separated • robust shell scripts often set PATH to ensure this doesn’t happen, e.g.: list of directory specified by the variable PATH is searched PATH=/bin/:/usr/bin/:$PATH • for example if PATH=/bin/:/usr/bin/:/home/z1234567/bin and the command is kitten the shell will check (stat) these files in order: # equivalent to PATH=.:/bin:/usr/bin:/home/z1234567/bin • /bin/kitten /usr/bin/kitten /home/z1234567/bin $ PATH=:/bin:/usr/bin:/home/z1234567/bin • the first that exists and is executable will be run $ cat>cat < 17 Problem: ./cat is being run rather /bin/cat 18 Shell Scripts Shell Built-in Variables We can execute shell commands in a file: Some shell built-in variables with pre-assigned values: $ cat hello $0 the name of the command echo Hello, John Connor - the time is $(date) $1 the first command-line argument $ sh hello $2 the second command-line argument Hello, John Connor - the time is Fri 29 Aug 1997 02:14:00 EST ...... $9 the ninth command-line argument • Unix-like systems allow an interpreter to be specified in a #! line ${10} the tenth command-line argument • allows program to be executed directly without knowing it is shell ...... $ cat hello ${255} the two hundred and fifty-fifth (last) command-line argument #!/bin/sh $# count of command-line arguments echo Hello, John Connor $* all the command-line arguments (separately) echo The time is $(date) "$*" all the command-line arguments (together) $ chmod 755 hello $@ all the command-line arguments (separately) $ ./hello "$@" all the command-line arguments (as quoted) Hello, John Connor - the time is Fri 29 Aug 1997 02:14:00 EST $? exit status of the most recent command $$ process ID of this shell • use #!/bin/bash if you want bash 19 20 • $$ is useful for generating (somewhat) unique names in scripts. • see also the shift command Example - Shell Script using Built-in Variables Example - Simple Shell Script #!/bin/dash #!/bin/sh # A simple shell script demonstrating access to arguments. # l [file|directories...] - list files # written by [email protected] as a COMP(2041|9044) example # echo My name is"$0" # written by [email protected] as a COMP(2041|9044) example echo My process number is $$ # echo I have $# arguments # Short shell scripts can be used for convenience. # your not going to see any difference unless you use these in a loop # echo My arguments separately are $* # It is common to put these scripts in a directory echo My arguments together are"$*" # such as /home/z1234567/scripts echo My arguments separately are $@ # then add this directory to PATH e.g in .bash_login echo My arguments as quoted are"$@" # PATH=$PATH:/home/z1234567/scripts echo My 5th argument is" '$5'" # echo My 10th argument is" '${10}'" # Note: "$@" like $* expands to the arguments to the script, echo My 255th argument is" '${255}'" # but preserves whitespace in arguments. source code for args.sh ls -las "$@" source code for l 21 22 Example - Putting a Pipeline in a Shell Script Debugging Shell Scripts #!/bin/sh # Count the number of time each different word occurs # in the files given as arguments, or stdin if no arguments, # e.g. word_frequency.sh dracula.txt # written by [email protected] as a COMP(2041|9044) example Tip: debugging for shell scripts cat"$@" | # tr doesn't take filenames as arguments • test parts of shell script from command line tr '[:upper:]'' [:lower:]' | # map uppercase to lower case tr ''' \n' | # convert to one word per line • use echo to print the value of variables tr -cd "a-z'" | # remove all characters except a-z and ' • add set -x to see commands being executed grep -E -v '^$' | # remove empty lines • or equivalently run /bin/dash -x script.sh sort | # place words in alphabetical order • shell transforms commands uniq -c | # count how many times each word occurs • useful to see exactly what is being executed sort -rn # order in reverse frequency of occurrence # notes: # - first 2 tr commands could be combined # - sed 's/ /\n/g' could be used instead of tr '''\n' # - sed "s/[^a-z']//g" could be used instead of tr -cd "a-z'" 23 24 source code for word_frequency.sh Exit Status and Control The test command • when Unix-like programs finish they give the operating system an exit status • the return value of ‘main becomes the exit status of a C program • or if exit is called, its argument is the exit status • The test command performs a test or combination of tests and: • in Python exit status is supplied as an argument to sys.exit • does/prints nothing • an exit status is a (usually small) integer • returns a zero exit status if the test succeeds • by convention a zero exit status indicated normal/successful execution • returns a non-zero exit status if the test fails • a non-zero exit status indicates an error occurred • Provides a variety of useful operators: • which non-zero integer might indicate the nature of the problem • string comparison: = != • program exit status is often ignored • numeric comparison: -eq -ne -lt • not important writing single programs (COMP1511/COMP9021 ) • test if file exists/is executable/is readable: -f -x -r • very important when combining multiple programs COMP(2041|9044) • boolean operators (and/or/not): -a -o ! • flow of execution in Shell scripts based on exit status • if/while statement conditions use exit status • also available as ‘[’ instead of test - which many programmers prefer • two weird utilities • builtin to some shell (e.g. bash) but available as /bin/test or /bin/[ • /bin/true does nothing and always exits with status 0 • /bin/false does nothing and always exits with status 1 25 26 The test command examples Using Exit Status for Conditional Execution # does the variable msg have the value "Hello"? • all commands are executed if separated by ; or newline, e.g: test"$msg"= "Hello" cmd1 ; cmd2 ; ... ; cmdn # does x contain a numeric value larger than y? • when commands are separated by && test"$x" -gt"$y" cmd1 && cmd2 && ... && cmdn execution stops if a command has non-zero exit status # Error: expands to "test hello there = Hello"? cmd is executed only if cmd has zero exit status msg="hello there" n+1 n test $msg = Hello • when commands are separated by || cmd1 || cmd2 || ... || cmdn # is the value of x in range 10..20? execution stops if a command haszero exit status test"$x" -ge 10 -a"$x" -le 20 cmdn+1 is executed only if cmdn has non-zero exit status # is the file xyz a readable directory? • {} can be used to group commands test -r xyz -a -d xyz • () also can be used to group commands - but executes them in a subshell # alternative syntax; requires closing ] • changes to variables and current working directory have no effect outside the [ -r xyz -a -d xyz ] subshell 27 • exit status of group or pipeline of commands is exit status of last command28 Conditional Execution Examples {} versus () - example $ cd /usr/share # run a.out if it exists and is executablr $ x=123 test -x a.out && ./a.out $ ( cd /tmp; x=abc; ) $ echo $x # if directory tmp doesn't exist create it 123 test -d tmp || mkdir tmp $ pwd /usr/share # if directory tmp doesn't exist create it $ { cd /tmp; x=abc; } {test -d tmp || mkdir tmp;} && chmod 755 tmp $ echo $x abd # but simpler is $ pwd mkdir -p tmp && chmod 755 tmp /tmp # exit status • changes to variables and current working directory have no effect outside a if cut -d: -f1 /etc/passwd|grep '^admin$' subshell • pipelines also executed in subshell, but variables and directory not usually 29 changed in a pipeline 30 If Statements in Shell If Statements - Example • shell if statements have this form: if gcc main.c then if command1 echo your C compiles then elif python3 main.c then-commands echo you have written Python not C elif command2 else then echo program broken - send help elif-commands fi else else-commands if gcc a.c fi then # you can not have an empty body # use a : statement which does nothing • the execution path depends on the exit status of command1 and command2 : • command1 is executed and if its exit status is 0, else the then-commands are executed rm a.c • otherwise command2 is executed and if its exit status is 0, fi the elif-commands are executed 31 32 • otherwise the else-commands are executed While Statements in Shell example - seq - simple version • shell while statements have this form: #!/bin/sh # simple emulation of /usr/bin/seq for a COMP(2041|9044) example while command # [email protected] do # Print the integers 1..n with no argument checking body-commands last=$1 done number=1 while test $number -le"$last" do • the execution path depends on the exit status of command echo $number • command is executed and if its exit status is 0, number=$((number + 1)) the body-commands are executed done source code for seq.v0.sh and then command is executed and if its exit status is 0 $ ./seq.v0.sh 3 the body-commands are executed 1 and . . . 2 • if the exit status of command~ is not 0, execution of the loop stops 3 33 34 example - seq - argument handling added example - seq - using [] instead of test # Print the integers 1..n or n..m if [ $#=1] if test $#=1 then then first=1 first=1 last=$1 last=$1 elif [ $#=1] elif test $#=1 then then first=$1 first=$1 last=$2 last=$2 else else echo "Usage: $0 if [ $#=1] # Repeatedly download a specified web page then # until a specified regexp matches its source first=1 # then notify the specified email address. last=$1 # elif [ $#=1] # For example: then # watch_website.sh http://ticketek.com.au/ 'Ke[sS$]+ha' [email protected] first=$1 repeat_seconds=300 #check every 5 minutes last=$2 if test $#=3 else then echo "Usage: $0 • shell for statements have this form: for var in word1 word3 ... wordn while true do do body-commands if curl --silent"$url" |grep-E"$regexp">/dev/null done then echo "Generated by $0" | mail -s"$url now matches $regexp""$email_address" • the loop executes once for each word with var set to the word exit 0 • break & continue statements can be in used inside for & while loops fi with the same effect as C/Python sleep $repeat_seconds • keywords such for, if,*while, . . . are only recognised at the start of a done source code for watch_website.sh command, e.g.: $ echo when if else for when if else for 39 40 Example - Shell Script accessing Command-line Arguments example - renaming files - argument checking #! /bin/dash # Change the names of the specified files to lower case. # $ ./accessing_args.sh one two "three four" # (simple version of the perl utility rename) # Using "$*": # # one two three four # Note use of test to check if the new filename is unchanged. # Using $*: # # one # Note the double quotes around $filename so filenames # two # containing spaces are not broken into multiple words # three # Note the use of mv -- to stop mv interpreting a # four # filename beginning with - as an option # Using "$@": # Note files named -n or -e still break the script # one # because echo will treat them as an option, # two if test $#=0 # three four then echo 'Using $*:' echo "Usage $0: # written by [email protected] for COMP(2041|9044) # This means changes in comments won't affect comparisons. # Note use of temporary files # TMP_FILE1=/tmp/plagiarism_tmp1$$ # Run as plagiarism_detection.simple_diff.sh for file1 in"$@" # and change all identifiers to the letter 'v'. do # Hence changes in strings & identifiers will be ignored. for file2 in"$@" TMP_FILE1=/tmp/plagiarism_tmp1$$ do TMP_FILE2=/tmp/plagiarism_tmp2$$ test"$file1"="$file2" && # s/"["]*"/s/g changes strings to the letter 's' break # avoid comparing pairs of assignments twice # It won't match a few C strings which is OK for our purposes sed"$substitutions""$file1">$TMP_FILE1 # s/[a-zA-Z_][a-zA-Z0-9_]*/v/g changes variable names to 'v' sed"$substitutions""$file2">$TMP_FILE2 # It will also change function names, keywords etc. if diff -i -w $TMP_FILE1 $TMP_FILE2>/dev/null # which is OK for our purposes. then substitutions=' echo"$file1 is a copy of $file2" s/\/\/.*// fi s/"[^"]"/s/g done s/[a-zA-Z_][a-zA-Z0-9_]*/v/g' done source code for plagiarism_detection.identifiers.sh rm -f $TMP_FILE1 $TMP_FILE2 source code for plagiarism_detection.identifiers.sh 49 50 plagiarism detection - ignoring changes in code order robust creation & removal of temporary files for file1 in"$@" • our code can be more robust and more secure do by using mktemp to generate temporary file names for file2 in"$@" • we can also use the builtin shell trap command do to ensure temporary files are removed however the script exits test"$file1"="$file2" && break # avoid comparing pairs of assignments twice TMP_FILE1=$(mktemp /tmp/plagiarism_tmp.1.XXXXXXXXXX) sed"$substitutions""$file1" |sort>$TMP_FILE1 TMP_FILE2=$(mktemp /tmp/plagiarism_tmp.2.XXXXXXXXXX) sed"$substitutions""$file2" |sort>$TMP_FILE2 trap 'rm -f $TMP_FILE1 $TMP_FILE2;exit' INT TERM EXIT if diff -i -w $TMP_FILE1 $TMP_FILE2>/dev/null source code for plagiarism_detection.mktemp.sh then • temporary file creation is major source of security holes echo"$file1 is a copy of $file2" be very careful creating temporary files fi • in all languages, use existing robust & well-tested code such as mktemp done • don’t write your own code done rm -f $TMP_FILE1 $TMP_FILE2 • unfortunately mktemp is not standardized by POSIX source code for plagiarism_detection.reordering.sh • simple uses are portable to many platforms 51 52 Example - creating a temporary directory Cryptographic hash function # securely & robustly create a new temporary directory • algorithm maps byte sequence of any length to certain number of bits temporary_directory=$(mktemp -d /tmp/dir.XXXXXXXXXX) # ensure temporary directory + all its contents removed on exit • e.g sha256 input: any number of bytes, output 256 bits (= 8 bytes) hash trap 'rm -rf "$temporary_directory; exit"' INT TERM EXIT • one way function - not feasible to reverse # change working directory to the new temporary directory • given a hash, not feasible to compute an input which produces that hash cd"$temporary_directory" || exit 1 # we are now in an empty directory • collisions (different inputs producing the same hash) occur but are # and create any number of files & directories vanishingly rare # which all will be removed by the trap above • small change to input changes hash completely # e.g. create one thousand empty files seq 1 1000|xargs touch • many applications: # print current directory and list files • hashes of passwords stored rather than password itself pwd • integrity check on set of files ls -l • fingerprint a file source code for create_temporary_directory.sh 53 54 plagiarism detection - using hashing case statements in Shell # Improved version of plagiarism_detection.reordering.sh • shell case statements have this form: # Note use sha256sum to calculate a Cryptographic hash of the modified file # https://en.wikipedia.org/wiki/SHA-2 case word in # and use of sort && uniq to find files with the same hash pattern1) # This allows execution time linear in the number of files commands # We could use a faster less secure hashing fucntion instead of sha2 1 substitutions=' ;; s/\/\/.*// pattern2) s/"[^"]"/s/g commands2 s/[a-zA-Z_][a-zA-Z0-9_]*/v/g' ;; for file in"$@" patternn) do commands~n sha2hash=$(sed"$substitutions""$file" |sort|sha256sum) echo"$sha2hash $file" esac done| sort| • word is compared to each pattern in turn. uniq -w32 -d --all-repeated=separate| i cut -c36- • for the first patterni that matches source code for plagiarism_detection.hash.sh the corresponding commandsi is executed 55 56 and the case statement finishes. case statements in Shell case statement - examples # Checking number of command line args case $# in 0) echo "You forgot to supply the argument" ;; 1) filename=$1 ;; *) echo "You supplied too many arguments" ;; • case patterns use the same language as filename expansion (globbing) esac • in other words the special characters are * ? [] • patterns are not interpreted as regexes # Classifying a file via its name • shell programmer used to use case statements heavily for efficiency case "$file" in • much less important now and many shell programmers don’t use case *.c) echo"$file looks like a C source-code file" ;; • but use of case can still make shell code more readable *.h) echo"$file looks like a C header file" ;; *.o) echo"$file looks like a an object file" ;; ... ?) echo"$file 's name is too short to classify" ;; *) echo "I have no idea what $file is" ;; esac 57 58 read - shell builtin read - simple example • read is a shell builtin which reads a line of input into variables(s) #!/bin/dash • non-zero exit status on EOF # written by [email protected] for COMP(2041|9044) • newline is stripped # demonstrate simple use of read • leading and trailing whitespace stripped unless variable IFS unset • note -r option if input might contains backslashes echo -n "Do you like learning Shell? " • if more than one variable specified, line is split into fields on white space read answer • 1st variable assigned 1st field, 2nd variable assigned 2nd field . . . case "$answer" in • last variable entire remainder of line [Yy]*) • if insufficient fields variables assigned empty strings response=":)" • if more than one variable specified, line is split into fields on white space ;; $ readv [Nn]*) hello world response=":(" $ echo"$v" hello world ;; $ readabc *) 1 2 3 4 5 response="??" $ echo "a='$a' b='$b' c='$c'" esac a='1' b='2' c='3 4 5' echo"$response" source code for read_response.sh 59 60 emulating cat with read shell functions #!/bin/dash • shell functions have this form: # written by [email protected] for COMP(2041|9044) # over-simple /bin/cat emulation using read function_name () { # setting the special variable IFS to the empty trings commands # stops white space being stripped } for file in"$@" do while IFS= read -r line • function arguments passed in: $@ $1 $2 ... do • use return to stop function execution and return exit status echo"$line" • beware: exit in a function still terminates entire program done <$file • local limit scope of variables to function done source code for read_cat.sh • local is not POSIX, but is widely supported 61 62 example - shell function example - local variables in a shell function # print print numbers < 1000 #!/bin/dash # note use of local Shell builtin to scope a variable # written by [email protected] for COMP(2041|9044) # without the local declaration # the variable i in the function would be global # demonstrate simple use of a shell function # and would break the bottom while loop repeat_message() { # local is not (yet) POSIX but is widely supported is_prime() { n=$1 localni message=$2 n=$1 i=2 for i in $(seq1 $n) while test $i -lt $n do do test $((n % i)) -eq 0 && echo"$i: $message" return 1 done i=$((i + 1)) done } return 0 i=0 } i=0 while test $i -lt 4 while test $i -lt 1000 do do is_prime $i && echo $i repeat_message 3 "hello Andrew" i=$((i + 1)) done i=$((i + 1)) source code for local.sh done 63 64 source code for repeat_message.sh example - using a signal to provide a time limit intercepting signals with trap my_process_id=$$ • trap specifies commands to be executed if a signal is received, e.g.: # launch a asynchronous sub-shell that will kill # count slowly and laugh at interrupts (ctrl-C) # this process in a second # catch signal SIGINT and print message (sleep1 ; kill $my_process_id)& trap 'echo ha ha' INT i=0 n=0 while true while true do do echo $i echo"$n" i=$((i + 1)) sleep1 done source code for async.v0.sh n=$((n + 1)) done note: source code for laugh.sh • command & executes command but does not wait for it to finish • trap is useful for cleaning up temporary files before termination, e.g. • sleep 1 suspends execution for a second trap 'rm -f $TMP_FILE;exit' INT TERM EXIT • kill sends a signal to a process, which by default causes it to exit 65 66 example - catching a signal with trap example - compiling in parallel # compile the files of a muti-file C program in parallel # use create_1001_file_C_program.sh to create suitable test data # catch signal SIGTERM, print message and exit # On a CPU with n cores this can be (nearly) n times faster trap 'echo loop executed $n times in 1 second; exit 0' TERM # If there are large number of C files we # launch a sub-shell that will terminate # may exhaust memory or operating system resources # this process in 1 second for f in"$@" my_process_id=$$ do (sleep1 ; kill $my_process_id)& clang -c"$f" & n=0 done while true do # wait for the incremental compiles to finish # and then compile .o files into single binary n=$((n + 1)) done wait source code for async.v1.sh clang *.o -o binary source code for parallel_compile.v0.sh 67 68 example - compiling in parallel example - compiling in parallel $ ./create_1001_file_C_program.sh $ echo *.c file0.c file1.c file10.c file100.c file101.c file102.c ... $ echo *.c|wc -w # compile the files of a muti-file C program in parallel 1001 # use create_1001_file_C_program.sh to create suitable test data # compiling 1 file at a time $ time clang *.c # on Linux getconf will tell us how many cores the machine has real 0m20.875s # otherwise assume 8 user 0m13.016s max_processes=$(getconf _NPROCESSORS_ONLN 2>/dev/null) || sys 0m7.835s # compiling all 1001 files simultaneously max_processes=8 $ time ./parallel_compile.v0.sh *.c # NOTE: this breaks if a filename contains whitespace or quotes real 0m2.335s user 0m9.066s echo"$@" | sys 0m8.788s xargs --max-procs=$max_processes --max-args=1 clang -c # compiling 24 files at time clang *.o -o binary $ time ./parallel_compile.v1.sh *.c real 0m1.971s source code for parallel_compile.v1.sh user 0m18.694s sys 0m18.428s $ grep 'model name' /proc/cpuinfo|sed 1q model name : AMD Ryzen 9 3900X 12-Core Processor 69 70 example - compiling in parallel example - compiling in parallel # compile the files of a multi-file C program in parallel # use create_1001_file_C_program.sh to create suitable test data # find's -print0 option terminates pathnames with a '\0' # xargs's --null option expects '\0' terminated input # as '\0' can not appear in file names this can handle any filename # compile the files of a muti-file C program in parallel # on Linux getconf will tell us how many cores the machine has # use create_1001_file_C_program.sh to create suitable test data # if getconf assume 8 parallel clang -c '{}' :::"$@" max_processes=$(getconf _NPROCESSORS_ONLN 2>/dev/null) || clang *.o -o binary max_processes=8 source code for parallel_compile.v3.sh find"$@" -print0 | xargs --max-procs=$max_processes --max-args=1 --null clang -c clang *.o -o binary source code for parallel_compile.v2.sh 71 72 Shell Variable Expansion - More Syntax Bash arithmetic (()) extension example $ x=1 # print print numbers < 1000 # Rewritten to use bash arithmetic extension (()) $ y=fred # This makes the program more readable but less portable. $ echo $x$y is_prime() { 1fred localni $ echo $xy # the aim is to display "1y" n=$1 i=2 while(( i < n)) $ echo"$x"y do 1y if(( n % i == 0)) then $ echo ${x}y return 1 1y fi $ echo ${j-10} # give value of j or 10 if no value i=$((i + 1)) done 10 return 0 $ echo ${j=33} # set j to 33 if no value (and give $j) } 33 i=0 $ echo ${x:?No Value} # display "No Value" if $x not set while(( i < 1000)) do 1 is_prime $i && echo $i $ echo ${xx:?No Value} # display "No Value" if $xx not set i=$((i + 1)) done -bash: xx: No Value source code for bash_arithmetic.sh 73 74