Shell Programming and Unix Utilities

● Creating Shell Scripts

● Variables, Arguments, and Expressions

● Shell I/O and Debugging

● Testing Conditions and Control Statements

– test, if, case, while, until, for, break, continue

● Exit Status

● Command Substitution

● Regular Expressions and Utilities

– grep, wc, touch, awk, tr, sort, gtbl, groff, ghostview, cut, head, tail, sed, gnuplot Shell Scripts (35.2)

● Advantages of schell scripts.

– Can very quickly setup a sequence of commands to avoid a repetitive task.

– Can easily make several programs work together to meet a set of goals.

● Disadvantage of shell scripts.

– Little support for programming in the large.

– Shell scripts are much slower since they are interpreted. What Shell to Use? (27.3)

● The csh (C) shell and its derivative (tcsh) are recommended for use at the command line.

● The sh (Bourne) shell and its derivatives (ksh, bash) are recommended for writing shell scripts.

– All shell script examples in this course are given using the Bourne shell syntax. Finding Information about the Bourne Shell and Its Commands

● Type “man ” or “info ” where is the Bourne shell command of interest.

● Type “man sh” and look in the manual page.

● Look up information in the text or use other texts.

● See me or the TA. Creating a Shell Script (35.1, 36.2)

● General convention is to use a .sh extension (.sh) for shell scripts.

● Type in the first line of each shell script file: #!/bin/sh

● Type in comments after this to indicate the purpose of the shell script. # This shell script is used to create a new # schema. ● Change the permissions on the schell script so it can be executed by typing the name of the file. chmod +x .sh Printing a Line to Standard Output

● Use the echo command to print a line to stdout. form: echo exs: echo “Hello World!” echo “Please enter a schema name:” Implementation of Echo Shell Variables (35.9)

● Are just used and not declared.

● General convention is to use lowercase letters for shell variable names and uppercase letters for environment variable names.

● Shell variables are assigned strings, which can be interpreted as numeric values. Note that there can be no blanks before or after the '='. form: = exs: var=0 var=”Good” Shell Variables (cont.)

● Shell variables are initialized to be the empty string by default.

● Shell variables can be dereferenced. form: $ exs: var1=$var2 echo “====” $testcase “====” echo “deleted” $num “records from” $schema Reading Values into a Shell Variable (35.18)

● Use the read statement to read a line of standard input, split the line into fields of one or more strings, and assign these strings to shell variables. All leftover strings in the line are assigned to the last variable. form: read ... exs: read num read name field1 field2 read name1 name2 < input.txt Shell Arguments (35.20)

● Can pass arguments on the command line to a shell script, just like you can pass command line arguments to a program. nd.sh employee Intel

● $1, $2, ..., $9 can be used to refer to up to nine command line arguments (argv[1] ... argv[9]).

● $0 contains the name of the script (argv[0]). Example Using Shell Arguments

● Example shell script called prompt.sh: #!/bin/sh echo “Please enter fields for the” $1 “schema.”

● Example usage: prompt.sh Intel

● Example output: Please enter fields for the Intel schema. More on Cmd Line Args (35.20)

● $# contains the number of command line arguments.

● $@ will be replaced with a string containing the command line arguments.

● Example shell script called echo.sh. #!/bin/sh echo "The" $# "arguments entered were:" $@

● Example usage: echo.sh cat dog horse

● Example output: The 3 arguments entered were: cat dog horse Debugging Shell Scripts (35.25, 37.1)

● Shell scripts are interpreted by the shell.

– Syntax errors are not detected until that line is interpreted at run time.

– There is no debugger.

● You can invoke your shell script with command line flags to assist in debugging.

– The -v flag indicates the verbose option and the shell prints the commands as they are read.

– The -x flag causes the shell to print the interpolated commands after they are read. Debugging Shell Scripts (cont.)

● Invoke your shell script as follows so that each line of the script is echoed as it is interpreted. form: sh -xv ex usage: sh -xv echo.sh a b c ex output: #!/bin/sh echo "The" $# "arguments entered were:" $@ + echo The 3 arguments entered were: a b c The 3 arguments entered were: a b c ● You can also set these options in the shell script itself.

– The “set -xv” will turn on these options.

– The “set +xv” will turn off these options. Testing Conditions (35.26)

● Can test for various conditions. There are two general forms. test or [ ]

● I think the latter is easier to read. Be sure to include a space before and after the brackets.

● To test the result of a command, just use the command, which will return an exit status.

● Can reverse a condition by putting '!' before it. [ ! ]

● A ':' command always returns true. Testing File Attributes (35.26)

● Test if a file exists and is readable. [ -r schemas.txt ] [ -r $1.db ]

● Test if a file doesn't exist or is not writeable. [ ! -w databases.txt ]

● Test if a file exists and is executable. [ -x ns.sh ]

● Test if a file exists and is not a directory. [ -f tmp.txt ] Numeric Tests

● The following {-eq, -ne, -gt, -ge, -lt, -le } operators can be used for numeric tests. [ $1 -lt $2 ] [ $1 -gt 0 ] [ $1 -eq $# ] Exit (24.4, 35.12, 35.16)

● General form. The exit command causes the current shell script to terminate. There is an implicit exit at the end of each shell script. The status indicates the success of the script. If the status is not given, then the script will exit with the status of the last command. exit or exit Simple If Statement (35.13)

● General form. if condition then one-or-more-commands fi

● Example. if [ -r “tmp.txt” ] then echo “tmp.txt is readable.” fi Testing Strings

● Can perform string comparisons. It is good practice to put the shell variable being tested inside double quotes (“$v” instead of $v). [ “$1” = “yes” ]

● The following will give a syntax error when $1 is empty since: [ $1 != “yes” ]

● becomes [ != “yes” ] Testing Strings (cont.)

● Can check if a string is empty to avoid a syntax error. [ -z “$1” ]

● Can also check if a string is nonempty. [ -n “$1” ] Quoting Rules (27.12)

● 'xxx' disables all special characters in xxx.

● “xxx” disables all special characters in xxx except $, `, and \.

● \x disables the special meaning of character x. Quoting Examples animal=”horse” echo '$animal' # prints: $animal echo “$animal” # prints: horse cost=2000 echo 'cost: $cost' # prints: cost: $cost echo “cost: $cost” # prints: cost: 2000 echo “cost: \$cost” # prints: cost: $cost echo “cost: \$$cost” # prints: cost: $2000 String Relational Operators

● The string relational operators are the operators: =, !=, >, >=, <, <=

● The (>, >=, <, <=) assume an ASCII ordering when performing comparisons. They have to be used with the expr command, which will be covered later. The backslash has to be placed before these operators so they are not confused with I/O redirection.

● The following expression is true: “12” \< “2”

● The following expression is false: “12” -lt “2” Testing with Multiple Conditions (35.14)

● Can check if multiple conditions are all met. [ “$1” = “yes” ] && [ -r $2.txt ]

● Can check if one of a set of conditions are met. [ “$2” = “no” ] || [ ! -r $1.db ] General If Statement (35.13)

● General form. The “if condition”, initial “then”, and “fi” are required. Can have zero or more elif's. The else is also optional. if condition then one-or-more-commands elif condition then one-or-more-commands ... else one-or-more-commands fi If Statement Examples

if [ ! -r $1 ] then echo “file” $1 “not readable” fi

if [ $1 = $2 ] then echo $1 “is equal to” $2 else echo $1 “is not equal to” $2 fi If Statement Examples (cont.) if [ $var1 -lt $var2 ] then echo $var1 “is less than” $var2 elif [ $var1 -gt $var2 ] then echo $var1 “is greater than” $var2 else echo $var1 “is equal to” $var2 fi Case Statement (35.10)

● General form. Compares stringvalue to each of the strings in the patterns. On first match it does the corresponding commands. ;; indicates to jump to the statement after esac. *) means the default case. case stringvalue in pattern1) one-or-more-commands ;; pattern2) one-or-more-commands ;; ... *) one-or-more-commands ;; esac Case Statement Example echo “Please answer yes or no.” read answer case $answer in “yes”) echo “We are pleased you agree.” ... ;; “no”) echo “We are sorry you disagree.” ... ;; *) echo “Please enter \”yes\”or \”no\”.” esac While Statement (35.15)

● General form. The commands in the loop are performed while the condition is true. while condition do one-or-more-commands done While Statement Examples # process commands until a stop is # encountered read cmd while [ $cmd != “stop” ] do ... read cmd done

# process a loop forever while : do ... done Until Statement (35.15)

● General form. The commands in the loop are performed until the condition is true. until condition do one-or-more-commands done Until Statement Example read cmd until [ $cmd = “stop” ] do ... read cmd done Shift (35.22) ● The shift command removes an argument from the script file's argument list by shifting all the others over one ($1=$2; $2=$3; $3=$4; ...). It also decrements the $# value by one.

● Example: # print the command line arguments # two per line while [ $# -gt 0 ] do echo $1 $2 shift if [ $# -gt 0 ] then shift fi done Exit Status (35.12, 35.16)

● Zero normally indicates success. Nonzero values normally indicate some type of failure. It is a good practice to end each shell script with an “exit 0” command if everything succeeded.

● Most unix utilities, which are written in C, will call “exit();” upon termination to pass a value back to the shell or utility that invoked it so the utility's status can be checked.

Exit Command Example

● Below could be a script to get a yes/no answer. #!/bin/sh

echo “Please answer yes or no.”; read answer while : do case $answer in “yes”) exit 0 ;; “no”) exit 1 ;; *) echo “Invalid response.” echo “Please answer yes or no.” read answer ;; esac done Testing the Exit Status

● All conditions tested in control statements can also be the exit status of commands. The condition below uses the exit status of the yes.sh script shown in the previous slide. if yes.sh then echo “Please enter filename:” ... fi Regular Expressions (32.3)

● Many shell commands and Unix utilities use regular expressions.

● A regular expression is a description of a possible sequence of symbols (e.g. characters). Regular Expressions Operations (32.3)

● Concatentation is implicit. ab # 'a' followed by 'b' abc # 'a' followed by 'b' followed by 'c' ● * indicates zero or more instances of the preceding regular expression. a* # “”, a, aa, aaa, ... a*b # b, ab, aab, aaab, ... ● + indicates one or more instances. a+ # a, aa, aaa, ... a+b # ab, aab, aaab, ... Character Classes (32.8)

● '.' indicates any single character except a newline a.b # 'a' followed by any character followed by 'b'

● Use [...] to indicate one of a set of characters. The '-' operator within [] indicates a range. The '^' after the '[' means match anything not in the set. [abc] # a, b, c [0-9] # any decimal digit [a-z] # any lowercase letter [A-Z] # any uppercase letter [a-zA-Z] # any letter [^0-9] # any character other than a decimal digit [^\n] # same as '.' Anchors (32.5)

● Anchors can be used to indicate that a pattern will only match when it is at the beginning or end of a line. ^echo # “echo” at the beginning of the line [A-Za-z]+$ # a name at the end of the line ^done$ # “done” on a line by itself Alternation and Grouping

● Use the '|' character to choose between alternatives. Use parentheses for grouping. a|b # a or b a*|b # “”, a, aa, aaa, ..., b (ab)*c # c, abc, ababc, abababc, ... (a|b)*c # any combination of a's or b's followed by c

Additional RE References

● See 32.21 for additional operations, a quick reference guide, and more examples. Grep (13.2)

● grep searches for strings in files that match a regular expression and prints the lines that contain these matches to stdout.

● General form. The pattern is the regular expression. You can specify zero or more files. If no files are specified, then grep reads from standard input. The [...] below means that whatever is inside the brackets is optional. We will discuss the command line options later. grep [-i] [-w] [-c] [-v] pattern [files] Grep Examples grep [Ww]halley *.txt # Where Whalley or whalley occurs # in .txt files. grep interger *.c # Where did I misspell integer? grep ^\.TS report.tr # Did I use tables in the report.tr file? grep Whalley report.tex | grep David > tmp.out # Where did I put both my last and # first name on the same line of # report.tex? Place output in # tmp.out. Grep Options (13.2, 13.3)

● -i will make the search case insensitive.

● -c will cause the number of lines matched to be printed.

● -w will make the search look for entire words.

● -v will cause the lines that don't match to be output.

● -A num will print num lines after each line that was matched.

● -B num will print num lines before each line that was matched.

● -C num will print num lines before and after each line that was matched. Grep Examples Using Options grep -i whalley tmp.tr # finds both Whalley and whalley grep -c “do {“ prog.c # counts number of do-whiles in prog.c grep -w “int abs” *.c # find implementation of abs function grep -wc if prog.c # counts if statements in prog.c grep -v “^#” prog.c # prints all lines but preprocessor # commands in prog.c grep -c “/[\*/]” prog.c # counts number of comments in prog.c grep -C 1 Whalley # prints three lines (before, matched, # after) for each line containing Whalley Wc (16.6)

● wc counts the number of lines, words, and characters in files and prints this information to stdout.

● General form. You can specify zero or more files. Like grep and most Unix utilities, it reads from standard input if no files are specified. Again the [...] form means that the ... inside the brackets is optional. The options indicate to only print the number of lines (-l), words (-w), or characters (-c). wc [-l] [-w] [-c] [files] Wc Examples wc *.txt *.c # How big are all of my .txt and .c files? wc -l *.c # How many lines of code have I written? wc -w report.tex # How many words is my report? wc -c doc.pdf # How many bytes is this document that # that I downloaded over the internet? Touch

● Touch creates a file with no data if it does not exist. Sometimes you need to create an empty file. touch employee.txt

● It updates the last modification date/time to be the current date/time if the file does exist. This feature can be useful if you copied files into a directory and you need to have the Makefile recompile everything. touch *.cpp Command Substitution (28.14)

● A pair of backquotes, `...`, does command substitution. This means that the standard output of the command within the backquotes is used as arguments to another command. count=`wc -w < $1` # assigns to count the number # of words in the file $1

if [ `wc -l < $2.txt` -lt 1000 ] # checks if the number of lines # in the $2.txt file is < 1000 Xargs (28.17)

● The xargs command reads a group of arguments from standard input and then runs the specified Unix command with that group of arguments.

● General form. xargs

● Example: ls *.c > all_c_files.txt # capture *.c filenames vi all_c_files.txt # can delete some filenames xargs gcc -g -c < all_c_files.txt # compile the list of files For Statement (28.9, 35.21)

● General form: The shell variable is assigned each word in the list, where the set of commands is performed each time the word is assigned to the variable. If the “in ” is omitted, then the variable is assigned each of the command-line arguments. for [ in ] do one-or-more-commands done For Statement Examples

# make a backup of each of the C # files in the current directory for file in `ls *.c` do cp $file “$file”.bak done

# echo each of the command line # arguments to a separate line for arg do echo $arg done For Statement Examples (cont.)

# place the command line arguments into # a single string variable where the # arguments are separated by blanks s= for arg do s=”$s $arg” done ... # compile each of the files specified in # the list in $s for file in $s do gcc -g -c $file done Break and Continue (35.23)

● A break statement causes immediate termination of a loop. A continue statement skips the rest of the commands in the loop and starts the next iteration.

for file in `ls *.c` do if [ ! -r $file ] && [ ! -d $file ] continue elif ! gcc -c $file then echo “could not compile” $file break fi done Expr (36.21)

● Expr evaluates an arithmetic or relational expression and prints its results to standard output. Useful when your shell script needs to perform a calculation. Outputs 1 (true) or 0 (false) when evaluating a relational expression.

● General form. Note that the arguments and the operators must be separated by spaces. At least one argument must be specified. expr arg1 [ oper1 arg2 [ oper2 arg3 ... ] ] Expr Examples var=`expr “$var” + 1` # add 1 to var if [ `expr “$s1” \< “$s2”` = 1 ] # check if $s1 < $s2, works # for strings as well a=`expr “$a” \* 2` # double the value of a Awk (20.10)

● Awk is a Unix utility that can manipulate text files that are arranged in columns. Like most other Unix utilities, it reads from standard input if no files are specified.

● General forms: awk [-Fc] 'script' [files] awk [-Fc] -f scriptfile [files]

● -Fc indicates a field separator to be the specified character c. For instance, one could specify -F: to indicate that ':' is the field separator. By default the field separator is blanks and tabs. Awk Scripts (20.10)

● Each script consist of one or more pairs of: pattern { procedure }

● Awk processes each line of a file, one at a time. For each pattern that matches, then the corresponding procedure is performed.

● If the pattern is missing, then the procedure is applied to each line.

● If the { procedure } is missing, then the matched lines are written to standard output. Awk Patterns (20.10)

● Awk patterns can be any of the following. Awk patterns, except for BEGIN and END, can be combined using the logical operators ||, &&, and !.

– / regular expression /

– relational expression

– pattern-matching expression

– BEGIN

– END Awk Regular Expressions

● A pattern that uses a regular expression indicates that somewhere in the line the regular expression matches. The regular expression returns true or false depending on if it matches within the current line. Using a pattern only (causes each line to be printed that matches) results in similar functionality as grep. /D[Rr]\./ # matches any line containing DR. or Dr. /^#/ # matches any line beginning with '#' Awk Relational Expressions (20.10)

● An awk relational expression can consist of strings, numbers, arithmetic/string operators, relational operators, defined variables, and predefined variables.

– $0 means the entire line

– $n means the nth field in the current line

– NF is a predefined variable indicating the number of fields in the current line

– NR is the number of the current line Example Awk Relational Exprs (20.10)

$1 == “if” # Is first field an “if”? $1 == $2 # Are first two fields the same? NR > 100 # Have already processed 100 lines? NF > 5 # Does current line have > 5 fields? NF > 5 && $1 == $2 # Compound condition. /if/ && /

● Pattern matching expressions can check if a regular expression matches (~) or does not match (!~) a field. Note that the pattern does not have to match the entire field. $1 ~ /D[Rr]\./ # First field match “DR.” or “Dr.”? $1 !~ /#/ # First field does not match “#”? Awk BEGIN and END Patterns (20.10)

● The BEGIN pattern lets you specify a procedure before the first input line is processed.

● The END pattern lets you specify a procedure after the last input line is processed. Awk Procedures (20.10)

● An awk procedure specifies the processing of a line that matches a given pattern. An awk procedure is contained within the '{' '}' and consists of commands separated by semicolons or newlines.

● Commonly used awk commands include: print [args] # print arguments, if none print $0 var = expr # assignment to variables Example Awk Scripts (20.10)

{ print $1, $2 } # print the first two fields of each line

$0 !~ /^$/ # print lines with one or more chars

$2 > 0 && $2 < 10 # print second field when its value {print $2} # is in the range 1..9

BEGIN {sum=0} # sums the values of the fields in the {sum += $1} # first column and prints the sum END {print sum} Tr (21.11)

● The tr unix utility is a character translation filter. The most common usage is to replace one character for another. It reads from standard input and writes to standard output.

● General forms. tr string1 string2 # replace character in string1 with # corresponding character in string2 tr -d string # delete all occurrences of characters # that appear in string tr -s string1 string2 # replace instances of repeated chars # in string1 with a single char in # string2 Tr Examples tr “[a-z]” “[A-Z]” # convert lower case to uppercase tr '&' '#' # convert &'s to #'s tr -s “\t” “\t” # squeeze consecutive tabs to a # single tab tr -d '\015' # deletes the carriage return # characters from a DOS file Basename (36.13)

● The basename utility strips off the portion of a pathname that precedes and including the last slash and prints the result to standard output. % basename /home/faculty/whalley/cop4342exec/plot.p plot.p

# The gcc script needs to strip off the path and create a file called “echo.o” # in the current directory. % gcc -c /home/faculty/whalley/test/src/echo.c Dirname (36.13)

● The dirname utility strips off the last slash and everything that follows it in the pathname. % dirname /home/faculty/whalley/cop4342exec/plot.p /home/faculty/whalley/cop4342exec

● Often shell scripts will determine the directory in which they reside. dir=`dirname $0` if $dir/database_exists.sh $1 then ... fi Sort (22.2)

● Sort divides each line into fields separated by a whitespace (blanks or tabs) character and sorts the lines by field, from left to right. It writes to standard output.

● General form. The -b option causes leading blank characters for each field to be ignored. The -r option causes the sort to be in reverse order. The +pos# indicates the position of the next field to start sorting. The -pos# indicates the position of the field to stop sorting. Field positions are numbered starting from 0. If no files are specified, then it sorts the standard input. If more than one file is specified, then it sorts all the files together.

sort [-b] [-r] [+pos# [-pos#]]* [files] Sort Examples

● Say the file “data.txt” contains the following: Smith Bob 27 32312 Jones Carol 34 32306 Miller Ted 27 32313 Smith Alice 34 32312 Miller Ted 45 32300

● “sort data.txt” would produce: Jones Carol 34 32306 Miller Ted 27 32313 Miller Ted 45 32300 Smith Alice 34 32312 Smith Bob 27 32312 Sort Examples (cont.)

● “sort +2 data.txt” would produce: Smith Bob 27 32312 Miller Ted 27 32313 Jones Carol 34 32306 Smith Alice 34 32312 Miller Ted 45 32300

● “sort +2 -3 +0 data.txt” would produce: Miller Ted 27 32313 Smith Bob 27 32312 Jones Carol 34 32306 Smith Alice 34 32312 Miller Ted 45 32300 Sort Examples (cont.)

● “sort +3 data.txt” would produce: Miller Ted 45 32300 Jones Carol 34 32306 Smith Alice 34 32312 Smith Bob 27 32312 Miller Ted 27 32313

● “sort +1 -2 +0 -1 +3 data.txt” would produce: Smith Alice 34 32312 Smith Bob 27 32312 Jones Carol 34 32306 Miller Ted 45 32300 Miller Ted 27 32313 Reverse Sort (22.6)

● Use the -r option to reverse the order of the sort.

● “sort -r < data.txt” would produce: Smith Bob 27 32312 Smith Alice 34 32312 Miller Ted 45 32300 Miller Ted 27 32313 Jones Carol 34 32306 Type of Sort (22.5)

● By default, sort orders the lines in alphabetical (actually ASCII) order. You can specify that a particular field is to be sorted in numerical order by including an 'n' after the field position.

● Say the file “data2.txt” contained: Smith 42 Jones 7 Miller 100

● “sort +1” produces and “sort +1n” produces Miller 100 Jones 7 Smith 42 Smith 42 Jones 7 Miller 100 Ignoring Extra Whitespace (22.6)

● The -b option indicates to ignore extra whitespace before each field. But you have to specify the fields that you want to sort.

● Say we have the following file called “data.txt”. Smith Bob 27 32312 Jones Carol 34 32306 Miller Ted 27 32312 Smith Alice 34 32312 Miller Ted 45 32300

● What would “sort < data.txt” produce?

● How could you get an ordering where extra whitespace is ignored and the age field is sorted numerically? Latex

● Latex is document markup language and document preparation system. ● It is used as a text formatter (batch) as opposed to a WYSIWYG editor (interactive), like MS Word. ● Latex converts a text file with embedded latex commands into a dvi file. ● Below is the general form of a Latex file. \documentclass{class} \begin{document} ... \end{document}

● Example invocation. latex tmp.tex # produces a tmp.dvi file dvipdf tmp.dvi tmp.pdf # produces a tmp.pdf file Latex Table Specification Format

● Multiple tables may appear in a latex document.

● Table specification format: \begin{table} table alignment \begin{tabular}{column format} data to be laid out in the table ... data to be laid out in the table \end{tabular} \end{table} Latex Table Example

● Assume the following text in a tmp.tex file. \documentclass{article} % document class article \pagestyle{empty} % no page numbers \begin{document} % start of document \begin{table} % start of table \centering % center table \begin{tabular}{|l|l|r|c|} % specify column format \hline % line at top of table Last & First & Age & Zipcode\\ % headings \hline % line under headings Jones & Carol & 34 & 32306\\ Miller & Ted & 27 & 32313\\ Miller & Ted & 45 & 32300\\ Smith & Alice & 34 & 32312\\ Smith & Bob & 27 & 32312\\ \hline % line at end of table \end{tabular} % end of tabular \end{table} % end of table \end{document} % end of document Latex Example (cont.)

● One can view a pdf file with acroread. Issuing the following commands will result in the table being displayed. latex tmp.tex dvipdf tmp.dvi tmp.pdf acroread tmp.pdf Another Latex Table Example ● Assume the following text in a tmp2.tex file. \documentclass{article} % document class article \pagestyle{empty} % no page numbers \usepackage{multirow} % used for spanning rows \begin{document} % start of document \begin{table} % start of table \centering % center table \begin{tabular}{|l|l|r|c|} % specify column format \hline % line at top of table \multicolumn{2}{|c|}{Name} & \multirow{2}{*}{Age} & \multirow{2}{*}{Zipcode}\\ % headings \cline{1-2} % line under first two cols Last & First & & \\ \hline % line under headings Jones & Carol & 34 & 32306\\ ... Smith & Bob & 27 & 32312\\ \hline % line at end of table \end{tabular} % end of tabular \end{table} % end of table \end{document} % end of document Another Latex Example (cont.)

● Issuing the following commands will result in the previous table being displayed. latex tmp2.tex dvipdf tmp2.dvi tmp2.pdf acroread tmp2.pdf Fmt Utility (21.2)

● The fmt utility is a simple text formatter that fills and joins lines to produce output up to the number of characters specified or 72 by default.

● General form. fmt [-w width] [inputfiles] Fmt Utility (21.2)

● The fmt utility is a simple text formatter that fills and joins lines to produce output up to the number of characters specified or 72 by default.

● General form. fmt [-w width] [inputfiles] Fmt Example

● Say you have the following text in a tmp.txt file. Four score and seven years ago our fathers brought forth on this continent, a new nation, conceived in Liberty, and dedicated to the proposition that all men are created equal.

● Using the command “fmt -w 30 tmp.txt”, the output would be: Four score and seven years ago our fathers brought forth on this continent, a new nation, conceived in Liberty, and dedicated to the proposition that all men are created equal. Cut (21.14)

● Cut allows you to select a list of fields from one or more files. If no files are specified, then it reads from standard input. It writes to standard output.

● General form. The -d option allows you to change the field delimiter. By default it is a tab. The -f indicates a list of fields to be cut. Fields are numbered starting from 1. The list can also include ranges of fields. At least one field must be specified. cut [-d“”] -f[,]* [file]* Cut Examples cut -d“ ” -f1 tmp.db # print the first field from tmp.db cut -f2,4 data.txt # print the 2nd and 4th fields from data.txt cut -d: -f1 /etc/passwd # print the user id's from the passwd file cut -f1,3-5 data.txt # print 1st, 3rd,4th, 5th fields from data.txt Paste (21.18)

● The paste utility merges files together by concatenating the corresponding lines of the specified input files. It writes to standard output. If one input file is not as long as another, then paste will treat the shorter file as having empty lines at the end. By default, paste replaces each newline character with a tab, except for the newline character in the last file. You can replace the tab with some other character by using the -d option.

● General form. paste [-d] files Paste Example

● Paste is often used together with the cut utility. # cut the first field of file1.txt and file2.txt cut -f1 file1.txt > tmp1.txt cut -f1 file2.txt > tmp2.txt

# paste these fields together separated by a blank paste -d“ ” tmp1.txt tmp2.txt > file3.txt

# remove temporary files rm tmp1.txt tmp2.txt Head (12.12) and Tail (12.8-12.10)

● Head prints the first few lines of one or more files. Tail prints the last few lines of a file.

● General form. If - is not specified, then 10 lines are printed by default. The -f option indicates that tail will monitor the file every second and update it if it changes. head [-] [files] tail [-] [-f] [file] Head and Tail Examples head data.txt # print first 10 lines of data.txt head data1.txt data2.txt # print first 10 lines of data1.txt # and data2.txt grep Smith report.tr | head -5 # print first 5 lines containing # Smith in report.tr tail -100 data.txt # print last 100 lines of data.txt tail -f tmp.out # monitor last 10 lines of tmp.out

Sed (34.2, 34.3, 34.5, 34.6)

● The sed utility is a stream editor that reads one or more files, makes changes according to a script of editing commands, and writes the results to standard output.

● General forms. “script” is a sed command. You can specify more than one command by using the -e option or use the -f option and place the sed commands in a file. sed “script” [file]* sed -e “script” [-e “script”]* [file]* sed -f scriptfile [file]* Sed, Ed, and Ex (20.1)

● Sed performs many ed or ex commands. Ed and ex are line editors, which were the original type of editors. Line editing commands are easier for scripting.

● Line editing commands can be used in the vi editor and many commands can be used in the sed utility. Line Addressing (20.3) n # absolute line number n i,j # range of lines from line i to line j # '.' means current line, '$' means last line + # go forward one line - # go back one line +n # go forward n lines -n # go backward n lines Line Editing Commands addr # goto addr [addr] l # print line(s) to standard output / # goto first line that matches pattern ? # goto last line that matches pattern [addr] a text . # append text (line by line) after addr # until a '.' is typed in the first column [addr] i text . # insert text (line by line) before addr # until a '.' is typed in the first column [addr] d # delete specified line(s) [addr] c text . # replace specified line(s) with text # until a '.' is typed in the first column [addr] s/pat1/pat2/ # substitute pat2 for pat1 at addr Using Editing Commands in Sed

● Sed editing commands are implicitly global (applied to every line in the file).

● Example sed scripts: 50d # delete line 50 /^#/d # deletes all lines beginning with '#' /Smith/,$d # finds first line containing “Smith” and # deletes that line and all remaining lines s/old/new/ # substitute “old” with “new” in the 1st # occurrence of each line s/old/new/g # substitute “old” with “new” each place # it occurs s/old/new/2 # substitute “old” with “new” on the 2nd # occurrence of each line Performing Multiple Commands on a Match (34.5)

● Can use curly braces to perform multiple commands on a single match.

● Example: Make each list element within an ordered list a paragraph and change the ordered list to an unordered list in an html file. /^

/,/^<\/ol>/{ # find lines comprising an ordered list s/^
/
/ # change start to be an unordered list s/^
• /

• / # change list elements to be a paragraph s/^<\/ol>/<\/ul>/ # change end to be an unordered list } Referencing the Search String (34.10)

  ● The ampersand (&) represents the extent of the pattern that was matched and can be referenced in the replacement string.

  ● Example: s/[A-Z][A-Z][A-Z]*/“&”/g # quote all acronyms

  s/[Uu]nix/“&”/g # quote all references to # Unix or unix Referencing Portions of a Search String (34.11)

  ● Can reference portions of a search string by enclosing them in escaped parentheses, “\(” and “\)”, and referencing them by \ in the replacement string.

  ● Example: Reversing the order of the first two integer values separated by a colon in a file. s/^\([0-9]*\):\([0-9]*\)/\2:\1/ Gnuplot

  ● GNU is a free software foundation and provides a significant number of free software packages. Gnuplot is one of these packages.

  ● Gnuplot takes a sequence of plotting commands and generates plots. It can take its commands in an interactive (from standard input) or batch (from a file) mode.

  ● For detailed documentation, see: http://www.gnuplot.info/documentation.html

  ● General form. gnuplot [files] Some Common Gnuplot Commands set term # set output type and font size set output # redirect output to a file set xlabel # assign label for x axis set ylabel # assign label for y axis set xr [:] # set min and max of x range set yr [:] # set min and max of y range set xtics ,, # label tics on x axis set ytics ,, # label tics on y axis set boxwidth # set default width of boxes set key # set location of legend Gnuplot Plot Command

  ● General Form. plot “” using [:] title “