Unix 201

Purdue University - ITaP

Gladys Andino Dan Dietz Jieyu Gao Lev Gorenstein Erik Gough Stephen Harrell Randy Herban Steve Kelley Boyu Zhang Xiao Zhu

rcac-help@purdue.edu

February 6th and 8th, 2018

Slides available: https://www.rcac.purdue.edu/training/unix201/ Acknowledgments

Acknowledgments

Logging In

Text Manipulation

Regular Expressions

Advanced Text Acknowledgments Manipulation

Redirects and Loops

Bash Programming

Conditionals and Loops

2 / 143 Acknowledgments

Acknowledgments Logging In A few additional acknowledgments to the many people who Text have helped make this workshop possible. Manipulation

Regular • The material in this workshop was prepared by the Purdue Expressions University ITaP Research Computing team. Advanced Text • Special thanks to Eric Adams and Megan Dale for Manipulation organizing the workshop sessions. Redirects and Loops • We have drawn from documentation provided by the Bash Programming Purdue Bioinformatics Core used in the UNIX for

Conditionals Biologists workshop and Next-generation Transcriptome and Loops Analysis Workshop Manual provided by Professor Michael Gribskov and Professor Esperanza Torres.

3 / 143 Logging In

Acknowledgments

Logging In Windows Mac Activity Files Text Manipulation Logging In Regular Expressions Windows Advanced Mac Text Manipulation Activity Files Redirects and Loops

Bash Programming

Conditionals and Loops

4 / 143 Logging In

Acknowledgments Logging In We will be using the Radon cluster: Windows Mac • www.rcac.purdue.edu/compute/radon/ Activity Files Text • Everyone has been given an account on the cluster for the Manipulation duration of the workshop Regular Expressions • If you wish to continue using Radon or other cluster after Advanced Text the workshop concludes, please make a request under your Manipulation advisor or PI’s name: Redirects and Loops https://www.rcac.purdue.edu/account/request/

Bash Programming

Conditionals and Loops

5 / 143 Logging In Windows

Acknowledgments Logging In Many clients are available for Windows: Windows Mac • We will use the PuTTY SSH client Activity Files Text • Download PuTTY, no install required Manipulation

Regular • http://www.chiark.greenend.org.uk/ sgtatham/putty/- Expressions download.html Advanced Text (or Google search putty) Manipulation • Download putty.exe for Intel x86 to your desktop Redirects and Loops

Bash Programming

Conditionals and Loops

6 / 143 Logging In Windows

Acknowledgments Logging In Host Name for Radon is radon.rcac.purdue.edu Windows Mac Activity Files Text Manipulation

Regular Expressions

Advanced Text Manipulation

Redirects and Loops

Bash Programming

Conditionals and Loops

7 / 143 Logging In Windows

Acknowledgments Logging In One tweak: enable system colors in Appearance → Colours Windows Mac Activity Files Text Manipulation

Regular Expressions

Advanced Text Manipulation

Redirects and Loops

Bash Programming

Conditionals and Loops

8 / 143 Logging In Mac

Acknowledgments Logging In Connect using: Windows Mac ssh [email protected] Activity Files Text Manipulation

Regular Expressions

Advanced Text Manipulation

Redirects and Loops

Bash Programming

Conditionals and Loops

9 / 143 Logging In Mac

Acknowledgments Logging In Linux also has a built in terminal client, similar to Mac: Windows Mac ssh [email protected] Activity Files Text Manipulation

Regular Expressions

Advanced Text Manipulation

Redirects and Loops

Bash Programming

Conditionals and Loops

10 / 143 Logging In Activity Files

Acknowledgments Logging In We’ll need a few files for some of the hands-on activities Windows Mac cd Activity Files $ $ cp -r /depot/itap/unix101 . Text Manipulation

Regular Expressions

Advanced Text Manipulation

Redirects and Loops

Bash Programming

Conditionals and Loops

11 / 143 Text Manipulation

Acknowledgments

Logging In

Text Manipulation wc cut sort Text Manipulation uniq Exercises wc Regular cut Expressions sort Advanced Text uniq Manipulation Exercises Redirects and Loops

Bash Programming

Conditionals and Loops

12 / 143 Text Manipulation wc

Acknowledgments Logging In The wc (word count) command simply counts the number of Text lines, words, and characters. Manipulation wc cut sort General syntax: uniq Exercises wc [OPTIONS] FILENAME Regular Expressions include: Advanced OPTIONS Text Manipulation • -l count lines only Redirects and • -w count words only Loops Bash • -c count characters only Programming

Conditionals and Loops

13 / 143 Text Manipulation wc

Acknowledgments Logging In Try this: Text Manipulation $ cd ~/unix101/Shakespeare wc $ cat wcdemo.txt cut This is just a very simple sort uniq text file that we'll use Exercises to demonstrate wc Regular Expressions $ wc wcdemo.txt Advanced 3 14 70 wcdemo.txt Text Manipulation This tells us that the file wcdemo.txt has: Redirects and Loops • 3 lines Bash Programming • 14 words Conditionals and Loops • 70 characters The we’ll in the file looks like one word to Unix text processing commands. 14 / 143 Text Manipulation cut

Acknowledgments Logging In The cut command is used to select sections of each line of a Text file or files. Manipulation wc cut sort General syntax: uniq Exercises cut [OPTIONS] FILENAME Regular Expressions include: Advanced OPTIONS Text Manipulation • -d specify a character instead of TAB for field delimiter Redirects and • -f select only these fields; also print any line that contains Loops

Bash no delimiter character Programming

Conditionals and Loops

15 / 143 Text Manipulation cut

Acknowledgments Logging In Try this: Text Manipulation $ cd ~/unix101/protein wc cut $ head -n 5 1UBQ.pdb sort uniq HEADER CHROMOSOMAL PROTEIN 02-JAN-87 1UBQ Exercises TITLE STRUCTURE OF UBIQUITIN REFINED AT 1.8 ANGSTROMS RESOLUTION Regular COMPND MOL_ID: 1; Expressions COMPND 2 MOLECULE: UBIQUITIN; Advanced COMPND 3 CHAIN: A; Text Manipulation $ cut -f1 -d' ' 1UBQ.pdb | head -n 5 Redirects and HEADER Loops TITLE Bash COMPND Programming COMPND Conditionals COMPND and Loops

16 / 143 Text Manipulation sort

Acknowledgments Logging In The sort command is used to sort lines of a text file. Text Manipulation wc General syntax: cut sort sort [OPTIONS] FILENAME uniq Exercises Regular OPTIONS include: Expressions

Advanced • -n compare according to numerical value. Text Manipulation • -r reverse the result of comparisons. Redirects and Loops • -u return only unique lines. Bash Notes: Programming

Conditionals • By default, lines are sorted alphabetically. and Loops • By default, lines starting with numbers are not sorted numerically. For example, ”8 9 10 11” would be sorted as ”10 11 8 9”. 17 / 143 Text Manipulation sort

Acknowledgments Logging In Try sort on words and num.txt: Text cd ~/unix101/Shakespeare Manipulation $ wc $ sort words_and_num.txt cut sort sort sort -n sort -r uniq $ $ $ Exercises 1 ana zoo Regular 11 MAX MAX Expressions 23 zoo ana

Advanced 3 1 7 Text 5 3 5 Manipulation 7 5 3 Redirects and ana 7 23 Loops MAX 11 11 Bash zoo 23 1 Programming Count unique values in first column: Conditionals and Loops $ cd ~/unix101/protein $ cut -f1 -d'' 1UBQ.pdb | sort -u | wc -l 27

18 / 143 Text Manipulation uniq

Acknowledgments Logging In The uniq command simply takes a sorted file and outputs the Text unique lines in it. The input must be sorted first. Manipulation wc cut sort General syntax: uniq Exercises uniq [OPTIONS] INPUT Regular Expressions include: Advanced OPTIONS Text Manipulation • -c count how many times each line occurred. Redirects and • -d only print duplicated lines. Loops

Bash Programming

Conditionals and Loops

19 / 143 Text Manipulation uniq

Acknowledgments Logging In Try this: Text Manipulation $ cd ~/unix101/Shakespeare wc cut $ sort HamletWords.txt | uniq -c | head -n 5 sort uniq 36 1 Exercises 12 2 Regular 531 a Expressions 3 'a Advanced 1 abate Text $ sort HamletWords.txt | uniq -c | head -n 5 | sort -n Manipulation 1 abate Redirects and 3 'a Loops 12 2 Bash 36 1 Programming 531 a Conditionals and Loops

20 / 143 Text Manipulation uniq

Acknowledgments Logging In Try this: Text Manipulation $ cd ~/unix101/Shakespeare wc cut $ sort HamletWords.txt | uniq -u | head -n 5 sort uniq abate Exercises abatements Regular abhorred Expressions ability Advanced Able Text $ sort HamletWords.txt | uniq -u > uniques Manipulation $ cat uniques Redirects and $ sort HamletWords.txt | uniq -u | wc -l Loops 3145 Bash Programming

Conditionals and Loops

21 / 143 Text Manipulation Exercises

Acknowledgments Logging In Try the following command sequence: Text Manipulation 1. Change directory to ~/unix101/data wc cut 2. Using a single line command, "ls" all the files in this sort uniq directory and sort alphabetically then "ls -l" and sort Exercises Regular 3. Find out how many times "TAIR00" and "TAIR10" Expressions appear in the file at genes.txt Advanced Text 4. Using a single line command find out how many unique Manipulation descriptions appear for column 3 in the "at genes.txt", Redirects and Loops please perform the search in a numerical order Bash Programming 5. Display 1st, 4th and 5th column of the "at genes.txt"

Conditionals file, sorted in ascending order according to second field and Loops

22 / 143 Text Manipulation Exercises

Acknowledgments Logging In Answers: Text 1. Change directory to ~/unix101/data Manipulation wc cut $ cd ~/unix101/data sort $ pwd uniq /home/gandino/unix101/data Exercises Regular Expressions 2. Using a single line command, "ls" all the files in this Advanced directory and sort alphabetically then "ls -l" and sort Text Manipulation $ ls | sort Redirects and at_genes.txt Loops awkdata.txt Bash grepdata.txt Programming Conditionals $ ls -l | sort and Loops -rw-r--r-- 1 gandino entm 215 Feb 3 18:13 awkdata.txt -rw-r--r-- 1 gandino entm 6677 Feb 3 18:13 at_genes.txt -rw-r--r-- 1 gandino entm 744 Feb 3 18:13 grepdata.txt

23 / 143 Text Manipulation Exercises

Acknowledgments Logging In Answers: Text Manipulation 3. Find out how many times "TAIR00" and "TAIR10" wc appear in the file at genes.txt cut sort uniq $ cut -f2 at_genes.txt | sort | uniq -c Exercises 2 TAIR00 Regular 98 TAIR10 Expressions

Advanced # adding grep to the line Text cut -f2 at_genes.txt | sort |grep TAIR |uniq -c Manipulation $ 2 TAIR00 Redirects and 98 TAIR10 Loops

Bash Programming

Conditionals and Loops

24 / 143 Text Manipulation Exercises

Acknowledgments Logging In Answers: Text Manipulation 4. Using a single line command find out how many unique wc descriptions appear for column 3 in the "at genes.txt", cut sort please perform the search in a numerical order uniq Exercises $ cut -f 3 at_genes.txt | sort | uniq -c | sort -n Regular Expressions 1 chromosome 4 gene Advanced Text 5 mRNA Manipulation 5 protein 6 five_prime_UTR Redirects and Loops 6 three_prime_UTR 35 CDS Bash Programming 38 exon

Conditionals and Loops

25 / 143 Text Manipulation Exercises

Acknowledgments Logging In Answers: Text Manipulation 5. Display 1st, 4th and 5th field of the "at genes.txt" file, wc sorted in ascending order according to second field cut sort uniq $ cut -f1,4,5 at_genes.txt | sort -n -k 2 | head -n8 Exercises Chr1 1 30427671 Regular Chr1 3631 3759 Expressions Chr1 3631 3913 Advanced Chr1 3631 5899 Text Chr1 3631 5899 Manipulation Chr1 3760 3913 Redirects and Chr1 3760 5630 Loops Chr1 3996 4276 Bash Programming

Conditionals and Loops

26 / 143 Regular Expressions

Acknowledgments

Logging In Text Regular Expressions Manipulation

Regular Overview Expressions Simple Example Overview Simple Example Character Groups Character Groups Quantifiers Quantifiers Character Classes Character Classes Escaping Negating Escaping Anchors Grouping Negating Modifiers References Anchors Exercises Grouping Advanced Text Modifiers Manipulation

Redirects and References Loops Exercises Bash Programming 27 / 143 Conditionals and Loops Regular Expressions Overview

Acknowledgments Logging In Regular expressions, what are they? Text Manipulation • Expression that defines a search pattern Regular • Can define a search for complex patterns Expressions Overview • Extract matches from text Simple Example Character Groups • grep examples from last workshop very simple version of Quantifiers Character regular expression Classes Escaping Negating • Can get way more fancy! Anchors Grouping • Deep complex field in computer science Modifiers References Exercises • Well just brush the surface and hit the basics Advanced Text Manipulation

Redirects and Loops

Bash Programming 28 / 143 Conditionals and Loops Regular Expressions Overview

Acknowledgments Logging In For all of these examples we will be searching the quote from Text Shakespeare’s Hamlet: Manipulation

Regular Expressions ”Though this be madness, yet there is method in it.” Overview Simple Example Character Groups Yes, this is all madness but there is a reason behind it! Quantifiers Character Classes Escaping Negating Anchors Grouping Modifiers References Exercises Advanced Text Manipulation

Redirects and Loops

Bash Programming 29 / 143 Conditionals and Loops Regular Expressions Simple Example

Acknowledgments Logging In Lets say we just want to search for the word ”madness”. Think Text of regular expressions as a ”flow chart”. Start at the beginning Manipulation of the input string and expression. Regular Expressions Overview Simple Example Expression: Character Groups /madness/ Quantifiers Character Classes Escaping Negating Input: Anchors Grouping Though this be madness, yet there is method in it. Modifiers References Exercises m a d n e s s Advanced Text Manipulation Result: Redirects and Loops Though this be madness, yet there is method in it.

Bash Programming 30 / 143 Conditionals and Loops Regular Expressions Simple Example

Acknowledgments Logging In Lets try to search for the word ”is” Text Manipulation

Regular Expressions Expression: Overview Simple Example /is/ Character Groups Quantifiers Character Classes Input: Escaping Negating Though this be madness, yet there is method in it. Anchors Grouping Modifiers References Exercises Result: Advanced Text Though this be madness, yet there is method in it. Manipulation

Redirects and Loops

Bash Programming 31 / 143 Conditionals and Loops Regular Expressions Simple Example

Acknowledgments Logging In Lets try to refine this Text Manipulation

Regular Expressions Expression: Overview Simple Example / is / Character Groups Quantifiers Character Classes Input: Escaping Negating Though this be madness, yet there is method in it. Anchors Grouping Modifiers References Exercises Result: Advanced Text Though this be madness, yet there is method in it. Manipulation

Redirects and Loops

Bash Programming 32 / 143 Conditionals and Loops Regular Expressions Character Groups

Acknowledgments Logging In Can define a group of characters with [] Text Manipulation Expression: Regular Expressions / madness / Overview Simple Example Character Groups Better: Quantifiers Character Classes / madness[,. ]/ Escaping Negating Anchors Grouping Input: Modifiers References Though this be madness, yet there is method in it. Exercises Advanced Text Manipulation Result: Redirects and Loops Though this be madness, yet there is method in it. Bash Programming 33 / 143 Conditionals and Loops Regular Expressions Character Groups

Acknowledgments Logging In Another example Text Manipulation Expression: Regular Expressions / i[sn] / Overview Simple Example Character Groups Input: Quantifiers Character Classes Though this be madness, yet there is method in it. Escaping Negating Anchors s Grouping ”” i ”” Modifiers References Exercises n Advanced Text Result: Manipulation Though this be madness, yet there is method in it. Redirects and Loops

Bash Programming 34 / 143 Conditionals and Loops Regular Expressions Character Groups

Acknowledgments Logging In Character groups can specify range of characters: Text Manipulation [A-Za-z] Regular Expressions [0-9] Overview Simple Example Character Groups Quantifiers Character Classes Escaping Negating Anchors Grouping Modifiers References Exercises Advanced Text Manipulation

Redirects and Loops

Bash Programming 35 / 143 Conditionals and Loops Regular Expressions Quantifiers

Acknowledgments Logging In We can specify how many of a thing we want with quantifiers: Text Manipulation • Use * to say ”zero or more times” Regular • Applies to the preceding ”thing” (character, group, etc) Expressions Overview Expression: Simple Example Character Groups /madnes*/ Quantifiers Character Classes Escaping Input: Negating Anchors Though this be madness, yet there is method in it. Grouping Modifiers References s Exercises Advanced Text m a d n e Manipulation

Redirects and Loops Result:

Bash Though this be madness, yet there is method in it. Programming 36 / 143 Conditionals and Loops Regular Expressions Quantifiers

Acknowledgments Logging In Another example: bogus character. Remember, zero or more Text times. Manipulation

Regular Expressions Expression: Overview Simple Example / madnessq*,/ Character Groups Quantifiers Character Classes Input: Escaping Negating Though this be madness, yet there is method in it. Anchors Grouping Modifiers References Exercises Result: Advanced Text Though this be madness, yet there is method in it. Manipulation

Redirects and Loops

Bash Programming 37 / 143 Conditionals and Loops Regular Expressions Quantifiers

Acknowledgments Logging In Can also apply to character groups Text Manipulation Expression: Regular Expressions / madness[,. ]*/ Overview Simple Example Character Groups Input: Quantifiers Character Classes Though this be madness, yet there is method in it. Escaping Negating Anchors Grouping Modifiers References Result: Exercises Though this be madness, yet there is method in it. Advanced Text Manipulation

Redirects and Loops

Bash Programming 38 / 143 Conditionals and Loops Regular Expressions Quantifiers

Acknowledgments Logging In Use ? to say ”zero or one times”, or ”optional” Text Manipulation Expression: Regular Expressions / madness?,/ Overview Simple Example / madnesss?,/ Character Groups Quantifiers Character Classes Input: Escaping Negating Though this be madness, yet there is method in it. Anchors Grouping Modifiers References m a d n e s s Exercises Advanced , Text Manipulation

Redirects and Loops Result: Bash Programming Though this be madness, yet there is method in it. 39 / 143 Conditionals and Loops Regular Expressions Quantifiers

Acknowledgments Logging In Use + to say ”one or more times” Text Manipulation Expression: Regular Expressions / madnes+,/ Overview Simple Example Character Groups Input: Quantifiers Character Classes Though this be madness, yet there is method in it. Escaping Negating Anchors s Grouping Modifiers References m a d n e s , Exercises Advanced Text Manipulation Result: Redirects and Loops Though this be madness, yet there is method in it. Bash Programming 40 / 143 Conditionals and Loops Regular Expressions Quantifiers

Acknowledgments Logging In Can specify precise counts with {} Text Manipulation Expression: Regular Expressions /s{2}/ Overview Simple Example Character Groups Input: Quantifiers Character Classes Though this be madness, yet there is method in it. Escaping Negating Anchors Grouping Modifiers References Result: Exercises Though this be madness, yet there is method in it. Advanced Text Manipulation

Redirects and Loops

Bash Programming 41 / 143 Conditionals and Loops Regular Expressions Quantifiers

Acknowledgments Logging In Can specify precise count ranges, or even open ended ranges Text Manipulation Expression: Regular Expressions /s{1,2}/ Overview Simple Example /s{1,}/ Character Groups Quantifiers Character Classes Input: Escaping Negating Though this be madness, yet there is method in it. Anchors Grouping Modifiers References Exercises Result: Advanced Text Though this be madness, yet there is method in it. Manipulation

Redirects and Loops

Bash Programming 42 / 143 Conditionals and Loops Regular Expressions Character Classes

Acknowledgments Logging In Most flavors of regular expressions have the notion of a Text character class. They are a special syntax to specify complex Manipulation character group ranges. Regular Expressions Overview Simple Example Word class: Character Groups /\w+/ /[A-Za-z0-9 ]+/ Quantifiers Character Classes Escaping Negating Space class: Anchors Grouping /\s+/ /[ \t\r\n\f]+/ Modifiers References Exercises Advanced Text Manipulation

Redirects and Loops

Bash Programming 43 / 143 Conditionals and Loops Regular Expressions Character Classes

Acknowledgments Logging In What if we want the two words before commas? Text Manipulation Expression: Regular Expressions /\w+\s\w+,/ Overview Simple Example Character Groups Input: Quantifiers Character Classes Though this be madness, yet there is method in it. Escaping Negating Anchors Grouping Modifiers References Result: Exercises Though this be madness, yet there is method in it. Advanced Text Manipulation

Redirects and Loops

Bash Programming 44 / 143 Conditionals and Loops Regular Expressions Character Classes

Acknowledgments Logging In There is a special character ”.” Text It does everything! Manipulation

Regular Expressions Expression: Overview Simple Example /madness.*/ Character Groups Quantifiers Character Classes Input: Escaping Negating Though this be madness, yet there is method in it. Anchors Grouping Modifiers References Exercises Result: Advanced Text Though this be madness, yet there is method in it. Manipulation

Redirects and Loops

Bash Programming 45 / 143 Conditionals and Loops Regular Expressions Escaping

Acknowledgments Logging In What if we to search for one of those special characters? Text Escape with \ Manipulation

Regular Expressions Expression: Overview Simple Example /\./ Character Groups Quantifiers Character Classes Input: Escaping Negating Though this be madness, yet there is method in it. Anchors Grouping Modifiers References Exercises Result: Advanced Text Though this be madness, yet there is method in it. Manipulation

Redirects and Loops

Bash Programming 46 / 143 Conditionals and Loops Regular Expressions Negating

Acknowledgments Logging In What if we don’t want to match something? Use ˆ in a Text character class Manipulation

Regular Expressions Expression: Overview Simple Example /[ˆmad]/ Character Groups Quantifiers Character Classes Input: Escaping Negating Though this be madness, yet there is method in it. Anchors Grouping Modifiers References Exercises Result: Advanced Text Though this be madness, yet there is method in it. Manipulation

Redirects and Loops

Bash Programming 47 / 143 Conditionals and Loops Regular Expressions Anchors

Acknowledgments Logging In We can anchor an expression in a particular part of a string Text ˆ for beginning of line (not to be confused with negation) Manipulation

Regular Expressions Expression: Overview Simple Example /ˆ\w+/ Character Groups Quantifiers Character Classes Input: Escaping Negating Though this be madness, yet there is method in it. Anchors Grouping Modifiers References Exercises Result: Advanced Text Though this be madness, yet there is method in it. Manipulation

Redirects and Loops

Bash Programming 48 / 143 Conditionals and Loops Regular Expressions Anchors

Acknowledgments Logging In $ for end of line Text Manipulation Expression: Regular Expressions /[\w]+[,\.!\?]+$/ Overview Simple Example Character Groups Input: Quantifiers Character Classes Though this be madness, yet there is method in it. Escaping Negating Anchors Grouping Modifiers References Result: Exercises Though this be madness, yet there is method in it. Advanced Text Manipulation

Redirects and Loops

Bash Programming 49 / 143 Conditionals and Loops Regular Expressions Anchors

Acknowledgments Logging In Can anchor at word boundaries with \b Text Manipulation Expression: Regular Expressions /\b\w+\b/ Overview Simple Example Character Groups Input: Quantifiers Character Classes Though this be madness, yet there is method in it. Escaping Negating Anchors Grouping Modifiers References Result: Exercises Though this be madness, yet there is method in it. Advanced Text Manipulation

Redirects and Loops

Bash Programming 50 / 143 Conditionals and Loops Regular Expressions Grouping

Acknowledgments Logging In Can create match groups with () Text Use | for logical or Manipulation

Regular Expressions Expression: Overview Simple Example /\b(is|in|it|be)\b/ Character Groups Quantifiers Character Classes Input: Escaping Negating Though this be madness, yet there is method in it. Anchors Grouping Modifiers References Exercises Result: Advanced Text Though this be madness, yet there is method in it. Manipulation

Redirects and Loops

Bash Programming 51 / 143 Conditionals and Loops Regular Expressions Grouping

Acknowledgments Logging In Can use quantifiers on groups Text Manipulation Expression: Regular Expressions /(\w+\s?)+/ Overview Simple Example Character Groups Input: Quantifiers Character Classes Though this be madness, yet there is method in it. Escaping Negating Anchors Grouping Modifiers References Result: Exercises Though this be madness, yet there is method in it. Advanced Text Manipulation

Redirects and Loops

Bash Programming 52 / 143 Conditionals and Loops Regular Expressions Modifiers

Acknowledgments Logging In There are several modifiers that can be applied to regular Text expressions: Manipulation

Regular • A single letter is specified after the expression Expressions Overview • Vary a bit from implementation to implementation, but Simple Example some common ones: Character Groups Quantifiers • g (global: returns ALL matches, implied on the previous Character Classes examples) Escaping Negating • i (case insensitive: shortcut for specify both cases) Anchors m (multi-line: the ˆ and anchor will match newlines - ie, Grouping • $ Modifiers enter key) References Exercises • Several other modifiers related to multi-line handling Advanced Text Examples: Manipulation

Redirects and /mad/g Loops /mad/i Bash Programming 53 / 143 Conditionals and Loops Regular Expressions References

Acknowledgments Logging In www.regular-expressions.info - Great resource for Text reference and everything you need to know about regular Manipulation expressions. Regular Expressions www.regex101.com - Great tool for testing your regular Overview Simple Example expressions in various different environments. Character Groups Quantifiers Character Classes Escaping Negating Anchors Grouping Modifiers References Exercises Advanced Text Manipulation

Redirects and Loops

Bash Programming 54 / 143 Conditionals and Loops Regular Expressions Exercises

Acknowledgments Logging In Lets try a few of these, using the live web regex tester Text Manipulation Open the web page: www.regex101.com Regular Expressions Overview Simple Example Print the sample text, highlight and copy to your clipboard: Character Groups Quantifiers $ cd ~/unix101/regex/ Character Classes $ cat hamlet_sample.txt Escaping Negating Anchors Grouping Modifiers References Exercises Advanced Text Manipulation

Redirects and Loops

Bash Programming 55 / 143 Conditionals and Loops Regular Expressions Exercises

Acknowledgments Logging In What do these regular expression do: Text Manipulation

Regular Expressions 1. /[Mm]ad/g Overview Simple Example Character 2. /mad/gi Groups Quantifiers Character 3. /\w+/g Classes Escaping 4. /\bmad\b/g Negating Anchors Grouping 5. /\w+./g Modifiers References 6. /[Ww](hat|hy)?/g Exercises Advanced Text Manipulation

Redirects and Loops

Bash Programming 56 / 143 Conditionals and Loops Regular Expressions Exercises

Acknowledgments Logging In Craft a regular expression to find every word at the end of a Text sentence: Manipulation

Regular Expressions This business is well ended.-- Overview Simple Example My liege, and madam,--to expostulate Character Groups What majesty should be, what duty is, Quantifiers Character Classes Why day is day, night is night, and time is time. Escaping Negating Were nothing but to waste night, day, and time. Anchors Grouping Therefore, since brevity is the soul of wit, Modifiers References And tediousness the limbs and outward flourishes, Exercises I will be brief:--your noble son is mad: Advanced Text Mad call I it; for to define true madness, Manipulation What is’t but to be nothing else but mad? Redirects and Loops But let that go. Bash Programming 57 / 143 Conditionals and Loops Regular Expressions Exercises

Acknowledgments Logging In Craft a regular expression to find every word at the beginning Text of a line, that starts with a W: Manipulation

Regular Expressions This business is well ended.-- Overview Simple Example My liege, and madam,--to expostulate Character Groups What majesty should be, what duty is, Quantifiers Character Classes Why day is day, night is night, and time is time. Escaping Negating Were nothing but to waste night, day, and time. Anchors Grouping Therefore, since brevity is the soul of wit, Modifiers References And tediousness the limbs and outward flourishes, Exercises I will be brief:--your noble son is mad: Advanced Text Mad call I it; for to define true madness, Manipulation What is’t but to be nothing else but mad? Redirects and Loops But let that go. Bash Programming 58 / 143 Conditionals and Loops Regular Expressions Exercises

Acknowledgments Logging In Craft a regular expression to find a two letter word followed by Text a 3 letter word: Manipulation

Regular Expressions This business is well ended.-- Overview Simple Example My liege, and madam,--to expostulate Character Groups What majesty should be, what duty is, Quantifiers Character Classes Why day is day, night is night, and time is time. Escaping Negating Were nothing but to waste night, day, and time. Anchors Grouping Therefore, since brevity is the soul of wit, Modifiers References And tediousness the limbs and outward flourishes, Exercises I will be brief:--your noble son is mad: Advanced Text Mad call I it; for to define true madness, Manipulation What is’t but to be nothing else but mad? Redirects and Loops But let that go. Bash Programming 59 / 143 Conditionals and Loops Regular Expressions Exercises

Acknowledgments Logging In /\w+[.?]/g Text /[A-Za-z]+[.?]/g Manipulation

Regular Expressions This business is well ended.-- Overview Simple Example My liege, and madam,--to expostulate Character Groups What majesty should be, what duty is, Quantifiers Character Classes Why day is day, night is night, and time is time. Escaping Negating Were nothing but to waste night, day, and time. Anchors Grouping Therefore, since brevity is the soul of wit, Modifiers References And tediousness the limbs and outward flourishes, Exercises I will be brief:--your noble son is mad: Advanced Text Mad call I it; for to define true madness, Manipulation What is’t but to be nothing else but mad? Redirects and Loops But let that go. Bash Programming 60 / 143 Conditionals and Loops Regular Expressions Exercises

Acknowledgments Logging In /ˆW\w+/gm Text /(ˆ|\n)W\w+/g Manipulation

Regular Expressions This business is well ended.-- Overview Simple Example My liege, and madam,--to expostulate Character Groups What majesty should be, what duty is, Quantifiers Character Classes Why day is day, night is night, and time is time. Escaping Negating Were nothing but to waste night, day, and time. Anchors Grouping Therefore, since brevity is the soul of wit, Modifiers References And tediousness the limbs and outward flourishes, Exercises I will be brief:--your noble son is mad: Advanced Text Mad call I it; for to define true madness, Manipulation What is’t but to be nothing else but mad? Redirects and Loops But let that go. Bash Programming 61 / 143 Conditionals and Loops Regular Expressions Exercises

Acknowledgments Logging In /\b\w{2}\s\w{3}\b/g Text /\[A-Za-z]{2}\s[A-Za-z]{3}\b/g Manipulation

Regular Expressions This business is well ended.-- Overview Simple Example My liege, and madam,--to expostulate Character Groups What majesty should be, what duty is, Quantifiers Character Classes Why day is day, night is night, and time is time. Escaping Negating Were nothing but to waste night, day, and time. Anchors Grouping Therefore, since brevity is the soul of wit, Modifiers References And tediousness the limbs and outward flourishes, Exercises I will be brief:--your noble son is mad: Advanced Text Mad call I it; for to define true madness, Manipulation What is’t but to be nothing else but mad? Redirects and Loops But let that go. Bash Programming 62 / 143 Conditionals and Loops Advanced Text Manipulation

Acknowledgments

Logging In

Text Manipulation

Regular Expressions Advanced Text Manipulation Advanced Text grep Manipulation grep awk awk sed sed Redirects and Loops

Bash Programming

Conditionals and Loops

63 / 143 Advanced Text Manipulation grep

Acknowledgments Logging In grep (globally search for regular expression and print) Text Manipulation General syntax: Regular Expressions grep [OPTIONS] PATTERN FILENAME Advanced Text Manipulation Typical scenarios: grep awk sed • Extract specific line(s) from the simulation output Redirects and • Strip header/footer/comments lines from an input file Loops

Bash • Select files of interest Programming Count number of occurrences of a pattern in a file Conditionals • and Loops

64 / 143 Advanced Text Manipulation grep

Acknowledgments

Logging In Useful options:

Text Option Meaning Manipulation -v inverts the match (finds lines NOT containing pattern) Regular --color colors the matched text for easy visualization Expressions -F interprets the pattern as literal string Advanced Text -E interprets the pattern as an extended regular Manipulation expressions (more powerful, friendlier syntax) grep awk -H, -h print, don’t print the matched filename sed -i ignore case for pattern matching Redirects and Loops -l lists the file names containing the pattern Bash -n prints the line number containing the pattern Programming -c counts the number of matches Conditionals and Loops -w forces the pattern to match an entire word -x forces patterns to match the whole line

65 / 143 Advanced Text Manipulation grep

Acknowledgments Logging In Move to Shakespeare directory: Text Manipulation $ cd ~/unix101/Shakespeare/

Regular Expressions Try these grep commands: Advanced Text 1. Search for the given string in a single file Manipulation grep Scene Hamlet.txt grep awk sed 2. Check for the given string in multiple files Redirects and grep Scene *.txt Loops

Bash 3. Highlight the search Programming grep --color Scene Hamlet.txt Conditionals and Loops 4. Case insensitive search grep -i Scene Hamlet.txt 5. Count the number of matches grep -c Scene Hamlet.txt

66 / 143 Advanced Text Manipulation grep

Acknowledgments Logging In More examples: Text Manipulation 6. Show line number while displaying the output

Regular grep -n Scene Hamlet.txt Expressions 7. Display only the file names which matches the given Advanced Text pattern Manipulation grep grep -l Scene *.txt awk sed 8. Search in all files recursively Redirects and Loops grep -r Scene * Bash 9. Check for full words, not for sub-strings Programming

Conditionals grep -w all *.txt and Loops 10. Invert match grep -v a Hamlet.txt

67 / 143 Advanced Text Manipulation grep

Acknowledgments Logging In Try these examples yourself using the Lear.txt file. Text Manipulation 1. Find the lines that contain the word Madam and highlight Regular the word. Expressions

Advanced 2. Find the lines that contain the phrase good sir in all cases. Text Manipulation 3. List the line number of the lines that contain the exact grep awk word sleep. sed Redirects and 4. Count the number of the lines that do not contain the Loops word thy. Bash Programming

Conditionals and Loops

68 / 143 Advanced Text Manipulation grep

Acknowledgments Logging In Regular expression examples using the system file Text /usr/share/dict/words. This is a file containing a list of Manipulation dictionary words and is installed on all Linux systems. Regular Expressions 1. Beginning of line (ˆ) or end of line ($) Advanced Text $ grep -w "^hall" /usr/share/dict/words Manipulation grep 2. Character group ([0-9][a-z][A-Z]) awk sed $ grep "gr[ae]y" /usr/share/dict/words Redirects and $ grep "qa[^u]" /usr/share/dict/words Loops $ grep "[0-9]th" /usr/share/dict/words Bash Programming $ grep "[0-9][0-9]th" /usr/share/dict/words

Conditionals and Loops

69 / 143 Advanced Text Manipulation grep

Acknowledgments Logging In More regular expression examples: Text 3. Wildcards (use the ”.” for a single character match) Manipulation

Regular $ grep "U.S" /usr/share/dict/words Expressions $ grep "U\.S" /usr/share/dict/words Advanced Escaping the dot (\) Text Manipulation 4. Quantifiers (?/*/+/{N}), grouping grep awk $ egrep "^a.t$" /usr/share/dict/words sed $ egrep "^a.?t " /usr/share/dict/words Redirects and $ Loops $ egrep "^a.*t$" /usr/share/dict/words Bash $ egrep "e{3}" /usr/share/dict/words Programming $ egrep "a{2,3}" /usr/share/dict/words Conditionals and Loops $ egrep "[ae]{2}" /usr/share/dict/words

70 / 143 Advanced Text Manipulation grep

Acknowledgments Logging In grep OR Text $ egrep "blue|green" /usr/share/dict/words Manipulation

Regular Expressions grep AND Advanced grep blue /usr/share/dict/words | grep green Text $ Manipulation grep awk grep vs egrep sed Redirects and Loops egrep is the same as grep -E. It interprets PATTERN as an Bash Programming extended regular expression.

Conditionals and Loops

71 / 143 Advanced Text Manipulation grep

Acknowledgments Logging In grep practice: Text Manipulation What would you expect to grep? Regular Expressions $ egrep "^[0-9]+-\w+$" /usr/share/dict/words Advanced Text Manipulation $ grep -i "^[^aeiou]" /usr/share/dict/words grep awk sed Redirects and Loops Select all lines starting with a lower case letter and ending in Bash Programming upper case letter in /usr/share/dict/words.

Conditionals and Loops Find the number of empty lines in the file Hamlet.txt?

72 / 143 Advanced Text Manipulation grep

Acknowledgments Logging In Change directories: Text cd ~/unix101/data/ Manipulation $

Regular Expressions Working with the grepdata.txt file: Advanced 1. Print all lines that contain CA in either uppercase or Text Manipulation lowercase. grep awk 2. Print all lines that contain an email address (they have an sed @ in them), preceded by the line number. Redirects and Loops 3. Print all lines that do not contain the word Sep. Bash (including the period). Programming 4. Print all lines that contain the word de as a whole word. Conditionals and Loops 5. Print all lines that contain a phone number with an extension (the letter x or X followed by four digits). 6. Print all lines that begin with 3 digits followed by a blank. 7. Print all lines that do not begin with a capital S.

73 / 143 Advanced Text Manipulation awk

Acknowledgments Logging In awk Text Manipulation A simple structured programming language. Powerful, yet Regular Expressions simple and convenient enough for processing text organized in Advanced lines and columns. Text Manipulation grep awk sed Redirects and Loops

Bash Programming

Conditionals and Loops

74 / 143 Advanced Text Manipulation awk

Acknowledgments Logging In Usage: Text awk [OPTION] '/PATTERN/ ACTIONS' FILENAME Manipulation awk [OPTION] -f PROGRAMFILE FILENAME Regular Expressions • PATTERN - a regular expression. Advanced Text Manipulation • ACTIONS - statement(s) to be performed. grep awk • several patterns and actions are possible in awk. sed Redirects and • FILENAME - input file. Loops

Bash Some special cases: Programming • No search pattern means ”apply to all lines” Conditionals and Loops • No actions means ”apply default action” (print the line) • An explicitly empty action ('{}') means ”do nothing”

75 / 143 Advanced Text Manipulation awk

Acknowledgments Logging In Steps: Text Manipulation 1. Read a line from the file into a variable named $0. Regular Expressions 2. Split up the fields. The first field is placed in variable $1,

Advanced the second in $2, and so forth. Use -F to tell what the Text Manipulation delimiter is. If you don’t give a delimiter, then fields are grep delimited by whitespace (space, tab). awk sed 3. Do whatever command or commands are in the braces ({ Redirects and Loops and }) Bash Programming 4. Lather, rinse, repeat.

Conditionals and Loops

76 / 143 Advanced Text Manipulation awk

Acknowledgments Logging In Example: Text Manipulation Adams, Ansel;photographer;1902-02-20;1984-04-22 Asimov, Isaac;author;1920-01-02;1992-04-06 Regular Expressions Janney, Allison;actress;1959-11-19 La Rue, Lash;actor;1917-06-15;1996-05-21 Advanced Sagan, Carl;astronomer/writer;1934-11-09;1996-12-20 Text Manipulation Sharif, Omar;actor;1932-04-10 grep awk sed By default fields are separated by whitespace:

Redirects and $1 $2 $3 Loops Adams, Ansel;photographer;1902-02-20;1984-04-22 Asimov, Isaac;author;1920-01-02;1992-04-06 Bash Janney, Allison;actress;1959-11-19 Programming La Rue, Lash;actor;1917-06-15;1996-05-21 Sagan, Carl;astronomer/writer;1934-11-09;1996-12-20 Conditionals Sharif, Omar;actor;1932-04-10 and Loops

77 / 143 Advanced Text Manipulation awk

Acknowledgments Logging In Example: Text Manipulation Adams, Ansel;photographer;1902-02-20;1984-04-22 Asimov, Isaac;author;1920-01-02;1992-04-06 Regular Expressions Janney, Allison;actress;1959-11-19 La Rue, Lash;actor;1917-06-15;1996-05-21 Advanced Sagan, Carl;astronomer/writer;1934-11-09;1996-12-20 Text Manipulation Sharif, Omar;actor;1932-04-10 grep awk sed Use -F';' to get a smarter separation of fields: Redirects and Loops $1 $2 $3 $4 Adams, Ansel photographer 1902-02-20 1984-04-22 Bash Programming Asimov, Isaac author 1920-01-02 1992-04-06 Janney, Allison actress 1959-11-19 Conditionals La Rue, Lash actor 1917-06-15 1996-05-21 and Loops Sagan, Carl astronomer/writer 1934-11-09 1996-12-20 Sharif, Omar actor 1932-04-10

78 / 143 Advanced Text Manipulation awk

Acknowledgments Logging In Example: Text Adams, Ansel;photographer;1902-02-20;1984-04-22 Manipulation Asimov, Isaac;author;1920-01-02;1992-04-06 Regular Janney, Allison;actress;1959-11-19 Expressions La Rue, Lash;actor;1917-06-15;1996-05-21 Advanced Sagan, Carl;astronomer/writer;1934-11-09;1996-12-20 Text Manipulation Sharif, Omar;actor;1932-04-10 grep awk Simple printing sed Redirects and awk -F';''{print $1, "was born", $3 "."}' people.txt Loops

Bash Programming NF - containing # of the field in the current line Conditionals awk -F';''{print $NF}' people.txt and Loops awk -F';''NF < 4 {print $1 " is alive and was born in " $3}' people.txt

NR - the row number being currently processed awk -F';''NR < 3 {print $1}' people.txt 79 / 143 Advanced Text Manipulation awk

Acknowledgments Logging In Example: Text Manipulation Adams, Ansel;photographer;1902-02-20;1984-04-22 Asimov, Isaac;author;1920-01-02;1992-04-06 Regular Expressions Janney, Allison;actress;1959-11-19 La Rue, Lash;actor;1917-06-15;1996-05-21 Advanced Sagan, Carl;astronomer/writer;1934-11-09;1996-12-20 Text Manipulation Sharif, Omar;actor;1932-04-10 grep awk sed Matching pattern Redirects and awk -F';''/Adams/{print $1, "was born", $3 "."}' people.txt Loops awk -F ; /^A.*s/{print 1, "was born", 3 "."} people.txt Bash ' '' $ $ ' Programming

Conditionals and Loops Matching pattern in a field awk -F';' '$3 ~ /193[0-9]/ {print $1, "was born", $3 "."}' people.txt

80 / 143 Advanced Text Manipulation awk

Acknowledgments Logging In Awk variables Text Manipulation • It’s a programming language, of course it has them! Regular • They can be used in either PATTERN or ACTION parts of Expressions the program. Advanced Text Manipulation • You can define your own. grep awk • Some are predefined for you and can be used to change sed program behavior (and some even change dynamically Redirects and Loops with each read line). Bash Programming

Conditionals and Loops

81 / 143 Advanced Text Manipulation awk

Acknowledgments Logging In Awk variables Text FS Field Separator (default ANY WHITESPACE) Manipulation OFS Output Field Separator (default SPACE) Regular Expressions NF Number of Fields in the current input record (line)

Advanced NR Number of Records (lines) in the input Text Manipulation FNR File Number of Records (in current file as grep awk opposed to all input) sed RS Record Separator (default NEWLINE) Redirects and Loops ORS Output Record Separator (default NEWLINE) Bash $N Nth field of the line where N can be any number Programming (eg. $0 = entire line, $1 = first field, Conditionals and Loops $2 = second field and so on). Expressions allowed: $(NF-3) IGNORECASE If not zero, regexp matching is case insensitive (default =0)

82 / 143 Advanced Text Manipulation awk

Acknowledgments Logging In Handy awk one-liners Text Manipulation awk 'NF>0 {print}' FILE Deletes all blank lines (by the book) awk 'NF>0' FILE Deletes all blank lines (simpler) Regular awk 'NF' FILE Deletes all blank lines (simplest) Expressions awk 'NF>4' FILE Prints all lines with more than 4 fields awk '$NF>4' FILE Prints all lines with value of the last field >4 (note the difference w/above!) Advanced awk 'END{print $NF}' FILE Prints value of the last field of the last line Text awk 'NR==25,NR==100' FILE Prints lines between 25 and 100 Manipulation awk 'END{print}' FILE Prints the last line of the file grep awk '$5==”abc123”' FILE Prints lines which have ’abc123’ in 5th field awk awk 'BEGIN{ORS=”\n\n”}; print' FILE Double spaces the file sed awk '{print $2,$1}' FILE Prints only 2nd and 1st fields (swapping columns) Redirects and awk '{$2=””; print}' FILE Prints the file without 2nd column Loops awk '/REGEX/' FILE Prints all the lines having REGEX awk '!/REGEX/' FILE Prints all the lines not having the REGEX Bash awk '/AAA|BBB|CCC/' FILE Prints all the lines having either AAA, BBB or CCC Programming awk 'length>50' FILE Prints line having more than 50 characters awk '/POINTA/,/POINTB/' FILE Prints section of file between POINTA and POINTB Conditionals and Loops

83 / 143 Advanced Text Manipulation awk

Acknowledgments Logging In Change directories and look at file: Text Manipulation $ cd ~/unix101/data/ $ cat awkdata.txt Regular Expressions Advanced 1. Print every line from the file. Text Manipulation grep 2. Print the fields that contain the name and salary. awk sed 3. Print the list of employees who has employee id greater Redirects and Loops than 200. Bash 4. Print the list of employees in Technology department. Programming

Conditionals and Loops

84 / 143 Advanced Text Manipulation awk

Acknowledgments Logging In Change directories and look at file: Text Manipulation $ cd ~/unix101/data/ $ cat awkdata.txt Regular Expressions

Advanced Print a report as below Text Manipulation Name Designation Department Salary grep Thomas Manager Sales $5,000 awk sed Jason Developer Technology $5,500 Sanjay Sysadmin Technology 7,000 Redirects and $ Loops Nisha Manager Marketing $9,500 Randy DBA Technology 6,000 Bash $ Programming Report Generated ------Conditionals and Loops

85 / 143 Advanced Text Manipulation sed

Acknowledgments Logging In sed -(stream editor) Text Manipulation • Reads one or more text files line by line, makes changes

Regular according to editing script, and writes the results to Expressions standard output. Advanced Text • Editing script can be defined to selectively Manipulation grep add/delete/modify fragments of text awk sed (paragraph/lines/words/characters) as needed. Redirects and Loops • Most commonly used to substitute (’s’) text matching a

Bash pattern: Programming sed [OPTIONS] 's/REGEXP/REPLACEMENT/FLAGS' FILENAME Conditionals sed [OPTIONS] ANCHOR s/REGEXP/REPLACEMENT/FLAGS FILENAME and Loops ' '

(ANCHOR can be another regexp or some line numbers)

86 / 143 Advanced Text Manipulation sed

Acknowledgments Logging In Change directories: Text Manipulation $ cd ~/unix101/Shakespeare/

Regular Expressions sed Examples: Advanced Text Manipulation sed 's/SCENE/Scene/' Othello.txt grep awk sed '33 s/SCENE/Scene/' Othello.txt sed sed /Castle/ s/SCENE/scene/ Othello.txt Redirects and ' ' Loops Bash See handout for more practical examples and links. Programming

Conditionals and Loops

87 / 143 Advanced Text Manipulation sed

Acknowledgments Logging In Exercises: Text Manipulation 1. What is the output on your screen of this command line: echo hi | sed -e s/HI/HO/ Regular ' ' Expressions a. ho Advanced b. hi Text Manipulation c. HO grep d. no output on screen awk sed e. HI Redirects and Loops 2. Which sed command finds every line that ends in the digits and removes the first occurrence of the string Bash 123 Programming xyzzy from those lines: Conditionals a. /[0-9][0-9][0-9]$/s/xyzzy// and Loops b. /xyzzy.*123$/123/ c. /123$/s/xyzzy// d. s/^.*xyzzy. ∗ 123$/\1/ e. /xyzzy/s/[0-9][0-9][0-9]\$//

88 / 143 Redirects and Loops

Acknowledgments

Logging In

Text Manipulation

Regular Expressions Redirects and Loops Advanced Text Redirects Manipulation Pipes Redirects and Loops For Loops Redirects Pipes For Loops Bash Programming

Conditionals and Loops

89 / 143 Redirects and Loops Redirects

Acknowledgments Logging In With every UNIX program three standard streams are created Text Manipulation • Standard output (stdout):

Regular Normal output, printed to your screen Expressions • Standard error (stderr): Advanced Text Error messages, printed to your screen Manipulation

Redirects and • Standard input (stdin): Loops Redirects File for command to read in as input Pipes For Loops Bash Change directories: Programming Conditionals $ cd ~/unix101/redirects/ and Loops

90 / 143 Redirects and Loops Redirects

Acknowledgments Logging In Using redirects and pipelines, we can redirect these streams Text elsewhere such as to a file or another command. Manipulation

Regular Expressions Why? Advanced Text • Your code or program spams your screen with a ton of Manipulation text and output. Rather than scrolling your screen for Redirects and Loops hours, we can send output to a file. With the output in a Redirects Pipes file, we can use one of the tools (or many others) we have For Loops talked about so far to search for interesting lines. Bash Programming • Send output of one command to another one for further Conditionals and Loops processing or refinement.

91 / 143 Redirects and Loops Redirects

Acknowledgments Logging In Change output: Text Manipulation COMMAND > FILE Regular Expressions Advanced Take output of a command and put it into FILE, rather than Text Manipulation print it on your screen. This overwrites FILE if it is already Redirects and present, so be careful! Loops Redirects Pipes For Loops Example: Bash Programming $ ls -l > out.log Conditionals $ cat out.log and Loops total 0 -rw-r--r-- 1 ddietz rcacsupp 16 Jan 24 13:07 file1.txt

92 / 143 Redirects and Loops Redirects

Acknowledgments Logging In Change input: Text Manipulation COMMAND < FILE Regular Expressions Advanced Take contents of FILE and feed it into a command. Some Text Manipulation commands, such as tr, cannot take a file name (like the Redirects and commands we have seen so far) as an argument so you must Loops Redirects feed it in by changing its standard input. Pipes For Loops Bash Programming Example: Conditionals $ cat file1.txt and Loops This is a file. $ tr i u < file1.txt Thus us a fule.

93 / 143 Redirects and Loops Redirects

Acknowledgments Logging In Change input and output: Text Manipulation COMMAND < FILE > OTHERFILE Regular Expressions Advanced Example: Text Manipulation $ tr i u < file1.txt > out.log Redirects and $ cat out.log Loops Redirects Thus us a fule. Pipes For Loops Bash Programming

Conditionals and Loops

94 / 143 Redirects and Loops Redirects

Acknowledgments Logging In Append to file rather than wipe out original: Text Manipulation COMMAND >> FILE Regular Expressions Advanced Example: Text Manipulation $ ls >> out.log Redirects and $ cat out.log Loops Redirects total 0 Pipes -rw-r--r-- 1 ddietz rcacsupp 16 Jan 24 13:07 file1.txt For Loops total 0 Bash -rw-r--r-- 1 ddietz rcacsupp 16 Jan 24 13:07 file1.txt Programming

Conditionals and Loops

95 / 143 Redirects and Loops Redirects

Acknowledgments Logging In Change standard error with 2> Text Manipulation $ ls -l notafile 2> error.log $ cat error.log Regular Expressions ls: cannot access notafile: No such file or directory

Advanced Text Here notafile is a file that does not exist. This is done on Manipulation purpose to force an error message so that redirection of Redirects and Loops standard error can be demonstrated. In real life, you probably Redirects Pipes aren’t going to have errors on purpose, but should they occur For Loops you may want the error messages saved into a separate file. Bash Programming

Conditionals and Loops

96 / 143 Redirects and Loops Redirects

Acknowledgments Logging In Let’s combine them: Text Manipulation $ ls notafile file1.txt >& out.log $ cat out.log Regular Expressions ls: cannot access notafile: No such file or directory file1.txt Advanced Text Manipulation We force an error by purposely requesting a non-existent file in Redirects and addition to standard output with a real file. This generates two Loops Redirects separate streams that we can direct into a single file (instead of Pipes For Loops printing both to your screen). Bash Programming

Conditionals If your program generates a ton of output, it may be helpful to and Loops put it into a file so that is easy to search through later.

97 / 143 Redirects and Loops Redirects

Acknowledgments Logging In Divide and conquer: Text Manipulation $ ls notafile file1.txt 2> error.log > out.log $ cat error.log Regular Expressions ls: cannot access notafile: No such file or directory $ cat out.log Advanced file1.txt Text Manipulation

Redirects and We force an error by purposely requesting a non-existent file in Loops Redirects addition to standard output with a real file. This generates two Pipes For Loops separate streams that we can direct into two separate files Bash (instead of printing both to your screen). Programming

Conditionals and Loops

98 / 143 Redirects and Loops Redirects

Acknowledgments Logging In We can throw away errors with /dev/null Text Manipulation ls -l * 2>/dev/null Regular Expressions Advanced We can throw away everything too Text Manipulation Redirects and ls -l * >& /dev/null Loops Redirects Pipes For Loops /dev/null is a special file on UNIX systems. Anything written is Bash Programming thrown away (permanently). Perhaps your program generates a

Conditionals ton of useless output. You could send the standard output into and Loops the garbage, while keeping only the error messages.

99 / 143 Redirects and Loops Pipes

Acknowledgments Logging In We can tell one to go into the same place as another: Text ls -l notafile file1.txt 2>&1 |less Manipulation

Regular Expressions Pipes will only send standard output into the next program. Advanced Normally any messages to standard error will be printed on Text Manipulation your screen. By combining error into out, we can pipe error Redirects and messages into the next program instead of your screen. Loops Redirects Pipes For Loops Bash Programming

Conditionals and Loops

100 / 143 Redirects and Loops Pipes

Acknowledgments Logging In What if we have a chatty program, want to save the output in Text a file for later viewing, but also want to monitor the progress of Manipulation the command in real-time? A special command called tee can Regular Expressions accomplish this. Advanced Text $ ls notafile file1.txt 2>&1 | tee out.log Manipulation ls: cannot access notafile: No such file or directory Redirects and file1.txt Loops $ cat out.log Redirects ls: cannot access notafile: No such file or directory Pipes For Loops file1.txt Bash Programming

Conditionals and Loops

101 / 143 Redirects and Loops For Loops

Acknowledgments Logging In We’ll discuss for loops more in bash programming, but they are Text useful even on the command line Manipulation

Regular $ for i in "one" "two" "three"; do echo $i; done Expressions one Advanced two Text three Manipulation

Redirects and Loops Redirects Pipes For Loops Bash Programming

Conditionals and Loops

102 / 143 Redirects and Loops For Loops

Acknowledgments Logging In Command substitution (we’ll discuss more later on): Text Manipulation $ cd ~/unix101/redirects/ $ mkdir backup Regular Expressions $ ls *.* error.log file1.txt out.log Advanced for i in ls *.* ; do cp " i" backup/; done Text $ ` ` $ Manipulation

Redirects and This example takes each item from ls *.*, and runs a Loops Redirects command(s) on each file. Here we are copying each file into Pipes For Loops the backup directory. Of course, this is very simplistic (you Bash could just do cp *.* backup/ but imagine you want to do Programming more complex operations on a list of files, and you don’t want Conditionals and Loops to type the same command a bunch of times.

Be very cautious of files with spaces in the name (don’t do it!) as for iterates by spaces (remember awk).

103 / 143 Bash Programming

Acknowledgments

Logging In

Text Manipulation Regular Bash Programming Expressions Shell basics Advanced Text Shell Types Manipulation Variables Redirects and Loops String Operations Bash Arithmetic Operations Programming Shell basics Command Substitution Shell Types Variables Quoting Characters String Operations Arithmetic Operations Command Substitution Quoting Characters Conditionals and Loops 104 / 143 Bash Programming Shell basics

Acknowledgments Logging In The first line of the shell script defines the program that Text interprets the script Manipulation

Regular #!/bin/bash Expressions Advanced End of a command using ; or a newline Text Manipulation #!/bin/bash Redirects and ls; pwd; Loops cd $HOME Bash ls Programming Shell basics Shell Types Variables String Operations Arithmetic Operations Command Substitution Quoting Characters Conditionals and Loops 105 / 143 Bash Programming Shell basics

Acknowledgments Logging In Make the script FILENAME executable Text Manipulation $ chmod +x FILENAME

Regular Expressions Execute a shell script script.sh in dir /path/to Advanced Text $ /path/to/script.sh Manipulation Redirects and Execute a shell script script.sh in your current working directory Loops

Bash $ ./script.sh Programming Shell basics Shell Types Variables String Operations Arithmetic Operations Command Substitution Quoting Characters Conditionals and Loops 106 / 143 Bash Programming Shell basics

Acknowledgments Logging In Exercise: Text Manipulation

Regular • Change to directory: Expressions $ cd ~/unix101/scripts Advanced Text Manipulation • Check the permission of the script Redirects and Loops $ ls -l script1 Bash $ -rwxr-xr-x 1 gandino student 72 Feb 15 2016 script1 Programming # should have x! Shell basics Shell Types Variables String Try different ways to run the scripts Operations • Arithmetic Operations $ $HOME/unix101/scripts/script1 Command Substitution $ ./script1 Quoting Characters Conditionals and Loops 107 / 143 Bash Programming Shell Types

Acknowledgments Logging In UNIX/Linux systems offer a variety of shell types Text Manipulation • bash (Bourne Again shell) Regular • csh or C Shell Expressions

Advanced • tcsh or TENEX C Shell Text Manipulation • sh or Bourne Shell Redirects and Loops Note: different shells have different syntax to do the same

Bash thing! Programming Shell basics $ echo $SHELL Shell Types /usr/local/bin/bash Variables String Operations Arithmetic Operations Command Substitution Quoting Characters Conditionals and Loops 108 / 143 Bash Programming Variables

Acknowledgments Logging In Variables may be of different value types: Text Manipulation • Bash does not force variables to have types Regular • Operations on variables depend on the content of the Expressions variables Advanced Text • Depending on contents, variables are of 4 types: Manipulation

Redirects and • String Loops • Integer Bash • Constant Programming Shell basics • Array Shell Types Variables String Operations Arithmetic Operations Command Substitution Quoting Characters Conditionals and Loops 109 / 143 Bash Programming Variables

Acknowledgments Logging In Create variable named VARNAME and set equal to value, and Text then print back the value of the variable: Manipulation

Regular $ VARNAME="value" Expressions $ echo $VARNAME Advanced Text Manipulation Dereference the variable VARNAME by placing a $ in front Redirects and Loops • No spaces around the = sign Bash • Variable names Programming Shell basics • Case sensitive Shell Types Variables • A combination of letters, numbers, and underscores; names String Operations starting with numbers are invalid Arithmetic Operations • Avoid using reserved words: if, else, fi, for Command Substitution • Avoid using environment variables: PATH, SHELL ... (see Quoting Characters printenv command) Conditionals and Loops 110 / 143 Bash Programming Variables

Acknowledgments Logging In Variables in Bash have scope, or variables in Bash are only Text accessible from specific environments: Manipulation

Regular • The variable created in your shell is only available to the Expressions current shell (the one you are typing in, and only the one Advanced Text you are typing in) Manipulation • Child processes of the current shell (such as a script you Redirects and Loops are trying execute) will not see this variable Bash Programming • To pass variables to subshells or scripts, we need to export Shell basics Shell Types variables: Variables String export VARNAME="value" Operations $ Arithmetic Operations Command Substitution Quoting Characters Conditionals and Loops 111 / 143 Bash Programming Variables

Acknowledgments Logging In Exercises: Text Manipulation • Check your current shell type and make sure it is bash Regular Expressions $ echo $SHELL

Advanced Text Manipulation • Create a integer variable Redirects and $ INT1=765 Loops $ echo $INT1 Bash Programming Shell basics • Create a string variable Shell Types Variables String $ STR1="Hello World" Operations echo STR1 Arithmetic $ $ Operations Command Substitution Quoting Characters Conditionals and Loops 112 / 143 Bash Programming Variables

Acknowledgments Logging In Arrays allow you to store a list of values inside a single variable: Text Manipulation • An array variable contains multiple values, index starts Regular from 0 Expressions

Advanced • Array declaration Text Manipulation $ declare -a MYARRAY Redirects and $ MYARRAY=(value1 value2) Loops Bash Programming • declare -a MYARRAY declares MYARRAY as an array Shell basics variable, with no initial values Shell Types Variables String • MYARRAY=(value1 value2) assigns values to the array Operations Arithmetic Operations Command Substitution Quoting Characters Conditionals and Loops 113 / 143 Bash Programming Variables

Acknowledgments Logging In Once defined, elements of an array can be accessed in several Text ways: Manipulation

Regular • Array elements reference Expressions $ MYARRAY=(value1 value2 value3) Advanced Text $ echo ${MYARRAY[*]} Manipulation $ echo ${MYARRAY[0]} Redirects and $ echo ${MYARRAY} Loops Bash { } refers to the whole array MYARRAY Programming • $ MYARRAY[*] Shell basics Shell Types • ${MYARRAY[0]} refers to the first element of MYARRAY Variables String { } also refers to the first element of MYARRAY Operations • $ MYARRAY Arithmetic Operations Command Substitution Quoting Characters Conditionals and Loops 114 / 143 Bash Programming Variables

Acknowledgments Logging In Individual elements of an array can be redefined at any time: Text Manipulation • Assign value to an array element Regular $ MYARRAY=(value1 value2 value3) Expressions $ echo ${MYARRAY[0]} Advanced value1 Text MYARRAY[0]=newval1 Manipulation $ $ echo ${MYARRAY[0]} Redirects and newval1 Loops

Bash Programming • MYARRAY[index]=val assigns val to the element Shell basics Shell Types MYARRAY[index]. val can be of any type such as a string Variables String or number. Operations Arithmetic Operations Command Substitution Quoting Characters Conditionals and Loops 115 / 143 Bash Programming Variables

Acknowledgments Logging In Exercise; create an array: Text • Name: ARRAY1 Manipulation • 1st element is ”hello” Regular Expressions • 2nd element is 10

Advanced • 3rd element is 48 Text 4th element is 20 Manipulation • • 5th element is ”world” Redirects and Loops #Way 1: Bash $ declare -a ARRAY1 Programming ARRAY1[0]="hello" Shell basics $ Shell Types $ ARRAY1[1]=10 Variables ... String Operations $ echo ${ARRAY1[*]} Arithmetic Operations Command #Way 2: Substitution Quoting $ ARRAY1=("hello" 10 48 20 "world") Characters $ echo ${ARRAY1[*]} Conditionals and Loops 116 / 143 Bash Programming Variables

Acknowledgments Logging In There are some variables that Bash predefines for you: Text Manipulation • These variables can only be referenced Regular • $0, $1, $2 ...: positional parameters Expressions • $0: the name of the executable as it was called Advanced Text • $1: first command line argument that you gave to the Manipulation executable Redirects and • 2: second command line argument Loops $ • #: number of command line parameters Bash $ Programming Shell basics • Example: positional.sh Shell Types Variables String Operations Arithmetic Operations Command Substitution Quoting Characters Conditionals and Loops 117 / 143 Bash Programming Variables

Acknowledgments Logging In Exercise: Text Manipulation cat positional.sh Execute positional.sh Regular $ • #!/bin/bash Expressions • Execute positional.sh: Advanced # positional.sh ./positional.sh hello world Text # This script reads first 3 positional parameters Manipulation and prints them out. • Execute positional.sh: ./positional.sh "hello world" Redirects and echo Loops echo "Name of the script being executed is $0" • Execute positional.sh with 3 echo parameters Bash echo "$1 is the first positional parameter, \$1" Programming echo "$2 is the second positional parameter, \$2" ./positional.sh "hello world 10" Shell basics echo "$3 is the thrid positional parameter, \$3" 20 30 Shell Types echo Variables echo "The total number of positional parameters is • Execute positional.sh with 5 String $#." Operations parameters Arithmetic ./positional.sh hello world 10 Operations Command 20 30 Substitution Quoting • Bonus: echo $0 Characters Conditionals and Loops 118 / 143 Bash Programming String Operations

Acknowledgments Logging In Length of a variable ${#VAR} Text Manipulation $ echo $SHELL /usr/local/bin/bash Regular Expressions $ echo ${#SHELL} 19 Advanced Text Manipulation String concatenation STR="$STR1$STR2" Redirects and Loops $ str1="Hello" Bash $ str2="World" Programming $ str="$str1 $str2" Shell basics $ echo "$str" Shell Types Variables Hello World String Operations Arithmetic Operations Command Substitution Quoting Characters Conditionals and Loops 119 / 143 Bash Programming String Operations

Acknowledgments Logging In Substring extraction ${VAR:OFFSET:LENGTH} Text Manipulation • OFFSET: the index of the character the substring starts

Regular from Expressions • OFFSET starts from 0 Advanced Text Manipulation • LENGTH: the number of characters to keep in the substring Redirects and • When LENGTH is omitted, the reminder of the string is Loops taken Bash Programming Shell basics $ MYSTRING="thisisaverylongname" Shell Types echo {MYSTRING:4} Variables $ $ String isaverylongname Operations Arithmetic Operations $ echo ${MYSTRING:6:5} Command Substitution avery Quoting Characters Conditionals and Loops 120 / 143 Bash Programming String Operations

Acknowledgments Logging In Exercise; create a string variable: Text • Name: STR Manipulation • Value: ”Welcome to Research Computing” Regular Expressions Create a substring of the STR variable: Advanced Text • Name: SUB1 Manipulation • Value: ”to Research” Redirects and Loops Create a substring of STR variable:

Bash • Name: SUB2: Programming Shell basics • Value: ”Com” Shell Types Variables $ STR="Welcome to Research Computing" String Operations $ echo $STR Arithmetic Welcome to Research Computing Operations Command $ SUB1=${STR:8:11} Substitution to Research Quoting Characters $ SUB2=${STR:20:3} Conditionals Com and Loops 121 / 143 Bash Programming Arithmetic Operations

Acknowledgments Logging In Bash allows for simple integer arithmetic: Text Manipulation (( EXPRESSION )) or let VAR=EXPRESSION Regular Expressions Spaces around EXPRESSION do not matter. Dereferencing Advanced variables in EXPRESSION is optional. There is no overflow Text Manipulation checking, except for division by 0. Redirects and Loops $ x=1 $ y=$((x+2)) Bash Programming $ echo $y Shell basics 3 Shell Types Variables String $ y=$(($x+2)) Operations $ echo $y Arithmetic Operations 3 Command Substitution Quoting $ let y=$x+2 Characters $ echo $y Conditionals 3 and Loops 122 / 143 Bash Programming Arithmetic Operations

Acknowledgments

Logging In Exercise: Text Manipulation • Create a variable named X and assign the value 10. Regular • Create a variable named Y and assign the value 3*X with (( )). Expressions Advanced • Create a variable named Z and assign the value as X*Y with let. Text Manipulation • Create a variable named W and assign the value as X+Z. Redirects and Loops

Bash Programming Shell basics Shell Types Variables String Operations Arithmetic Operations Command Substitution Quoting Characters Conditionals and Loops 123 / 143 Bash Programming Arithmetic Operations

Acknowledgments

Logging In Exercise: Text • Create a variable named X and assign the value 10. Manipulation

Regular • Create a variable named Y and assign the value 3*X with (( )). Expressions Create a variable named Z and assign the value as X*Y with let. Advanced • Text Manipulation • Create a variable named W and assign the value as X+Z. Redirects and Answers: Loops

Bash $ X=10 Programming $ Y=$((X*3)) Shell basics echo Y Shell Types $ $ Variables 30 String let Z= X* Y Operations $ $ $ Arithmetic $ echo $Z Operations Command 300 Substitution $ W=$(($X+$Z)) Quoting Characters $ echo $W Conditionals 310 and Loops 124 / 143 Bash Programming Command Substitution

Acknowledgments Logging In Command substitution: Text Manipulation • Commands between backticks `` are replaced by the

Regular output of the command, minus the trailing newline Expressions characters Advanced Text • variable= (command), saving the output of a command Manipulation $ into a variable Redirects and Loops

Bash $ date Programming Wed Feb 24 14:11:45 EST 2016 Shell basics Shell Types Variables $ x=`date` String echo x Operations $ $ Arithmetic Wed Feb 24 14:12:10 EST 2016 Operations Command Substitution $ x=$(date) Quoting Characters $ echo $x Conditionals Wed Feb 24 14:12:25 EST 2016 and Loops 125 / 143 Bash Programming Quoting Characters

Acknowledgments Logging In Escape characters remove the special meaning of a single Text character that follows. Bash uses a non-quoted backslash \ as Manipulation the escape character. Regular Expressions

Advanced Text Manipulation Example: using \ to remove the special meaning of $ Redirects and (dereference the variable year): Loops Bash $ year=2016 Programming $ echo $year Shell basics Shell Types 2016 Variables String Operations $ echo \$year Arithmetic year Operations $ Command Substitution Quoting Characters Conditionals and Loops 126 / 143 Bash Programming Quoting Characters

Acknowledgments Logging In Double quotes ” ” preserve the literal value of each character Text enclosed with the quotes, except for $, backticks ` `, and Manipulation backslash \ Regular Expressions A ” may occur between ” ”, by preceding it with \ Advanced and ` ` retain their special meaning within double quotes Text $ Manipulation $ year=2016 Redirects and $ echo "$year" Loops 2016 Bash Programming echo date Shell basics $ ` ` Shell Types Wed Feb 24 14:11:45 EST 2016 Variables $ echo "`date`" String Operations Wed Feb 24 14:12:10 EST 2016 Arithmetic Operations Command $ echo "\\" Substitution Quoting \ Characters Conditionals and Loops 127 / 143 Bash Programming Quoting Characters

Acknowledgments Logging In Single quotes ’ ’ preserve the literal value of each character Text enclosed with the quotes. Manipulation A ’ may not occur between ’ ’, even when preceded by \ Regular Expressions $ year=2016 Advanced $ echo $year Text Manipulation $ 2016

Redirects and Loops $ echo '$year' $year Bash Programming Shell basics Shell Types Variables String Operations Arithmetic Operations Command Substitution Quoting Characters Conditionals and Loops 128 / 143 Bash Programming Quoting Characters

Acknowledgments Logging In Exercise: Text Manipulation • Test the difference of single and double quotes in your

Regular terminal Expressions $ STR1="Hello World" Advanced Text $ LSTR1="MORE $STR1" Manipulation $ LSTR2='MORE $STR1' Redirects and $ echo $LSTR1 Loops MORE Hello World echo " LSTR1" Bash $ $ Programming MORE Hello World Shell basics $ echo "$LSTR2" Shell Types MORE STR1 Variables $ String Operations Arithmetic Operations Command Substitution Quoting Characters Conditionals and Loops 129 / 143 Bash Programming Quoting Characters

Acknowledgments Logging In Exercise: Text Manipulation • Test command substitution code in your terminal Regular $ SERVERNAME=$(hostname) Expressions $ echo "Running command on $SERVERNAME...." Advanced $ right_now=$(date +"%x %r %Z") Text time_stamp="Updated on right_now by USER" Manipulation $ $ $ $ echo "$time_stamp" Redirects and Loops

Bash • Use backticks in the command substitution code Programming Shell basics $ SERVERNAME=`hostname` Shell Types Variables $ echo "Running command on $SERVERNAME...." String $ right_now=`date +"%x %r %Z"` Operations Arithmetic $ time_stamp="Updated on $right_now by $USER" Operations echo " time_stamp" Command $ $ Substitution Quoting Characters Conditionals and Loops 130 / 143 Conditionals and Loops

Acknowledgments

Logging In

Text Manipulation

Regular Expressions Conditionals and Loops Advanced Text test [ ] [[ ]] Manipulation if; then; elif; else; fi Redirects and Loops For loops (slight return)

Bash break continue Programming

Conditionals and Loops test [ ] [[ ]] if; then; elif; else; fi For loops (slight return) break continue

131 / 143 Conditionals and Loops test [ ] [[ ]]

Acknowledgments Logging In Is the expression True or False? Text Manipulation String Comparisons Regular Expressions [[ -n string ]] Is string non-zero length Advanced [[ -z string ]] Is string zero length Text Manipulation [[ string1 = string2 ]] Equal Redirects and [[ string1 != string2 ]] Not equal Loops [[ string1 > string2 ]] Sorts after Bash Programming [[ string1 < string2 ]] Sorts before Conditionals and Loops test [ ] [[ ]] if; then; elif; else; fi For loops (slight return) break continue

132 / 143 Conditionals and Loops test [ ] [[ ]]

Acknowledgments Logging In Is the expression True or False? Text Manipulation Numeric Comparisons – Note the operator syntax! Regular Expressions [[ int1 -eq int2 ]] Equal Advanced [[ int1 -ne int2 ]] Not equal Text Manipulation [[ int1 -lt int2 ]] Less than Redirects and [[ int1 -gt int2 ]] Greater than Loops [[ int1 -le int2 ]] Less than or equal Bash Programming [[ int1 -ge int2 ]] Greater than or equal Conditionals and Loops test [ ] [[ ]] if; then; elif; else; fi For loops (slight return) break continue

133 / 143 Conditionals and Loops test [ ] [[ ]]

Acknowledgments Logging In Is the expression True or False? Text Manipulation File and directory conditions Regular Expressions [[ -d string ]] Is string the name of a directory? Advanced [[ -f string ]] Is string the name of a file? Text Manipulation [[ -r string ]] Is string the name of a readable file? Redirects and [[ -w string ]] Is string the name of a writable file? Loops [[ -x string ]] Is string the name of an executable file? Bash Programming [[ -s string ]] Is string a file with non-zero size? Conditionals and Loops test [ ] [[ ]] if; then; elif; else; fi For loops (slight return) break continue

134 / 143 Conditionals and Loops if; then; elif; else; fi

Acknowledgments Logging In if condition; then do-this; fi Text if condition; then do-this; else do-that;fi Manipulation

Regular Expressions Advanced if condition Text Manipulation then do-this Redirects and elif other-condition Loops

Bash then do-that Programming else do-other-thing Conditionals and Loops fi test [ ] [[ ]] if; then; elif; else; fi For loops (slight return) break continue

135 / 143 Conditionals and Loops if; then; elif; else; fi

Acknowledgments Logging In Condition can be result of [[ ]] Text Manipulation

Regular Expressions #!/bin/bash Advanced if [[ 3 -eq 4 ]] Text then Manipulation echo "Not Really" Redirects and else Loops echo "Math does work" Bash fi Programming Conditionals Condition can also be success or failure of program. and Loops test [ ] [[ ]] #!/bin/bash if; then; elif; else; fi if grep -q "Laertes" Hamlet.txt For loops (slight return) then break continue echo "yes it's there" else echo "no Laertes in sight" fi 136 / 143 Conditionals and Loops For loops (slight return)

Acknowledgments

Logging In #!/bin/bash Text Manipulation for arg in list Regular do Expressions Commands ... Advanced More commands ... Text if [[ $MOOD = "I feel like it" ]] Manipulation then Redirects and Compound commands ... Loops fi Bash And other stuff ... Programming done Conditionals and Loops test [ ] [[ ]] if; then; elif; else; fi For loops (slight return) break continue

137 / 143 Conditionals and Loops For loops (slight return)

Acknowledgments Logging In Examples: Text Manipulation #!/bin/bash for name in "Bill" "Joe" "Mary" Regular Expressions do echo "Hi there, $name" Advanced done Text Manipulation

Redirects and Loops

Bash Programming

Conditionals and Loops test [ ] [[ ]] if; then; elif; else; fi For loops (slight return) break continue

138 / 143 Conditionals and Loops For loops (slight return)

Acknowledgments Logging In Examples: Text List can be a variable: Manipulation

Regular #!/bin/bash Expressions

Advanced FILES="Hamlet.txt Lear.txt" Text SEARCH="Laertes" Manipulation Redirects and for filename in $FILES Loops do Bash if grep -q $SEARCH $filename Programming then Conditionals echo "Found \"$SEARCH\" in $filename" and Loops else test [ ] [[ ]] echo "No \"$SEARCH\" in $filename" if; then; elif; else; fi fi For loops (slight return) done break continue

139 / 143 Conditionals and Loops For loops (slight return)

Acknowledgments Logging In Examples: Text List can be result of command – THIS IS USEFUL!! Manipulation

Regular #!/bin/bash Expressions

Advanced SEARCH="Laertes" Text Manipulation for filename in $(ls *.txt) Redirects and do Loops if grep -q "$SEARCH" $filename Bash then Programming echo "Found \"$SEARCH\" in $filename" Conditionals else and Loops echo "No \"$SEARCH\" in $filename" test [ ] [[ ]] fi if; then; elif; else; fi done For loops (slight return) break continue

140 / 143 Conditionals and Loops break continue

Acknowledgments Logging In break completely quits a loop. Text Manipulation $ for i in {1..25};do if [[ $i -eq 12 ]];then break;fi;echo $i;done

Regular Expressions With better style: Advanced Text $ for i in {1..25}; do Manipulation > if [[ $i -eq 12 ]] Redirects and > then break Loops > fi

Bash > echo $i Programming > done

Conditionals and Loops Notice that semicolon in the second example? Why is it there? test [ ] [[ ]] if; then; elif; else; fi For loops (slight return) break continue

141 / 143 Conditionals and Loops break continue

Acknowledgments Logging In continue skips to the next item. Text Manipulation #!/bin/bash

Regular Expressions for i in {1..25} do Advanced if [[ i -eq 12 ]] Text $ Manipulation then continue fi Redirects and Loops echo $i Bash Programming done

Conditionals and Loops test [ ] [[ ]] if; then; elif; else; fi For loops (slight return) break continue

142 / 143 Conditionals and Loops break continue

Acknowledgments Logging In Exercises: change directory to the Shakespeare directory Text Manipulation 1. Print only the names of files whose name (including any

Regular extension) is longer than 8 characters. Expressions 2. Print only the names of files which are longer than 1000 Advanced Text bytes. Manipulation

Redirects and Extra credit Take-Home (there is no credit, sorry) Loops ((man and man -k are your friends)) Bash Programming Conditionals • Find the directory in your PATH variable that contains and Loops $ test [ ] [[ ]] the largest number of files, and print the directory name if; then; elif; else; fi and number of files it contains. For loops (slight return) break continue

143 / 143