Unix 201 Purdue University - ITaP Gladys Andino Dan Dietz Jieyu Gao Lev Gorenstein Erik Gough Stephen Harrell Randy Herban Steve Kelley Boyu Zhang Xiao Zhu [email protected] February 6th and 8th, 2018 Slides available: https://www.rcac.purdue.edu/training/unix201/ Acknowledgments Acknowledgments Logging In Text Manipulation Regular Expressions Advanced Text Acknowledgments Manipulation Redirects and Loops Bash Programming Conditionals and Loops 2 / 143 Acknowledgments Acknowledgments Logging In A few additional acknowledgments to the many people who Text have helped make this workshop possible. Manipulation Regular • The material in this workshop was prepared by the Purdue Expressions University ITaP Research Computing team. Advanced Text • Special thanks to Eric Adams and Megan Dale for Manipulation organizing the workshop sessions. Redirects and Loops • We have drawn from documentation provided by the Bash Programming Purdue Bioinformatics Core used in the UNIX for Conditionals Biologists workshop and Next-generation Transcriptome and Loops Analysis Workshop Manual provided by Professor Michael Gribskov and Professor Esperanza Torres. 3 / 143 Logging In Acknowledgments Logging In Windows Mac Activity Files Text Manipulation Logging In Regular Expressions Windows Advanced Mac Text Manipulation Activity Files Redirects and Loops Bash Programming Conditionals and Loops 4 / 143 Logging In Acknowledgments Logging In We will be using the Radon cluster: Windows Mac • www.rcac.purdue.edu/compute/radon/ Activity Files Text • Everyone has been given an account on the cluster for the Manipulation duration of the workshop Regular Expressions • If you wish to continue using Radon or other cluster after Advanced Text the workshop concludes, please make a request under your Manipulation advisor or PI's name: Redirects and Loops https://www.rcac.purdue.edu/account/request/ Bash Programming Conditionals and Loops 5 / 143 Logging In Windows Acknowledgments Logging In Many clients are available for Windows: Windows Mac • We will use the PuTTY SSH client Activity Files Text • Download PuTTY, no install required Manipulation Regular • http://www.chiark.greenend.org.uk/ sgtatham/putty/- Expressions download.html Advanced Text (or Google search putty) Manipulation • Download putty.exe for Intel x86 to your desktop Redirects and Loops Bash Programming Conditionals and Loops 6 / 143 Logging In Windows Acknowledgments Logging In Host Name for Radon is radon.rcac.purdue.edu Windows Mac Activity Files Text Manipulation Regular Expressions Advanced Text Manipulation Redirects and Loops Bash Programming Conditionals and Loops 7 / 143 Logging In Windows Acknowledgments Logging In One tweak: enable system colors in Appearance ! Colours Windows Mac Activity Files Text Manipulation Regular Expressions Advanced Text Manipulation Redirects and Loops Bash Programming Conditionals and Loops 8 / 143 Logging In Mac Acknowledgments Logging In Connect using: Windows Mac ssh [email protected] Activity Files Text Manipulation Regular Expressions Advanced Text Manipulation Redirects and Loops Bash Programming Conditionals and Loops 9 / 143 Logging In Mac Acknowledgments Logging In Linux also has a built in terminal client, similar to Mac: Windows Mac ssh [email protected] Activity Files Text Manipulation Regular Expressions Advanced Text Manipulation Redirects and Loops Bash Programming Conditionals and Loops 10 / 143 Logging In Activity Files Acknowledgments Logging In We'll need a few files for some of the hands-on activities Windows Mac cd Activity Files $ $ cp -r /depot/itap/unix101 . Text Manipulation Regular Expressions Advanced Text Manipulation Redirects and Loops Bash Programming Conditionals and Loops 11 / 143 Text Manipulation Acknowledgments Logging In Text Manipulation wc cut sort Text Manipulation uniq Exercises wc Regular cut Expressions sort Advanced Text uniq Manipulation Exercises Redirects and Loops Bash Programming Conditionals and Loops 12 / 143 Text Manipulation wc Acknowledgments Logging In The wc (word count) command simply counts the number of Text lines, words, and characters. Manipulation wc cut sort General syntax: uniq Exercises wc [OPTIONS] FILENAME Regular Expressions include: Advanced OPTIONS Text Manipulation • -l count lines only Redirects and • -w count words only Loops Bash • -c count characters only Programming Conditionals and Loops 13 / 143 Text Manipulation wc Acknowledgments Logging In Try this: Text Manipulation $ cd ~/unix101/Shakespeare wc $ cat wcdemo.txt cut This is just a very simple sort uniq text file that we'll use Exercises to demonstrate wc Regular Expressions $ wc wcdemo.txt Advanced 3 14 70 wcdemo.txt Text Manipulation This tells us that the file wcdemo.txt has: Redirects and Loops • 3 lines Bash Programming • 14 words Conditionals and Loops • 70 characters The we'll in the file looks like one word to Unix text processing commands. 14 / 143 Text Manipulation cut Acknowledgments Logging In The cut command is used to select sections of each line of a Text file or files. Manipulation wc cut sort General syntax: uniq Exercises cut [OPTIONS] FILENAME Regular Expressions include: Advanced OPTIONS Text Manipulation • -d specify a character instead of TAB for field delimiter Redirects and • -f select only these fields; also print any line that contains Loops Bash no delimiter character Programming Conditionals and Loops 15 / 143 Text Manipulation cut Acknowledgments Logging In Try this: Text Manipulation $ cd ~/unix101/protein wc cut $ head -n 5 1UBQ.pdb sort uniq HEADER CHROMOSOMAL PROTEIN 02-JAN-87 1UBQ Exercises TITLE STRUCTURE OF UBIQUITIN REFINED AT 1.8 ANGSTROMS RESOLUTION Regular COMPND MOL_ID: 1; Expressions COMPND 2 MOLECULE: UBIQUITIN; Advanced COMPND 3 CHAIN: A; Text Manipulation $ cut -f1 -d' ' 1UBQ.pdb | head -n 5 Redirects and HEADER Loops TITLE Bash COMPND Programming COMPND Conditionals COMPND and Loops 16 / 143 Text Manipulation sort Acknowledgments Logging In The sort command is used to sort lines of a text file. Text Manipulation wc General syntax: cut sort sort [OPTIONS] FILENAME uniq Exercises Regular OPTIONS include: Expressions Advanced • -n compare according to numerical value. Text Manipulation • -r reverse the result of comparisons. Redirects and Loops • -u return only unique lines. Bash Notes: Programming Conditionals • By default, lines are sorted alphabetically. and Loops • By default, lines starting with numbers are not sorted numerically. For example, "8 9 10 11" would be sorted as "10 11 8 9". 17 / 143 Text Manipulation sort Acknowledgments Logging In Try sort on words and num.txt: Text cd ~/unix101/Shakespeare Manipulation $ wc $ sort words_and_num.txt cut sort sort sort -n sort -r uniq $ $ $ Exercises 1 ana zoo Regular 11 MAX MAX Expressions 23 zoo ana Advanced 3 1 7 Text 5 3 5 Manipulation 7 5 3 Redirects and ana 7 23 Loops MAX 11 11 Bash zoo 23 1 Programming Count unique values in first column: Conditionals and Loops $ cd ~/unix101/protein $ cut -f1 -d'' 1UBQ.pdb | sort -u | wc -l 27 18 / 143 Text Manipulation uniq Acknowledgments Logging In The uniq command simply takes a sorted file and outputs the Text unique lines in it. The input must be sorted first. Manipulation wc cut sort General syntax: uniq Exercises uniq [OPTIONS] INPUT Regular Expressions include: Advanced OPTIONS Text Manipulation • -c count how many times each line occurred. Redirects and • -d only print duplicated lines. Loops Bash Programming Conditionals and Loops 19 / 143 Text Manipulation uniq Acknowledgments Logging In Try this: Text Manipulation $ cd ~/unix101/Shakespeare wc cut $ sort HamletWords.txt | uniq -c | head -n 5 sort uniq 36 1 Exercises 12 2 Regular 531 a Expressions 3 'a Advanced 1 abate Text $ sort HamletWords.txt | uniq -c | head -n 5 | sort -n Manipulation 1 abate Redirects and 3 'a Loops 12 2 Bash 36 1 Programming 531 a Conditionals and Loops 20 / 143 Text Manipulation uniq Acknowledgments Logging In Try this: Text Manipulation $ cd ~/unix101/Shakespeare wc cut $ sort HamletWords.txt | uniq -u | head -n 5 sort uniq abate Exercises abatements Regular abhorred Expressions ability Advanced Able Text $ sort HamletWords.txt | uniq -u > uniques Manipulation $ cat uniques Redirects and $ sort HamletWords.txt | uniq -u | wc -l Loops 3145 Bash Programming Conditionals and Loops 21 / 143 Text Manipulation Exercises Acknowledgments Logging In Try the following command sequence: Text 1. Change directory to Manipulation ~/unix101/data wc cut 2. Using a single line command, "ls" all the files in this sort uniq directory and sort alphabetically then "ls -l" and sort Exercises Regular 3. Find out how many times "TAIR00" and "TAIR10" Expressions appear in the file at genes.txt Advanced Text 4. Using a single line command find out how many unique Manipulation descriptions appear for column 3 in the "at genes.txt", Redirects and Loops please perform the search in a numerical order Bash Programming 5. Display 1st, 4th and 5th column of the "at genes.txt" Conditionals file, sorted in ascending order according to second field and Loops 22 / 143 Text Manipulation Exercises Acknowledgments Logging In Answers: Text 1. Change directory to ~/unix101/data Manipulation wc cut $ cd ~/unix101/data sort $ pwd uniq /home/gandino/unix101/data Exercises Regular Expressions 2. Using a single line command, "ls" all the files in this Advanced directory and sort alphabetically then "ls -l" and sort Text Manipulation $ ls | sort Redirects and at_genes.txt Loops awkdata.txt Bash grepdata.txt Programming Conditionals