Introduction to Linux/Unix

Xiaoge Wang, ICER [email protected] Feb. 4, 2016 How does this class work

• We are going to cover some basics with hands on examples. • Exercises are denoted by the following icon:

• Bold means commands which I expect you type on your terminal in most cases. Green and red sticky

Use the scky notes provided to help me help you. – No scky = I am working – Green = I am done and ready to move on (yea!) – Red = I am stuck and need more me and/or some help Agenda

• Introducon • Linux – Commands • Navigaon • Get or create files • Organizing files • Closer look into files • Search files • Distribute files • File permission • Learn new commands – Scripts • Pipeline • Make you own command • Environment of a shell • Summary Agenda

• Linux – Commands • Navigaon • Get or create files • Organizing files • Closer look into files • Search files • Distribute files • File permission • Learn new commands – Scripts • Pipeline • Make you own command • Environment of a shell • Summary Introduction

• Get ready for adventure? – Ticket(account)? • Big map – Linux/Unix – Shell • Overview of the trail – Commands – Simple Shell script – Get ready for HPC. Exercise 0: Get ready

• Connect to HPCC (gateway) – $ ssh [email protected] • Windows users MobaXterm • Mac users Terminal • Linux users? • Read important message – $ mod • Go to a development node – $ ssh dev-nodename Message of the day (mod)

• Mac Show screen Big picture

Shell Shell Big picture

• Shell – CLI ✔ – GUI

OS shell example Overview of the trail

• Commands • Simple Shell script • Get ready for HPC Linux shell tour

Ready? GO! Agenda

• Introducon

– Scripts • Pipeline • Make you own command • Environment of a shell • Summary Example 1: Navigation

• Task: Wander around on a node. Change from one directory to another directory and see the list of files in a directory. Must see directories: home, /, /bin. • Commands – echo $HOME – ls, ls –l, ls –a, – cd, cd .., cd ~, cd – – pwd • Concepts learned: – Short cuts: ~ , .. , . – Directories: current, parent, home, hidden – Command format ( opons, parameters )

Exercise 1:

• Task: Wander around on scratch space. 1. Go to /mnt/ls15/scratch/users/yourNetID 2. Go to each directory and take look. 3. Go to $SCRATCH directory and take look.

We Learned: “what is scratch space?” Example 2.1: Create a files

• Task: make a directory “ex1” that contains the file “Done”. Write the following in the file “Done”: This is what I have done. …… List of command you have done …... • Commands – mkdir – editor: nano – history – output using redirect > • We Learned: – Standard output redirect – editor Example 2.2: Get files

• Task: Make a directory “workshop” that contains: 1. Cheat sheet of linux shell commands from hp://cli.learncodethehardway.org/ bash_cheat_sheet.pdf 2. “gunzip”. Copy from directory /bin • Commands – wget – cp, cp –r • We Learned: – Copy files and directory. Exercise 2:

• Task: Make a directory “Workshop” that contains: 1. Cheat sheet of linux shell commands from hp://cli.learncodethehardway.org/ bash_cheat_sheet.pdf 2. Copy from /mnt/home/class0/Intro2Linux 3. A file “Done” same as example2.1 Hint: follow steps in example2.1 and 2.2 Example 3: Organizing files

• Task: Reorganize files. – Create new directory – Move a file from one directory to another directory (both up and down). – Rename a file • Commands – cp – mv – rm – mkdir • We Learned: – absolute/relave file name (path) – /bin, /lib, /doc, /src Exercise 3:

• Task: Organize files in “Intro2Linux” by copying data files into a directory “~/DATA” and moving pdf files into the directory “Intro2linux/DOC”. 1. Make directories “DATA” and “DOC” 2. copy data files to “DATA” 3. move pdf files to “DOC” Note: 1. DATA and DOC are at different levels. DOC is inside Intro2linux but DATA is outside. 2. Intro2linux contains same files as before. Example 4: Closer look

• Task: Looking at the contents of the file “pet_store.csv”. Find out how many records of “dog” in it. • Commands: – cat, head, head –n , tail, tail –n, more, less – grep, grep –r, grep -i – wc, • We Learned: – Get more informaon from a file content. Exercise 4: Closer look

• Task: View the contents of the data file “polls.csv”. Find out how many polls were recorded from Michigan. – Go to the DATA directory and view the file – Use “MI” to find the data recorded from Michigan. Example 5: Searching a file

• Task: Searching for a file with a name. – Find data files “*.csv” in home directory – Find a file name “poll.csv” • Command: – $ find ~ –name “*.csv” • We Learned: – Search file in a place – Wild card * Exercise 5: Search a file

• Find all CSV data files “*.csv” in your scratch space. • Find a file name “ex5”. Note: try not to use “cd” and “ls”. Instead, use “find” Example 6: Distribute files

• Task: Pack files in Part1 into an archive file, compress it and ready for distribuon • Commands: – tar, – gzip, gunzip, • Learn: – Tar/untar – Compress/uncompress Exercise 6:

• Task: Make directory “Intro2Linux” into an archive file, compress it, copy it to your scratch space, open it. – Pack a tar file – Compress – Copy to scratch space – Open it Example 7: File permission

• Task: Protect a file from the damage of write/copy/mv. • Commands: – chmod • Learn: – Ownership of a file – Change permission Exercise 7:

• Task: Make sure the data files’ permission is right to allow people in the group to read and write. If it is not, then change it. Example 8 :Self learning

• Task: find out what is the command to shuffle lines of a data file “pet_store.csv”, then create a file “pet_shuffled.csv”. • Commands – man, -help, man -k – Google search “how to shuffle lines in a file linux shell”, “random permutaon in linux shell” • Learn: – How to find command – How to get details of a command – Command “shuf” Exercise 8: Seft learning

• Task: find out how to sort lines of a data file, then sort the file “polls.csv” to “polls_sort.csv”. 1. find out if there is a command could sort the file. 2. Try to sort data file “polls.csv”. Note: Do not sort the first line! Agenda

• Introducon • Linux – Commands • Navigaon • Get or create files • Organizing files • Closer look into files • Search files • Distribute files • File permission • Learn new commands

• Summary From Command to Script Shell script

Combine more commands together for more sophiscate/complicated task. • Grammar • Environment and scope • Parameters, expressions, expansion, • Control flow (branch, loops) • Execuon (source vs. direct run, ) Examples : Pipeline

• Task: count number of files under /bin. – Soluon 1: ls /bin > bin_list wc bin_list – Soluon 2: ls /bin |wc • Learned: – If a standard output could be used as standard output of next command, a pipeline could be build – Commonly used: grep, wc, sort, less, more, uniq, head, tail,…

Examples of pipeline

• Filtering output showq |grep BatchHold • Sorng the output ls –l |sort • Counng ls |wc • Beer viewing ps |more Exercise 9: Pipeline

• Task: Find out how many jobs in job queue. Among them, how many are running. 1. Count total jobs in job queue. Use “showq” to get list of jobs in queue. 2. Count only the running jobs. Use keyword “Running” to filter the jobs Example 10 : Make a command • Task: Get ready for running jobs. Need to – Copy files to workspace – Update a work log file – Make sure it is ready: data files – Make the whole task as a command • Commands – $ cp –r BuildModel $SCRATCH – $ date >> $SCRATCH/BuildModel/log – $ ls -l $SCRATCH/BuildModel/ • Learn: – If a task could be implemented by more than single line command/pipeline and is needed repeatedly – Environment variable $SCRATCH – >> vs. > – File permission need to set to executable – Add it to $PATH Exercise 10: Make a command

• Task: follow example 10 to make a command “data_on_scratch” Example 11 : environment

• Task: In addion to example 10, we would like to – Load module – Go to scratch space – Make sure it is ready: module and scratch space • Commands – $ module load PETSc – $ echo $PATH – $ cd $SCRATCH/BuildModel – $ pwd • Learn: – Shell variable scope – Environment variable $PATH, $PWD – Source run vs. direct run Exercise 11:

• Task: 1. Follow example 11 to make your command “run_on_scratch”. Example 12 :expression

• Task: Same as Example 10 except that copy data file only and only when data file is newer. Need to – Copy data files to scratch if newer – Update a work log file – Make sure it is ready: data files – Make the whole task as a command • Commands – $ cp ./Data.csv $SCRATCH/BuildModel/Data.csv – $ date >> $SCRATCH/BuildModel/log – $ ls -l $SCRATCH/BuildModel/Data.csv • Learn: – Expression Exercise 12: expression

• Task: follow example 12 to make a command “data_update_on_scratch” Example 13 : variables

• Task: There are 3 models under develop in BuildModel. Similar as Example 11, but we would like to go to parcular directory when working on a model. Use variable to specify the parcular module to load and directory to go to. – Take two variables • Commands – Similar as example 11 • Learn: – Posion variable and its usage. Exercise 13:

• Task: 1. Follow example 13 to make your command “run_model_on_scratch” with variable. Summary

• Commands – Cheat sheet for quick reference • Very simple shell script – Get examples from Internet & HPCC • Class material: Intro2Linux Q & A

Tank You!