Introduction to Linux/Unix

Xiaoge Wang [email protected]

How does this class works

• We are going to cover some basics with hands on exampes. • Exercises are denoted by the following icon in this presents:

• Examples will be showed and code will be provided. Please try to follow the demo. The history of demo will be provided at the end of lecture. Green and Red Sticky

• Use the scky notes provided to help me help you. – No Scky = I am working – Green = I am done and ready to move on (yea!) – Red = I am stuck and need more me and/or some help Agenda

• Introducon • Linux/Shell Tour – Commands • Set 1: look around • Set 2: Do something • Summary – Pipeline • Standard input and output and redirecon • Pipeline of commands – Scripts • Grammar(key words and special characters) • Scope of script and recursive • Variables, expression, expansion • Control flow • Execuon model (source vs. direct run, ) • Summary Agenda

• Linux/Shell Tour – Commands • Set 1: look around • Set 2: Do something • Summary – Pipeline • Standard input and output and redirecon • Pipeline of commands – Scripts • Grammar(key words and special characters) • Scope of script and recursive • Variables, expression, expansion • Control flow • Execuon model (source vs. direct run, ) • Summary Introduction

• Get ready – Ticket? Equipment? • Big Map – Linux/Unix – Shell • Overview of trail – Commands – Shell script – Get ready for HPC and more. Exercise 1: ssh

ssh yourne[email protected]

Connect to HPCC: go to any develop node. – Windows users MobaXterm

– Mac users Terminal

– Linux users? • Show ‘mod’ and ‘ssh dev-X’ Example of terminal

• Mac Show screen MobaXterm Big Picture

Shell Shell Big Picture

• Shell – CLI ✔ – GUI OS Shell Example Overview of the Tour

– Commands • Set 1: Look around • Set 2: Do something • Summary – Pipeline • Standard input and output and redirecon • Pipeline of commands – Scripts • Grammar(key words and special characters) • Scope of script and recursive • Variables, expression, expansion • Control flow • Execuon model (source vs. direct run, ) Linux/Shell Tour

Ready? GO! Agenda

• Introducon

– Pipeline • Standard Input and Output • Redirecon >, >>, .& • Pipeline of commands – Scripts • Grammar(key words and special characters) • Scope of script and recursive • Variables, expression, expansion, special charactors ($, backck, =, !, …) • Control flow • Execuon model (source vs. direct run, ) – Help and self-explore (man, --help, google, sites, examples, ) • Summary Set 1: Look Around

• What are in your home directory? • What you can tell about an item in the directory? • Where are commands I can see? • What is the content of a file? • What soware ready for use? • How many jobs are running? • Do I have jobs running? • How many jobs are waing in the queue? • Who are currently on this node? • Who are they? Who am I? • Why machine is so slow? Who is using it? • …...

Demo 1 ls, pwd, cd, more, less, cat, head, tail, env, set, module list/spider, export, echo, ps, top, mod, grep, ….... • (1) starng from home directory: – pwd: absolute path and relave path – ls: permission, ownership, others • (2) Look into file content – cat, more, less, head, tail, find • (3) Look into command repository (PATH) – echo $PATH – Module list/spider/show – Powertools – man, whas, which • (4) Look into what is running on the nodes – top, ps, mod, showq, qstat – who, whoami, finger, userinfo • (5) Look into users – who, whoami, finger, userinfo

Exercise 2

• Find the number of files in your home directory • Find the CPU load and memory usage of the dev node. Set 2: Do Something

• Addion – Bring in data/files – Create new files – Load something from HPCC • Subtracon – Delete file or folder – Unload modules • Change/move things Demo 2

Touch, cp, mv, rm, mkdir, module load, scp, curl, wget, vi, nano, > and >>, chmod, export, • (1) Add something – a) create locally: mkdir, cp mv, vi/vim/nano >, >> – b) cp from PC: scp, (not cover disk drive mapping) – c) down load from Internet site (not involve the GUI) – d) by “getexample” , getexample, module load • (2) Remove something, – rm, mv, module purge, module unload • (3) Change something – chmod, mv (with careful), touch, export Exercise 2

• Copy example files to from /mnt/scratch/ class0/intro2linux.tar to you home directory. • Untar it to a directory by $ tar –xvf tarfilename • Or copy directory intro2linux to your home directory Summary

• Command format – Command name – Opons: strings, integers, filenames – Input/output: stdin, stdout, stderr, redirecon • Command locaon – PATH and how to set • Tricks – Tab (Key board short cut) – ! – Arrows Agenda

• Introducon r – Commands • Set 1: look around • Set 2: Do something • Summary

– Scripts • Grammar(key words and special characters) • Scope of script and recursive • Variables, expression, expansion • Control flow • Execuon model (source vs. direct run, ) • Summary I/O and Redirection

• Standard input and output • >, >>, >&, < Demo 3

• Use input file to replace the stdin • Stdout to create a output Pipeline

• Task need more commands to complete • One command’s output is used for next command’s input • Format: Commd1 |commd2 • Commonly used – grep, wc, sort, less, more, uniq, head, tail,… Examples of pipeline

• Filtering output showq |grep userid • Sorng the output ls –l |sort • Counng ls |wc • Beer viewing ps |more Exercise 4

• Use pipeline to find out how many people are login. • Find out how many jobs in job queue. Among them, how many are running. Let’s Take a Break Agenda

• Introducon

– Commands • Set 1: look around • Set 2: Do something • Summary – Pipeline • Standard Input and Output and redirecon • Pipeline of commands

• Summary Shell Script

Combine more commands together for more sophiscate/complicated task. • Grammar • Environment and scope • Parameters, expressions, expansion, • Control flow (branch, loops) • Execuon (source vs. direct run, ) Environment and Scope

• Variables • Inializaon: • /etc/profile • ~/.profile • ~/.bashrc • Local variable Demo 4

• Set inial environment such that module “powertools” is loaded in Exercise 5

Add ~/intro2linux to your PATH by add following line into .bashrc file: export PATH=~/intro2linux:$PATH To check the result, use echo $PATH To see if it is added into the PATH Control Flow

• if • exit • for • while • unl • case • break • connue if

• First form if condion ; then commands Fi

• Second form if condion ; then commands else commands fi

• Third form if condion ; then commands elif condion ; then commands fi What is a "condition"?

• Test – First form test expression – Second form [ expression ] • Return value Expression examples

• -d file True if file exists and is a directory. • -e file True if file exists (regardless of type). • -f file True if file exists and is a regular file. • file1 -nt file2 True if file1 exists and is newer than file2. • s1 = s2 True if the strings s1 and s2 are idencal. • s1 < s2 True if string s1 comes before s2 based on the ASCII value of their characters. • n1 -eq n2 True if the integers n1 and n2 are algebraically equal. • expression1 -a expression2 True if both expression1 and expression2 are true.

NOTE: use “man test” to get more details of expression Example

$ more tesf $ ./tesf file1 file2 Case– More branching

• Form case word in paerns ) statements ;; esac

• Example (in case_ex)

#!/bin/bash

echo -n "Type a digit or a leer > " read character case $character in # Check for leers [a-z] | [A-Z] ) echo "You typed the leer $character" ;;

# Check for digits [0-9] ) echo "You typed the digit $character" ;;

# Check for anything else * ) echo "You did not type a leer or a digit" esac Loops: while, for, until

• Example 1 (in while_ex): #!/bin/bash

number=0 while [ $number –lt 5 ]; do echo "Number = $number” # do things here number=$((number + 1)) done

• Example 2 (in unl_ex): #!/bin/bash

number=0 unl [ $number –ge 5 ]; do echo "Number = $number” # do things here number=$((number + 1)) done Loops:

• for loop: for variable in words; do statements done • Example (in for_ex): #!/bin/bash

for number in 1 2 3 4 5; do echo “Number = $((number-1))”

# do things here # do not need this: number=$((number + 1)) done Quotes and Escaping

There are three recognized types: • per-character escaping using a backslash: \ $stuff • weak quong with double-quotes: "stuff" • strong quong with single-quotes: 'stuff' All three forms have the very same purpose: They give you general control over parsing, expansion and expansion results. NOTE: Bash contains ANSI C-like escape sequences Example: quotes and escaping

In “quote_escape_ex” #/bin/bash

# first $HOME is not expand, so as double quote echo \$HOME is set to \"$HOME\"

# weak quote echo "My PATH is: $PATH"

# strong quote echo 'My PATH is: $PATH’

Expansions and substitutions

• Parameter expansion – $WORD, ${STUFF…} • Pathname expansion – *.txt, page_1?.html • Arithmec expansion – $(( EXPRESSION )) • Command substuon – $( COMMAND ), ` COMMAND ` • Tilde expansion – ~, ~+, ~- • Brace expansion – {X,Y,Z}, {X..Y}, {X..Y..Z} • Process substuon – <( COMMAND ), >( COMMAND )

Parameter Expansion

• Posional parameters • Special parameters • Named variables – System defined variables – Script defined variables

Position Parameters

• The arguments given to your scripts when it is invoked (command line). • It could be from $1 to $N. • It has total number of $# • $@ expands the list of all • Note: When N consists of more than a single digit, it must be enclosed in a braces like ${N}. Special Parameters

variable Descripon

$# The number of arguments supplied to a script.

$* All the arguments are double quoted. If a script receives two arguments, $* is equivalent to $1 $2. $@ All the arguments are individually double quoted. If a script receives two arguments, $@ is equivalent to $1 $2. $? The exit status of the last command executed.

$$ The process number of the current shell. For shell scripts, this is the process ID under which they are executing. $! The process number of the last background command.

$0 The filename of the current script. Named Variables

Some names are taken:

BASH BASH_ARGC BASH_ARGV BASH_LINENO BASH_SOURCE BASH_VERSINFO BASH_VERSION COLUMNS DIRSTACK DISPLAY EDITOR EUID GROUPS HISTFILE HISTFILESIZE HISTSIZE HOME HOSTNAME IFS LANG LANGUAGE LC_ALL LINES LOGNAME LS_COLORS MACHTYPE MAILCHECK OLDPWD OPTERR OPTIND OSTYPE PATH PIPESTATUS PPID PROMPT_COMMAND PS1 PS2 PS4 PS3 PWD SHELL SHELLOPTS SHLVL TERM UID USER USERNAME XAUTHORITY

Examples of Expansion

• brace_expansion_ex • arthm_ex • pipe_in_shell • pleasant • posion_var_ex Execution model

• Directly run • Use “source” • What is the difference? • Example in “execuon_model_ex” Agenda

• Introducon • Linux/Shell Tour – Commands • Set 1: look around • Set 2: Do something • Summary – Pipeline • Standard Input and Output and Redirecon • Pipeline of commands – Scripts • Grammar(key words and special characters) • Scope of script and recursive • Variables, expression, expansion • Control flow • Execuon model (source vs. direct run, ) Summary

• Commands • Pipeline: special sequence of commands • Shell scripts: A composion of commands • Useful resources: – man page – internet – exisng command

Homework

• A directory contains 1000 text files. Create a list of all files that contain the word "Aries" and save the result to a files called results.txt. • Daily backup of your scratch space. • Send me quesons by email with subject “Intro2Linux”. Q & A

Tank You!