Brief Intro to / Brief Intro to Unix (contd) A Brief Introduction to o Brief of Unix o Compilers, Email, Text processing o Basics of a Unix session o Image Processing Linux/Unix – AMS 2019 o The Unix System Pete Pokrandt o Working with Files and Directories o The editor UW-Madison AOS Systems Administrator o Your Environment [email protected] o Common Commands Twitter @PTH1

History of Unix History of Unix o Created in 1969 by Kenneth Thompson and Dennis o Today – two main variants, but blended o It’s been around for a long Ritchie AT&T o Revised in-house until first public release 1977 o System V (Sun Solaris, SGI, Dec OSF1, AIX, o It was written by computer programmers for o 1977 – UC-Berkeley – Berkeley Software Distribution (BSD) linux) computer programmers o 1983 – Sun Workstations produced a Unix Workstation o BSD (Old SunOS, linux, Mac OSX/MacOS) o Case sensitive, mostly lowercase o AT&T unix -> System V abbreviations

1 Basics of a Unix Login Session Basics of a Unix Login Session Basics of a Unix Login Session o The – the command line interface, o Features provided by the shell o Logging in to a unix session where you enter commands, etc n Create an environment that meets your needs n login: username n Some common shells n shell scripts (batch files) n : tImpAw$ n Define command aliases (this Is my password At work $) Bourne Shell (sh) OR n Manipulate command history IHateHaving2changeMypasswordevery3weeks!!! (csh) n Automatically complete the command line (tab) The password speech… [use good ] DoIT Password guidelines: TC Shell (tcsh) n Edit the command line (arrow keys in tcsh) https://it.wisc.edu/guides/select-manage-protect-passwords Korn Shell (ksh) n Can log in than once, in several windows Bourne Again Shell () [OSX terminal]

Basics of a Unix Login Session Basics of a Unix Login Session Basics of a Unix Login Session o Logging in to a unix session o Logging off from a Unix session o Changing your password n logout, , ^d n (will ask for your old password, then your n Many people can be logged in at the same time via the network n For CentOS Linux, choose System/Log out new one, then new one again to confirm) n Remote login – secure shell [cat3/cat4/cat5.aos.wisc.edu] n In x-windows, click EXIT, right-click in n Windows – SecureCRT or putty (Xming for graphics) background, select logout/exit, try various n If you forget your password – see the systems buttons in the background. n OSX/Linux – from Terminal window administrator, they can change it for you. ssh [email protected] –Y (or –X) n SURE you are logged out, or others can n Starts in your home access your files, do things as you. Also, if the screen locks, others can’t the machine

2 Basics of a Unix Login Session The Unix The Unix File System o are you? o What is a file? o Types of files (contd) n id o Types of files .. – A special directory that refers to the n groups – what groups you belong to - Ordinary Files (text, programs, images, etc) parent directory (the one above where you are - Directories (file that holds other files, dirs) n root – the - administrator now) - Special files (used to represent physical . – A special directory that refers to the devices (printers, disks, etc) - Pipes (temporary file used to hold output from directory that you are in now one command until it is ready to be read by another All directories contain . and ..

The Unix File System The Unix File System The Unix File System

/ o Organized as a heirarchy of directories /bin o – where you are when you starting with / (the root directory) /home1 first log in (usually under /home - here, /home1/class/fall06 under /home1/class/fall18) /home1/class/fall06/poker /dev o / is similar to the Windows My o Open Terminal starts in home or /etc Desktop directory Computer, or the Mac Desktop/Finder. /home /home/aos o Current Working Directory () /home/user1 /lib o Absolute vs Relative Path Names /tmp /home1/class/fall18/poker/dir1/file1 /sbin dir1/file1

3 The Unix File System The Unix File System The Unix File System o Change directory () to dir1 o Common System Directories o Common System Directories (contd)

cd dir1 o / - root directory o /usr/include – C include files o /bin – binary executable files o /home/aos– home directory for user aos Now, ways to refer to that same file o /dev – special device files o /tornado/home1/class/fall06/poker – home directory for user poker o /etc – administrative/configuration files/programs o /usr/local – locally added programs, libraries, etc /home1/class/fall18/poker/dir1/file1 o /home – user directories o /usr/local/bin o /lib – libraries used by programs and languages file1 o /tmp – scratch area for temporary files o /usr/local/lib ../dir1/file1 o /usr – system files/directories shared by users ./file1

Using Unix Commands Using Unix Commands Using Unix Commands o Case Sensitive! ( not the same as Ls or LS) o Use backspace or delete to correct errors o Processes – unique ID number for stty erase [hit the key you want to every process that is running o The Prompt: where you enter your commands use] o Commands to identify processes agnes[poker] %1 (csh, tcsh) n (sh, ksh, bash) agnes$ o Online manual pages for almost all commands n ps –flu poker man man n ps –efl o General command syntax man passwd n ps –aux (bsd systems) command [-flags] arg1 arg2… man –k compiler

4 Using Unix Commands Using Unix Commands Using Unix Commands o ps –flu poker o Jobs – per shell shortcut of programs running o Killing processes pid # ps -flu poker agnes 27% jobs kill –STOP pid F S UID PID PPID C PRI NI ADDR SZ WCHAN STIME TIME CMD [1] + Running firefox kill -9 pid

8 S poker 1047 1 0 99 20 70cb0ec0 155 70cb0f2c o Jan 16 ? 0:00 /bin/sh ^Z – stop a running job 8 S poker 1049 1048 0 40 20 70ccd5f0 269 7015587a jobs Jan 16 ? 0:00 /var/tmp/lm_TMW12.ld fg %1 8 S poker 1048 1047 0 41 20 70ccceb8 133 70641c1c Jan 16 ? 0:00 sh -c while read line; bg %1

Using Unix Commands Using Unix Commands Using Unix Commands o Typical command locations o PATH – where linux looks for progs o Locating Programs – , n /bin agnes 1% $PATH n agnes 2% whereis pwd /research/linux_grads/grads-1.9b4/bin: n /usr/bin /research/linux_bin:/research/linux_idv: pwd: /bin/pwd /usr/bin/pwd n /usr/local/bin /research/linux_mcidas/bin: /research/ncl/bin: n agnes 3% which pwd n /home1/class/fall06/poker/bin /usr/local/weather/bin:/usr/local/bin: n /research/linux_bin /bin:/usr/bin:/usr/bin/X11:.: /usr/pwd /research/linux_gempak/GEMPAK7/os/linux/bin

agnes 2% rehash

agnes 3% ./prog

5 Using Unix Commands Using Unix Commands Using Unix Commands o Several commands can be entered on one o Command History (in C or TC shell) o Command History (in C or TC shell) command line, separated by a ; n history – list previous commands (numbered) n tcsh/bash – arrow keys ls ; date n !! – repeat previous command n up/down to cycle back/forward through command o Use output of one command as input to n !str – repeat previous command beginning with history another – separate by a | str n left/right to edit the command line ls -ltrF | n !N – repeat command number N n -a – beginning of line n o Run a command in the background n ^old^new – repeat previous command, replacing -e – end of line firefox & first occurrence of old with new n Dont need to to end of line before running

Using Unix Commands Using Unix Commands Using Unix Commands

o o Standard input/output/error from commands Redirecting standard input/output/error o Pipes (the vertical bar | ) (csh/tcsh) n Input – usually the keyboard ls –ltrF | tail n Output – usually the screen > >> >& >>& < << o Aliases – roll your own commands n Error – usually the screen ll '/bin/ls –ltrF' ls > file std output overwrites file ls >> file std output appends at end of file alias lt '/bin/ls –ltrF | tail' alias arch 'cd /bigtemp/poker/archive' ls >& file std output/error into file ls >>& file std output/error appended to file o Line Continuation character - \ ls < file std input from file ls << WORD std input until line identical to WORD [WORD must be first and only thing on the line, and unique]

6 Using Unix Commands Using Unix Commands Working with Files and Directories o Line Continuation character - \ o Shell Scripts – group of commands entered o Creating files /bin/ –r \ one by one in a file, executed as if you had n – concatenate files file1 \ typed them at the prompt file2 \ cat > file1 file3 \ this text will be put into file1 file4 #!/bin/csh echo Good Morning, Pete ^D echo Today is `date` echo Remember everything you need to do cat file1 file2 file3 > file4 exit cat file1 file2 > file1 o Used extensively for creating GEMPAK plots [overwrites original file1]

Working with Files and Directories Working with Files and Directories Working with Files and Directories o echo – echo commands to stdout (the screen?) o Editing files o Text editors n What is a text editor compared to a word n vi(m) - cryptic text editor included with all unix echo this text will be put processor? n nedit - graphical editor simliar to notepad into file1 > file1 n gedit - another graphical editor n pico/nano - nicer character based text editor n vi, nedit, gedit, nano, pico, emacs echo this text will be n emacs- powerful, customizable text editor appended after the last >> vi file1 o – update the modification time of a file, or file1 nedit file1 & create an empty file if it doesnt already exist

7 Working with Files and Directories Working with Files and Directories Working with Files and Directories o Displaying files o Listing files – ls o Copying files - n Cat – file scrolls up the screen cat file1 n cp file1 file2 copies file1 to file2 n Pagers (more, ) – pause between screenfuls ls n cp file1 dir1 creates a copy of file1 less file1 a b dir1 file1 in dir1 n Text editors (vi, nedit, pico, emacs) ls –a n cp file1 file2 file3 dir1 n – displays the first 10 lines of a file . .. a b dir1 file1 head -20 file (first 20 lines) creates copies of all 3 n Tail – displays the last 10 lines of a file ls –l file1 files in dir1 tail –f20 file (the last 20 lines, then anything -rw-r--r-- 1 poker user 203 Jan 13 16:39 file1 appended to file)

Working with Files and Directories Working with Files and Directories Working with Files and Directories o Moving/renaming files - o Deleting files – rm o Comparing two files – n mv file1 file2 renames file1 to file2 n mv file1 dir1 moves file1 to dir1/file1 rm file1 deletes file1 Line one is the same Line one is the same n mv file1 file2 file3 dir1 Line two is not the same Line two is different moves of all 3 files into dir1 rm –i file1 file2 file3 diff file1 file2 deletes file1, file2, file3, but 2c2 asks you for confirmation first < Line two is not the same --- o The –i flag works with cp and mv also > Line two is different

8 Working with Files and Directories Working with Files and Directories Working with Files and Directories o Searching the contents of files – o File permissions – controlling access to your o File permissions – controlling access to your grep EXPRESSION file1 file2 file3 files files grep –i expression file1 file2 file3 n NNN files n chmod [ugoa] [+/-] [rwx] files N = of read (4), write (2), execute (1) u – user, g – group, o – others, a – all o Sorting the contents of a file – + - add access sort file1 sorts contents of file1 chmod 761 file results in in alpha order - - remove access -rwxrw---x sort –n file1 sorts in numerical order r – read, – write, x – execute User read, write, execute sort –r file1 reverses order of sorting Group read, write sort –nr file1 reverses numerical order Other execute

Working with Files and Directories Working with Files and Directories Working with Files and Directories o – default permission mask o Wildcard characters o Determine file type – file agnes 3% file 12z28_300.ps n A 3 digit number that is subtracted from n * matches 0 or more of any characters 12z28_300.ps: PostScript document 666 for files n ? matches exactly one character 777 for directories n [Jj] matches exactly one J or j o Finding/Searching for a file – to get the default permissions find PATH –name filename –print n [1-5] matches exactly one 1,2,3,4, or 5 find /usr/people/poker –name *.txt –print n umask default is 022, resulting in default permissions of n ~ expands to full path to your home directory rwxr-xr-x (755) n ~poker expands to full path to pokers o Symbolic – a pointer to a file home dir –s original_file new_file

9 Working with Files and Directories Working with Files and Directories Working with Files and Directories o Printing files (cmd line) – lpr OR lp, lpq, lprm o Compressing files to save disk space o Transferring files from one machine to lpr –Psynoptic file_to_be_printed n gzip –v filename lpq –Psynoptic n gunzip –v filename.gz another lprm –Psynoptic idnum n scp – secure copy n compress –v filename n uncompress –v filename.Z lp –ddest file_to_be_printed scp filename username@remote:/path/filename scp username@remote:/path/filename . n pack filename o Only text or postscript files – no GIF, JPG, n unpack filename.z scp "*.txt" username@remote:/path/filename PDF, .gz, etc – although linux can handle them n bzip2 (.bz)

Working with Files and Directories Working with Files and Directories Working with Files and Directories o Transferring files from one machine to o Transferring files from one machine to o Directories another another – anonymous ftp n – create a directory n mv – move or rename a directory n ftp (only for anonymous ftp now..) n sftp – secure ftp (really sits on of scp) ftp machine.aos.wisc.edu n ls – list the contents of a directory user anonymous n cp – copy a directory sftp [email protected] cp –r dir1 dir2 copies all files/dirs in dir1 into dir2 if (use email addr as password) doesn't exist – copies dir1 and sub into dir2 if it does n Windows sftp clients: cd whateverdir n (or rm –r) – remove a directory o ssh secure shell file transfer bin (or ) o winscp (winscp.net) rmdir – remove directory only if empty get remotefile localfile o Software.wisc.edu / Campus Software Library - SecureFX (for rm –r dir1 – recursively remove dir1 and all in it windows) bye (or quit) n pwd – display full path to current directory

10 Your Environment Your Environment Your Environment o Environment variables o Common Environment variables o Common Environment variables n echo $VAR n DISPLAY for x-windows, the display location n NETCDF directory for netCDF libraries, etc n EDITOR your default text editor for , etc n LD_LIBRARY_PATH path to search for shared n setenv VAR value (in csh) n PAGER your default pager for man, etc libraries n VAR=value ; export VAR (in sh) n PATH the search path for programs n MATLABPATH – path for matlab files n NCARG_ROOT – path for NCAR graphics/ncl n export VAR=value (in bash/ksh) n PRINTER the default printer n SHELL the name of the shell you are using n [prints all environment variables] n TERM the type of terminal you are using n TZ the local time zone

Your Environment Your Environment Your Environment o Shell variables (tcsh)– usually lowercase o Startup files o Startup files - .tcshrc n # .cshrc echo $var n Used to set aliases, environment variables, paths, … n set var = value (string var in csh) etc. that you want set every time you log in switch ($TMP_OS) case irix: n @ i = 5 (set numeric variable in csh) n .cshrc – executed for all C shells # execute SGI stuff Breaksw # end of SGI stuff n set var myvar (in bash) n .tcshrc – executed for TC shell (.cshrc works too) case : n .login – only executed once at login time # execute Solaris (Sun) stuff n set [prints all environment variables] Breaksw # end of Solaris stuff n .profile – executed for Bourne, K shells case linux: # execute linux stuff Usually used to set shell specific preferences or n .bashrc or .bash_profile – for bash breaksw # end of linux stuff behavior – or in scripting

11 Your Environment Your Environment More commands o Startup files - .tcshrc (contd) o Startup files - .tcshrc (contd) o - clear your screen umask 22 # For Gempak o - display disk size, usage, amount free (512 limit coredumpsize 0 source /research/linux_gempak/NAWIPS/Gemenviron byte blocks – use –k option to get kb, -h) set path=(/research/linux_grads/grads-1.9b4/bin \ if ($?prompt ) then /research/linux_bin \ set history=32 o - display disk usage in 512 byte blocks (use /research/linux_idv \ endif –k option to get kb, -h to get Gb/Mb/Kb) … /usr/X11R6/bin \ alias ls ls –C . ) alias ll ls –ltrF du –sk * | sort –nr alias h history display disk usage in kb for each file, setenv NCARG_ROOT /research/ncl set prompt=` –n` \!% setenv GADDIR /research/solaris_grads/grads-1.9b4 directory, sorted by size, largest first breaksw # end linux stuff

More Commands More commands Even More commands o script - get a log of all commands entered and o ssh - connect to another machine over the o - pattern scanning and processing language their output (typescript) network o - stream editor o source - execute the contents of a file as if they o - displays a (cal 2001) were typed in at the prompt slogin machine.domain.edu –l username o date - sets or displays the date slogin [email protected] o tar - write one or more files/directories to tape ssh [email protected] o , - simple line-based text editors (vi is based or to an archive file, or extract from tape on these) or archive file -X or –Y to tunnel Xwindows traffic o hostname- set or display the machine name o - dumps octal, decimal, hexadecimal or ascii representations of files o who/w - who is logged into this machine right now o units - convert between units

12 Compilers/Programming languages Compilers/Programming languages Email o cc/gcc C compiler o perl Perl interpreter o mail – standard unix mail program o CC/g++- C++ compiler o python python interpreter mail [email protected] input text blah blah blah o f77/g77/pgf77/ifort - Fortran 77 compiler python file.py . jupyter notebook (web IDE) o f90/g95/pgf90/ifort - Fortran 90/95 compiler o Mail – slightly more advanced o o gfortranFortran compiler javac/java Java compiler/interpreter o elm gcc file.c - produces a.out o make maintain, update, regenerate o Pine programs and files o mozilla/thunderbird gfortran –o exefile file.f – creates exefile make o web email clients – in firefox/chrome make –f Makefile

Text Processing Image/movie processing Web browsing o Postscript – file begins with !PS… o pbmplus/netpbm- suite of image conversion progs o NO MS Internet Explorer / Edge / Safari preview with ggv, ghostview, gv o ImageMagick - suite of image conversion progs o Mozilla firefox (convert, display, identify, etc.) print using lpr o Google chrome o ffmpeg - movie creation/conversion o tex/latex – dvi files – xdvi, dvipdf, dvips o lynx - text based web browser o xv - image viewer o nroff/troff – old, mostly unix man pages o links - text based web browser o gimp - image prog similar to photoshop o xpdf – read pdf files o vlc - movie viewer o xanim - animation/movie viewer

13 Weather data viewing/plotting/searching Anaconda/Miniconda python For more … o Get miniconda from o Much of the information contained here came from a o weather - text info http://conda.pydata.org/miniconda.html document called Unix is a four-letter word… and vi o GEMPAK - graphical plotting, analysis is a two-letter abbreviation, and from UNIXhelp for o conda create --name aos330 python=3.6 Users, both available with other references at o AWIPS - graphical plotting, analysis o Must use bash (type bash to start) o McIDAS - graphical plotting, analysis https://www.aos.wisc.edu/~poker/unixhelp.html o grads - graphical plotting, analysis o source activate aos330 o vis5d - 3-d animation o idv - graphical plotting, analysis

The vi text editor The vi text editor The vi text editor o Two modes of keyboard input o Starting vi o When in insert mode, vi does what you would n Command mode – all keys used to move the n vi filename : cursor, yank/put lines, etc. n If filename did not already exist, you will see a n Characters you type are inserted into the file blank screen with a bunch of tildes (~) down the n Backspace/delete erase characters left side. This lets you know that the file is empty n Input mode – all keys are used to input the n will get you back into command mode characters that you would expect. (where the bottom is) o Typing a few times will always get you n Vi starts in command mode; certain characters place it in insert mode back to command mode

14 The vi text editor The vi text editor The vi text editor o Command mode is where you do everything o Some common keystrokes: o Some common keystrokes that isnt done in insert mode n Moving the cursor around n Moving the cursor around

h – move cursor one character to the LEFT 0 – move cursor to BEGINNING of LINE o In command mode, all the keys that would j – move cursor one line DOWN $ – move cursor to the END of the LINE normally insert characters into the file now k – move cursor one line UP G – move the cursor to the END of the FILE have completely different functions l – move cursor one character to the RIGHT 1G – move cursor to the TOP of FILE

The vi text editor The vi text editor The vi text editor o Some common keystrokes o If you try to move somewhere that vi doesnt o Inserting text (entering insert mode) n Moving the cursor around want you to move (press h to go left when n i – insert text starting before cursor your cursor is already at the left- column) n I – insert text starting before first character on line -f – move forward (down) one full screen vi will beep or flash your terminal. -b – move back (up) one full screen n a – append text after cursor -d – move down (forward) one half screen n A – append text after end of line -u – move up (back) one half screen n o – open a new line beneath the current line n O – open a new line above the current line

15 The vi text editor The vi text editor The vi text editor o Deleting text: o Saving and quitting o Copy, Delete, Move text:

n Ndd – delete N lines starting with the line the n x – delete the character that the cursor is on n :w – write to disk cursor is on. Those lines are placed in a n :wq – write to disk and exit (writes regardless storage area (buffer) that can be retrieved n – delete the line that the cursor is on of whether the file has changed or not) later on n ZZ – write to disk and exit (does not write if file has not changed) n Nyy – yank N lines starting with the line the n :q! – exit without writing to disk cursor is on. The lines are copied into a buffer; but also left intact.

The vi text editor The vi text editor The vi text editor o Copy, Delete, Move text: o Marking lines o Search and Replace: n You can mark 26 locations in the file with an invisible marker (a-z) n p – put the text from the buffer into the file n /text – search forward for next occurance of starting with the line below the cursor ma – marks the line as location a 'text' a – moves to the location marked as a n ?text – search backwards for next 'text' da – delete text from the line where the cursor n P – put the text from the buffer into the file n n – repeat the previous search, same starting with the line above the cursor is now, to the line marked with a ya – yank the text from the line where the direction cursor is now to the line marked with a n N – repeat previous search, opposite direction

16 The vi text editor The vi text editor The vi text editor o Search and Replace: o Search and Replace: o Undo n :s/search_string/replace_string/g n :.,$s/search/replace/g n u – undo the last command that you told vi to replaces every search_string on the replace every search between the current perform (usually limited to one command, current line with replace_string line (.) and the last line in the file ($) with vim under linux lets you undo many) n :s/search_string/replace_string/ replace n U – undo all of the changes made to the replaces only the first occurance on the line n :1,$s/search/replace/g current line since you moved there n :32,56s/search/replace/g n :%s/search/replace/g o Repeating commands: replaces every search occurring between both these replace every search in the n . – repeat the last command given lines 32 and 56 inclusive with replace entire file with replace

For more info… o Much of the information contained here came from a document called Unix is a four-letter word… and vi is a two-letter abbreviation, and from UNIXhelp for Users, both available with other references at

http://www.aos.wisc.edu/~poker/unixhelp.html o with X-windows on MS windows

http://www.aos.wisc.edu/~poker/windows_xwindows

17