1/20/17 Review: Unix Commands Today • What are some commands we can use to check • Process communicaon on processes? • Pipes • How do we set environment variables? • Filters Ø What are examples of environment variables? • How do we set up aliases? • Did you customize your prompt? Jan 20, 2017 Sprenkle - CSCI397 1 Jan 20, 2017 Sprenkle - CSCI397 2 Inter-process Communicaon • Ways in which processes communicate: Ø Passing arguments, environment variables Ø Read/write regular files Ø Exit values Ø Signals Ø Pipes PROCESS COMMUNICATION Jan 20, 2017 Sprenkle - CSCI397 3 Jan 20, 2017 Sprenkle - CSCI397 4 1 1/20/17 Pipes • General idea: The input of one program is the output of the other, and vice versa A B One of the cornerstones of UNIX PIPES • Both programs run at the same Pme Jan 20, 2017 Sprenkle - CSCI397 5 Jan 20, 2017 Sprenkle - CSCI397 6 Pipes RedirecPon/File Approach • O[en, only one end of the pipe is used Process 1 Process 2 standard out Run first program, Run second program, standard in save output into file using file as input A B • Unnecessary use of the disk Ø Slower • Could this be done with files/redirecPon? Ø Can take up a lot of space • Doesn’t take advantage of mulP-tasking Jan 20, 2017 Sprenkle - CSCI397 7 Jan 20, 2017 Sprenkle - CSCI397 8 2 1/20/17 Coordinang Pipes Filters • What if a process tries to read data but nothing is • Pipes are o[en chained together available? Ø Called filters Ø UNIX puts the reader to sleep unPl data available • What if a process can’t keep up reading from the standard out process that’s wriPng? standard in A B C Ø UNIX keeps a buffer of unread data • This is referred to as the pipe size Ø If the pipe fills up, UNIX puts the writer to sleep unPl the reader frees up space (by doing a read) • MulPple readers and writers possible with pipes Jan 20, 2017 Sprenkle - CSCI397 9 Jan 20, 2017 Sprenkle - CSCI397 10 Pipe Examples Geng Input: What’s the difference? • Output of one program becomes input to • Both of these commands send input to another command from a file instead of the terminal: • Example: $ who | wc -l Ø counts the number of users logged in $ cat file | command • Example: $ find . –mtime 0 | wc vs. Ø Counts number of files created in last day • Pipelines can be long: $ command < file who | awk '{print $1}' | sort | uniq Jan 20, 2017 Sprenkle - CSCI397 11 Jan 20, 2017 Sprenkle - CSCI397 12 3 1/20/17 Difference: An Extra Process IntroducPon to Filters • A class of Unix tools called filters $ cat file | command Ø UPliPes that read from standard input, transform the input, and write to standard out Using filters can be thought of as data-oriented cat command • programming Ø Each step of the computaon transforms data stream $ command < file command Jan 20, 2017 Sprenkle - CSCI397 13 Jan 20, 2017 Sprenkle - CSCI397 14 Examples of Filters tee • sort Unix Command Standard output Ø Input: lines from a file Ø Output: lines from the file sorted • grep file-list Ø Input: lines from a file Read from standard input and write to standard Ø Output: lines that match the argument • output and one or more files • awk Ø Captures intermediate results from a filter in the Ø Programmable filter pipeline Jan 20, 2017 Sprenkle - CSCI397 15 Jan 20, 2017 Sprenkle - CSCI397 16 4 1/20/17 tee Unix Text Files: Delimited Data • Syntax: tee [ -ai ] file-list Tab Separated Ø -a append to output file rather than overwrite, John 99 Lots of other delimiters, e.g., commas or pipes default is to overwrite (replace) the output file Anne 95 Andrew 50 Why do we use delimiters? Ø Tim 75 -i ignore interrupts Arun 33 Ø file-list one or more file names for capturing Sowmya 76 output Colon-separated root:x:0:0:root:/root:/bin/bash • Examples mail:x:8:12:mail:/var/spool/mail:/sbin/nologin ftp:x:14:50:FTP User:/var/ftp:/sbin/nologin $ ls | head –10 | tee first_10 | tail –5 sgoryl:x:1001:1001:Steve Goryl:/home/faculty/sgoryl:/bin/bash $ who | tee user_list | wc Jan 20, 2017 What is the end result of each command? Sprenkle - CSCI397 17 Jan 20, 2017 Sprenkle - CSCI397 18 cut: select columns cut examples • cut prints selected parts of input lines cut -f 1 newdata Ø Can select columns (assumes tab-separated input) cut -f 1-3 newdata Ø Can select a range of character posiPons Note how output is cut –f 4,2 newdata • Some opons: formaed cut –f 2- newdata à Columns joined by Ø -f listOfCols print only specified columns (tab- delimiter separated) on output cut –d’@' -f 1 newdata Ø -c listOfPos print only chars in specified posiPons cut -c 1-4 newdata Ø -d c use character c as the column separator • Lists are specified as ranges (e.g. 1-5) or comma- Unfortunately, there's no way to refer to "last column" separated (e.g. 2,4,5). without counFng the columns. Get me the list of domains from the email addresses Jan 20, 2017 Sprenkle - CSCI397 19 Jan 20, 2017 Sprenkle - CSCI397 20 5 1/20/17 paste: join columns paste example • paste displays several text files "in parallel" on output • If the inputs are files a, b, c cut -f 1 data > data1 Ø the first line of output is composed a b c cut -f 2 data > data2 of the first lines of a, b, c 1 3 5 Ø the second line of output is composed 2 4 6 cut -f 3 data > data3 of the second lines of a, b, c paste data1 data3 data2 > newdata • Lines from each file are separated by a 1 3 5 tab character 2 4 6 • If files are different lengths, output has all lines from longest file, with empty strings What is each command doing? for missing lines What is the final result? Jan 20, 2017 Sprenkle - CSCI397 21 Jan 20, 2017 Sprenkle - CSCI397 22 sort: Sort lines of a file sort: OpPons • sort copies input to output but ensures that • sort [-dftnr] [-o filename] [filename(s)] output is arranged in ascending order of lines. Opon Meaning Ø By default, sorPng is based on ASCII comparisons of the -d DicPonary order, only lehers, digits, and whitespace are whole line significant in determining sort order • Other features of sort: -f Ignore case (fold into lower case) Ø Understands text data that occurs in columns. -t Specify delimiter (can also sort on a column other than the first) -n Numeric order, sort by arithmePc value instead of first digit Ø Can disPnguish numbers and sort appropriately -r Sort in reverse order Ø Can sort files “in place” as well as behaving like a filter -o Filename – write output to filename, filename can be the same as Ø Capable of sorPng very large files one of the input files • Lots more opPons… Jan 20, 2017 Sprenkle - CSCI397 23 Jan 20, 2017 Sprenkle - CSCI397 24 6 1/20/17 uniq: list UNIQue items wc: CounPng results • Remove or report adjacent duplicate lines • The word count uPlity, wc, counts the number of • uniq [ -cdu] [input-file] [ output- lines, characters or words file] • Opons: Ø -c Supersede the -u and -d opPons and generate an output report with each line preceded -l Count lines by an occurrence count -w Count words Ø -d Write only the duplicated lines -c Count characters Ø -u Write only those lines which are not Default: count lines, words and chars duplicated • Ø The default output is the union (combinaon) of -d and -u Find all the unique email address domains Jan 20, 2017 Sprenkle - CSCI397 25 Jan 20, 2017 Sprenkle - CSCI397 26 wc and uniq Examples yes What does this command do? who | sort | uniq –d • wc my_essay who | wc sort file | uniq | wc –l sort file | uniq –d | wc –l sort file | uniq –u | wc -l Why do we have to execute sort before uniq? Can’t we just use uniq? (How do you think uniq is implemented?) Jan 20, 2017 Sprenkle - CSCI397 27 Jan 20, 2017 Sprenkle - CSCI397 28 7 1/20/17 yes • Syntax: yes [STRING] • Output a string repeatedly unl killed Jan 20, 2017 Sprenkle - CSCI397 29 8 .