1/20/17

Review: Unix Commands Today • What are some commands we can use to check • Process communicaon on processes? • Pipes • How do we set environment variables? • Filters Ø What are examples of environment variables? • How do we set up aliases? • Did you customize your prompt?

Jan 20, 2017 Sprenkle - CSCI397 1 Jan 20, 2017 Sprenkle - CSCI397 2

Inter-process Communicaon • Ways in which processes communicate: Ø Passing arguments, environment variables Ø Read/write regular files Ø Exit values Ø Signals Ø Pipes PROCESS COMMUNICATION

Jan 20, 2017 Sprenkle - CSCI397 3 Jan 20, 2017 Sprenkle - CSCI397 4

1 1/20/17

Pipes • General idea: The input of one program is the output of the other, and vice versa

A B

One of the cornerstones of UNIX PIPES • Both programs run at the same me

Jan 20, 2017 Sprenkle - CSCI397 5 Jan 20, 2017 Sprenkle - CSCI397 6

Pipes Redirecon/File Approach • Oen, only one end of the pipe is used Process 1 Process 2

standard out Run first program, Run second program, standard in save output into file using file as input A B

• Unnecessary use of the disk Ø Slower • Could this be done with files/redirecon? Ø Can take up a lot of space • Doesn’t take advantage of mul-tasking

Jan 20, 2017 Sprenkle - CSCI397 7 Jan 20, 2017 Sprenkle - CSCI397 8

2 1/20/17

Coordinang Pipes Filters • What if a process tries to read data but nothing is • Pipes are oen chained together available? Ø Called filters Ø UNIX puts the reader to sleep unl data available

• What if a process can’t keep up reading from the standard out process that’s wring? standard in A B C Ø UNIX keeps a buffer of unread data • This is referred to as the pipe size Ø If the pipe fills up, UNIX puts the writer to sleep unl the reader frees up space (by doing a read) • Mulple readers and writers possible with pipes

Jan 20, 2017 Sprenkle - CSCI397 9 Jan 20, 2017 Sprenkle - CSCI397 10

Pipe Examples Geng Input: What’s the difference? • Output of one program becomes input to • Both of these commands send input to another command from a file instead of the terminal: • Example: $ who | wc -l Ø counts the number of users logged in $ cat file | command

• Example: $ find . –mtime 0 | wc vs. Ø Counts number of files created in last day • Pipelines can be long: $ command < file who | awk '{print $1}' | sort | uniq

Jan 20, 2017 Sprenkle - CSCI397 11 Jan 20, 2017 Sprenkle - CSCI397 12

3 1/20/17

Difference: An Extra Process Introducon to Filters • A class of Unix tools called filters $ cat file | command Ø Ulies that read from standard input, transform the input, and write to standard out Using filters can be thought of as data-oriented cat command • programming Ø Each step of the computaon transforms data stream $ command < file

command

Jan 20, 2017 Sprenkle - CSCI397 13 Jan 20, 2017 Sprenkle - CSCI397 14

Examples of Filters tee

• sort Unix Command Standard output Ø Input: lines from a file Ø Output: lines from the file sorted • grep file-list Ø Input: lines from a file Read from standard input and write to standard Ø Output: lines that match the argument • output and one or more files • awk Ø Captures intermediate results from a filter in the Ø Programmable filter pipeline

Jan 20, 2017 Sprenkle - CSCI397 15 Jan 20, 2017 Sprenkle - CSCI397 16

4 1/20/17

tee Unix Text Files: Delimited Data

• Syntax: tee [ -ai ] file-list Tab Separated Ø -a append to output file rather than overwrite, John 99 Lots of other delimiters, e.g., commas or pipes default is to overwrite (replace) the output file Anne 95 Andrew 50 Why do we use delimiters? Ø Tim 75 -i ignore interrupts Arun 33 Ø file-list one or more file names for capturing Sowmya 76 output Colon-separated root:x:0:0:root:/root:/bin/bash • Examples mail:x:8:12:mail:/var/spool/mail:/sbin/nologin ftp:x:14:50:FTP User:/var/ftp:/sbin/nologin $ ls | head –10 | tee first_10 | tail –5 sgoryl:x:1001:1001:Steve Goryl:/home/faculty/sgoryl:/bin/bash $ who | tee user_list | wc

Jan 20, 2017 What is the end result of each command? Sprenkle - CSCI397 17 Jan 20, 2017 Sprenkle - CSCI397 18

cut: select columns cut examples • cut prints selected parts of input lines cut -f 1 newdata Ø Can select columns (assumes tab-separated input) cut -f 1-3 newdata Ø Can select a range of character posions Note how output is cut –f 4,2 newdata • Some opons: formaed cut –f 2- newdata à Columns joined by Ø -f listOfCols print only specified columns (tab- delimiter separated) on output cut –d’@' -f 1 newdata Ø -c listOfPos print only chars in specified posions cut -c 1-4 newdata Ø -d c use character c as the column separator • Lists are specified as ranges (e.g. 1-5) or comma- Unfortunately, there's no way to refer to "last column" separated (e.g. 2,4,5). without counng the columns.

Get me the list of domains from the email addresses Jan 20, 2017 Sprenkle - CSCI397 19 Jan 20, 2017 Sprenkle - CSCI397 20

5 1/20/17

paste: join columns paste example • paste displays several text files "in parallel" on output • If the inputs are files a, b, c cut -f 1 data > data1 Ø the first line of output is composed a b c cut -f 2 data > data2 of the first lines of a, b, c 1 3 5 Ø the second line of output is composed 2 4 6 cut -f 3 data > data3 of the second lines of a, b, c paste data1 data3 data2 > newdata • Lines from each file are separated by a 1 3 5 tab character 2 4 6 • If files are different lengths, output has all lines from longest file, with empty strings What is each command doing? for missing lines What is the final result?

Jan 20, 2017 Sprenkle - CSCI397 21 Jan 20, 2017 Sprenkle - CSCI397 22

sort: Sort lines of a file sort: Opons • sort copies input to output but ensures that • sort [-dftnr] [-o filename] [filename(s)] output is arranged in ascending order of lines. Opon Meaning Ø By default, sorng is based on ASCII comparisons of the -d Diconary order, only leers, digits, and whitespace are whole line significant in determining sort order • Other features of sort: -f Ignore case (fold into lower case) Ø Understands text data that occurs in columns. -t Specify delimiter (can also sort on a column other than the first) -n Numeric order, sort by arithmec value instead of first digit Ø Can disnguish numbers and sort appropriately -r Sort in reverse order Ø Can sort files “in place” as well as behaving like a filter -o Filename – write output to filename, filename can be the same as Ø Capable of sorng very large files one of the input files • Lots more opons…

Jan 20, 2017 Sprenkle - CSCI397 23 Jan 20, 2017 Sprenkle - CSCI397 24

6 1/20/17

uniq: list UNIQue items wc: Counng results • Remove or report adjacent duplicate lines • The word count ulity, wc, counts the number of • uniq [ -cdu] [input-file] [ output- lines, characters or words file] • Opons: Ø -c Supersede the -u and -d opons and generate an output report with each line preceded -l Count lines by an occurrence count -w Count words Ø -d Write only the duplicated lines -c Count characters Ø -u Write only those lines which are not Default: count lines, words and chars duplicated • Ø The default output is the union (combinaon) of -d and -u Find all the unique email address domains Jan 20, 2017 Sprenkle - CSCI397 25 Jan 20, 2017 Sprenkle - CSCI397 26

wc and uniq Examples yes What does this command do? who | sort | uniq –d • wc my_essay who | wc sort file | uniq | wc –l sort file | uniq –d | wc –l sort file | uniq –u | wc -l

Why do we have to execute sort before uniq? Can’t we just use uniq? (How do you think uniq is implemented?)

Jan 20, 2017 Sprenkle - CSCI397 27 Jan 20, 2017 Sprenkle - CSCI397 28

7 1/20/17

yes • Syntax: yes [STRING] • Output a string repeatedly unl killed

Jan 20, 2017 Sprenkle - CSCI397 29

8