Process Text Streams Using Filters OBJECTIVE: Candidates should should be able to apply filters to text streams. 1 Process Text Streams Using Filters KeyKEY knowledge KNOWLEDGE area(s): AREAS: Send text files and output streams through text utility filters to modify the output using standard UNIX commands found in the GNU textutils package. 2 Process Text Streams Using Filters KEY FILES,TERMS, UTILITIES cat nl tail cut paste tr expand pr unexpand fmt sed uniq head sort wc hexdump split join tac 3 cat cat the editor - used as a rudimentary text editor. cat > short-message we are curious to meet penguins in Prague Crtl+D *Ctrl+D - command is used for ending interactive input. 4 cat cat the reader More commonly used to flush text to stdout. Options: -n number each line of output -b number only non-blank output lines -A show carriage return Example cat /etc/resolv.conf ▶ search mydomain.org nameserver 127.0.0.1 5 tac tac reads back-to-front This command is the same as cat except that the text is read from the last line to the first. tac short-message ▶ penguins in Prague to meet we are curious 6 head or tail using head or tail - often used to analyze logfiles. - by default, output 10 lines of text. List 20 first lines of /var/log/messages: head -n 20 /var/log/messages head -20 /var/log/messages List 20 last lines of /etc/aliases: tail -20 /etc/aliases 7 head or tail The tail utility has an added option that allows one to list the end of a text starting at a given line. List text starting at line 25 in /var/log/messages: tail +25 /etc/log/messages tail can continuously read a file using the -f option. This is most useful when you are expecting a file to be modified in real time. 8 wc The wc utility counts the number of bytes, words, and lines in files. Options for wc output: -l count number of lines -w count number of words -c or -m count number of bytes or characters With no argument, wc will count what is typed in stdin. 9 nl The nl utility has the same output as cat -b. Number all lines including blanks nl -ba /etc/lilo.conf Number only lines with text nl -bt /etc/lilo.conf 10 expand / unexpand The expand command is used to replace TABs with spaces. One can also use unexpand for the reverse operations. 11 hexdump A common tool for viewing binary files. Another is od (octal dump) 12 split The split tool can split a file into smaller files using criteria such as size or number of lines. split -l 5 /etc/passwd This will create files called xaa, xab, xac, xad ... each file contains at least 5 lines. It is possible to give a more meaningful prefix name for the files (other than 'x') such as 'pass-5.' on the command line split -l 5 /etc/passwd passwd-5 This has created files identical to the ones above (aa, xab, xac, xad ...) but the names are now passwd-5aa, passwd-5ab, passwd-5ac, passwd-5ad ... 13 uniq The uniq tool will send to STDOUT only one version of consecutive identical lines. uniq> /tmp/UNIQUE line 1 line 2 line 2 line 3 line 3 line 3 line 1 ^D 14 uniq The file /tmp/UNIQUE has the following content: cat /tmp/UNIQUE line 1 line 2 line 3 line 1 non consecutive identical lines are still printed to STDOUT. sort | uniq > /tmp/UNIQUE 15 cut The cut utilility can extract a range of characters or fields from each line of a text. The –c option is used to manipulate characters. Syntax: cut –c {range1,range2} Example cut –c5-10,15- /etc/password 16 cut Syntax: cut -d {delimiter} -f {fields} Example cut -d: -f 1,7 --output-delimiter=" " /etc/passwd The default output-delimiter is the same as the original input delimiter. The --output-delimiter option allows you to change this. 17 join and paste paste concatenates two files next to each other. Syntax: paste text1 text2 With join you can further specify which fields you are considering. Syntax: join -j1 {field_num} -j2{field_num} text1 text2 or join -1 {field_num} -2{field_num} text1 text2 Text is sent to stdout only if the specified fields match. Comparison is done one line at a time and as soon as no match is made the process is stopped even if more matches exist at the end of the file. 18 sort By default, sort will arrange a text in alphabetical order. To perform a numerical sort use the -n option. 19 fmt You can modify the number of characters per line of output using fmt. concatenate lines and output 75 character lines. Options -w number of characters per line -s split long lines but do not refill -u place one space between each word and two spaces at the end of a sentence 20 pr Long files can be paginated to fit a given size of paper with the pr utility. One can control the page length (default is 66 lines) and page width (default 72 characters) as well as the number of columns. When outputting text to multiple columns each column will be evenly truncated across the defined page width. This means that characters are dropped unless the original text is edited to avoid this. 21 tr The tr utility translates one set of characters into another. Example changing uppercase letters into lowercase tr 'A-B' 'a-b' < file.txt Replacing delimiters in /etc/passwd: tr ':' ' ' < /etc/passwd tr has only two arguments. The file is not an argument. 22 sed The sed utility is most often used to search and replace patterns in text. It supports most regular expressions. Syntax: sed [options] ´command’ [INPUTFILE] The input file is optional since sed also works on file redirections and pipes. Example: (using file MODIF) Delete all commented lines: sed ‘/^#/ d ’ MODIF The search pattern is between the double slashs //. 23 sed Example continued: (using file MODIF) Substitute /dev/hda1 by /dev/sdb3: sed ‘s/\/dev\/hda1/\/dev\/sdb3/g’ MODIF The s in the command stands for ‘substitute’. The g stands for “globally” and forces the substitution to take place throughout each line. If the line contains the keyword KEY then substitute ‘:’ with ‘;’ globally: sed ‘ /KEY/ s/:/;/g’ MODIF 24 sed You can issue several commands each starting with –e. Example: (1) delete all blanks then (2) substitute ‘OLD’ by ‘NEW’ in the file MODIF sed –e ‘/^$/ d’ -e ‘s/OLD/NEW/g’ MODIF These commands can also be written to a file, then each line is interpreted as a new command to execute (no quotes are needed). An example COMMANDS file 1 s/old/new/ /keyword/ s/old/new/g 23,25 d The syntax to use this COMMANDS file is: sed -f COMMANDS MODIF 25 sed Summary of Options Commandline flags -e Execute the following command -f Read commands from a file -n Do not printout unedited lines sed commands d Delete an entire line r Read a file and append to output s Substitute w Write output to a file 26 cat – concatenate files and print on the standard output cut– remove sections from each line of files expand – convert tabs to spaces fmt– simple optimal text formatter head– output the first part of files join– join lines of two files on a common field nl – number lines of files od – dump files in octal and other formats paste – merge lines of files sort – sort lines of text files split – split a file into pieces tac– concatenate and print files in reverse tail – output the last part of files tr – translate or delete characters unexpand – convert spaces to tabs uniq – remove duplicate lines from a sorted file wc – print the number of bytes, words, and lines in files 27.