Unix Advanced Utilities Some History • grep/egrep – search for patterns in text files. ex was a very early UNIX line editor (unusable) • sed – non-interactive stream editor It included a number of commands to manipulate • awk – pattern scanning and processing language the file being edited. • Miscellaneous commands (text processing) Other UNIX commands sed ed vi were built on ex and use the same commands. Sed: a Stream EDitor Example What is sed ? The following sentence is input to the sed • A “non-interactive” text editor that is called from program: the Unix command line. • Input text flows through the program, is $ echo "Instrumental in ruining modified, and is directed to standard output. entire operation for a Midwest chain operation." | \ • Best when integrated into a shell-script sed 's/ruining/running/' – E.g., apply same change to lots of source files Instrumental in running entire • Filter, i.e., does not modify input file operation for a Midwest chain operation. 1 Sed Invocation How Sed Works $ sed [-e script] [-f script-file] [-n] [files...] • Sed reads an input line from a given file, one line -e an "in-line" script, i.e. a script to sed execute given on the command line. at a time, into the pattern space. Multiple command line scripts can be given, each with an -e option. Pattern Space = a data buffer - the “current -n by default, sed writes each line to stdout when it reaches the end of the text” as it’s being edited script (being whatever on the line) this option prevents that. i.e. no output unless there is a command to order SED specifically to do it • For each line, sed executes a series of editing commands (written by the user, you) on the -f read scripts from specified file, several -f options can appear pattern space. files are the files to read, if a "-" appears, read from stdin,if no files are given, • Writes the pattern space to STDOUT. read also from stdin Sed Scripts Example Consists of editing commands, one per line, of the • sed '/^$/d' following form: will delete all empty lines [address[, address]] command [arguments] • An address is either empty, a decimal number that • sed '/./,$!d' counts input lines cumulatively across files, $ that will delete all leading blank lines at the top of addresses the last line of input, or a context address, file which consists of a /regular expression/ • No address selects every pattern space • One address selects one pattern space • Two addresses select the inclusive range • Address! means address does not match 2 Sed Commands Append We will go over the only some basic sed commands. $ cat text Some text here a append c change lines $ cat com d delete lines 1a\ i insert New line p print lines $ sed –f com text s substitute Some text here New line Input Change $ cat text $ cat text Some text here Some text here $ cat com $ cat com 1i\ 1c\ New line New line $ sed –f com text $ sed –f com text New line New line Some text here 3 Delete Substitute $ cat text • Probably the most useful command Some badword in text $ cat text $ sed –e “/badword/d” text Some badword in text $ sed –e “s/badword//g” text Some in text The whole line is deleted! [address1[,address2]]s/pattern /replacement /[flags] – No flags—first occurrence in the string – -g flag—all occurrences in the string Windows to Unix and back Introducing AWK Unix and DOS (and, hence, Windows) mark end of • Named after its developers Aho, Weinberger, line differently. and Kernighan #!/bin/sh^M • Programming language which permits easy # "Shell commander" model solution by Boris Konev^M ^M manipulation of structured data and the • DOS to Unix generation of formatted reports $ sed –e “s/\r//” myDos.txt > myUnix.txt • Unix to DOS $ sed –e “s/$/\r/” myUnix.txt > myDos.txt 4 Awk applications • Unix file processing, e.g., as part of a pipe • Avoid read loop in shells, Perl • More intuitive syntax than shell scripts • Best limited to line-at-a-time processing 5.