this talk • show the historical background • introduce, explain and demonstrate ed, sed, awk • solve real life problems Old Software Treasuries please interrupt me at any time in case of questions markus schnalke <[email protected]> please tell me what I should demonstrate goal: motivate you to learn and use these programs old software treasuries origin • older than I am • classic Unix tools • standardized • from Bell Labs • various implementations • representations of the Unix Philosophy • freely available • good software historical background historical background 1968: 1970: • MULTICS fails • Kernighan suggests ‘‘Unix’’ • the last people working on it were Thompson, Ritchie, • deal: a PDP-11 for a document preparation system McIlroy, Ossanna 1973: 1969: • rewrite of Unix in C • they still think about operating systems 1974: • pushing idea: a file system (mainly Thompson) • first publication • in search for hardware: a rarely used PDP-7 computer hardware back then impact on software • computers are as large as closets • don’t waste characters • users work on terminals • be quiet • terminals are: a keyboard and a line printer • nothing like ‘‘screen-orientation’’ • line printers have a speed of 10–15 chars/s <picture of PDP-11> ed • by Ken Thompson in early 70s, based on qed • the Unix text editor • the standard text editor! • Unix was written with ed ed • first implementation of Regular Expressions ed: old-fashioned ed: reasons • already old-fashioned in 1984 • the standard (text editor) •‘‘So why are we spending time on such a old-fashioned • available as /bin/ed program?’’ • line oriented → works on any terminal •‘‘Although many readers will prefer some other editor for • needs few bandwidth ed daily use, is universally available, efficient and effec- • good for automated editing (scripts) tive.’’ • good for presentations (?) it’s not so much about using it, but about knowing it ed: overview ed: commands • usage: ed [-s] [file] i insert p print • commands: [a1[,a2]] cmd [params] a append n print with line number • default addresses, default params c change l list characters • success → no feedback d delete e edit file • problems → ‘‘?’’ m move w write file • about 30 commands, half of them important t transfer (copy) q quit s substitute u undo g global command ed: addresses 4 4th line $ last line . current line + next line (.+1) -- the same as .-2 <ed demo> /RE/ next line matching RE ?RE? previous line matching RE 1,$ all lines -,/foo/ from previous line to next /foo/ ed: derived software grep ed ex vi sed sed sed sed: differences to ed • by Lee McMahon in 1973 • line processing cycle •‘‘stream editor’’ • no forward/backward references • started as a one-night hack on ed (AFAIR) • hold space (g,G,h,H,x) • makes ed suitable for pipeline processing • labels and branches • irony: today, people use sed -i for in-place editing • bad i and a syntax <sed demo> awk awk awk: purpose • by Aho, Weinberger, Kernighan • for text processing • oawk in 1977, nawk in 1985 • avoids complex constructs in sh • name is abbreviation of surnames (but also ‘‘awkward’’) • floating point arithmetic • the second scripting language on Unix (besides sh) • ‘‘ sed meets C’’ awk: usage awk: features • awk ’commands’ <in >out • variables, assoc arrays • awk -f cmdfile <in >out • functions • program flow like in sed • pattern matching (EREs) • input automatically split in records and fields • dynamic typing • program is a list of blocks: cond { commands } • more high-level: no pointers awk: statements awk: functions much like C arithmetic much like in C • if, but no switch-case I/O getline(), system() • while, do-while string sub(), gsub() • for (;;) {} like in C substr(), index(), match(), • for (i in array) {} foreach split(), sprintf() • break, continue, exit, return tolower(), toupper(), • print and printf are statements! redirection possi- length(), int() ble with >, >>, | awk: variables awk: conditions RS, FS record/field separator examples: NR number of current record • NR == 1 {...} NF $1 $NF number of fields ( ... ) • NR == 1, NR == 5 {...} ARGC, ARGV like in C • /RE/ {...} FILENAME name of current input file • $1 == "string" {...} OFMT output format for numbers • $1 ˜ /RE/ {...} ORS, OFS output record/field separator SUBSEP a[1,2] equals a[1 SUBSEP 2] • BEGIN {...} • END {...} <awk demo> real life examples literature • ‘‘The Unix Programming Environment’’ by Kernighan this talk was prepared using tools of the Heirloom project: and Pike is highly recommended to support this talk http://heirloom.sf.net • it’s one of my favorite computer books the slides macros were taken from • you really should read it! http://repo.cat-v.org/troff-slider/ all editing was done with ed, of course :-) • ‘‘SED & AWK ge-packt’’ by Stephan Thesing • if you want a german book about sed and awk then get the slides and examples are available on my website this one http://marmaro.de/docs and on • you can get it for about 3 Euro http://ulm.ccc.de/ChaosSeminar/2009/10_Softwareschaetze 2009-10-12.