Chapter 2. Understanding Basic Operations Table of Contents Understanding Basic Operations ................................................................ 1 Awk, by Sed and Grep, out of Ed .................................................................................................................................................... 1 Command-Line Syntax .................................................................................................................................................................. 6 Using sed ........................................................................................................................................................................................ 8 Using awk ...................................................................................................................................................................................... 12 Using sed and awk Together ......................................................................................................................................................... 15 Chapter 2. Understanding Basic Operations sed & awk, 2nd Edition By Arnold Robbins, Dale Dougherty ISBN: 1-56592-225-5 Prepared for Jeffrey Berry, Safari ID: [email protected] Publisher: O'Reilly Media, Inc. Print Publication Date: 1997/03/01 User number: 925461 © 2009 Safari Books Online, LLC. This PDF is made available for personal use only during the relevant subscription term, subject to the Safari Terms of Service. Any other use requires prior written consent from the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved. Chapter 2. Understanding Basic Operations Page 1 Return to Table of Contents Chapter 2. Understanding Basic Operations If you are starting out to learn sed and awk, you can benefit from looking at how much they have in common. • They are invoked using similar syntax. • They are both stream-oriented, reading input from text files one line at a time and directing the result to standard output. • They use regular expressions for pattern matching. • They allow the user to specify instructions in a script. One reason they have soLicensed much in common is that their by origins can be found in the same line editor, ed. In this chapter, we begin by taking a brief look at ed and show how sed and awk were logical stepsJeffrey towards the creation of Berrya programmable editor. Where sed and awk differ is in the kind of instructions that control the work they do. Make no mistake—this is a major difference, and it affects the kinds of tasks that can best be performed with these programs. This chapter looks at the command-line syntax of sed and awk and the basic structure of scripts. It also offers a tutorial, using a mailing list, that will give you a taste of script writing. It is valuable to see sed and awk scripts side-by-side before you concentrate on either one of them. 2.1. Awk, by Sed and Grep, out of Ed You can trace the lineage of awk to sed and grep, and through those two programs to ed, the original UNIX line editor. Have you ever used a line editor? If so, it will be much easier for you to understand the line orientation of sed and awk. If you have used vi, a full-screen editor, then you are familiar with a number of commands that are derived from its underlying line editor, ex (which in turn is a superset of the features in ed). Let's look at some basic operations using the line editor ed. Don't worry—this is an exercise intended to help you learn sed and awk, not an attempt to convince you of the wonders of line editors. The ed commands that are shown in this exercise are identical to the sed commands you'll learn later on. Feel free to experiment with ed on your own to get a sense of how it works. (If you're already familiar with ed, feel free to skip to the next section.) Chapter 2. Understanding Basic Operations sed & awk, 2nd Edition By Arnold Robbins, Dale Dougherty ISBN: 1-56592-225-5 Prepared for Jeffrey Berry, Safari ID: [email protected] Publisher: O'Reilly Media, Inc. Print Publication Date: 1997/03/01 User number: 925461 © 2009 Safari Books Online, LLC. This PDF is made available for personal use only during the relevant subscription term, subject to the Safari Terms of Service. Any other use requires prior written consent from the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved. Chapter 2. Understanding Basic Operations Page 2 Return to Table of Contents To use a line editor, you work on one line at a time. It is important to know what line you are positioned at in the file. When you open a file using ed, it displays the number of characters in the file and positions you at the last line. $ ed test 339 There is no prompt. If you enter a command that ed does not understand, it prints a question mark as an error message. You can enter the print command, p, to display the current line. p label on the first box. By default, a command affects only the current line. To make an edit, you move to the line that you want to edit and then apply the command. To move to a line, you specify its address. An address might consist of a line number, a symbol indicating a specific position in the file, or a regular expression. You can go to the first line by entering the line number 1. Then you can enter the delete command to remove that line. 1 You might think of a regular expression d Entering "1" makes the first line the current line, displaying it on the screen. The delete command in ed is d and here it deletes the current line. Rather than moving to a line and then editing it, you can prefix an editing command with an address that indicates which line or range of lines is the object of the command. If you enter "1d", the first line would be deleted. You can also specify a regular expression as an address. To delete a line containing the word "regular," you could issue this command: /regular/d where slashes delimit the regular expression and "regular" is the string you want to match. This command deletes the first line containing "regular" and makes the line following it the current line. Make sure you understand that the delete command deletes the whole line. It does not just delete the word "regular" on the line. To delete all the lines that contain the regular expression, you'd prefix the command with the letter g for global. Chapter 2. Understanding Basic Operations sed & awk, 2nd Edition By Arnold Robbins, Dale Dougherty ISBN: 1-56592-225-5 Prepared for Jeffrey Berry, Safari ID: [email protected] Publisher: O'Reilly Media, Inc. Print Publication Date: 1997/03/01 User number: 925461 © 2009 Safari Books Online, LLC. This PDF is made available for personal use only during the relevant subscription term, subject to the Safari Terms of Service. Any other use requires prior written consent from the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved. Chapter 2. Understanding Basic Operations Page 3 Return to Table of Contents g/regular/d The global command makes all lines that match the regular expression the object of the specified command. Deleting text can take you only so far. Substituting text (replacing one bit of text with another) is much more interesting. The substitution command, s, in ed is: [address]s/pattern/replacement/flag pattern is a regular expression that matches a string in the current line to be replaced by replacement. For example, the following command replaces the first occurrence of "regular" with "complex" on the current line. s/regular/complex/ No address is specified, so it affects only the first occurrence on the current line. It is an error if "regular" is not found on the current line. To look for multiple occurrences on the same line, you must specify g as a flag: s/regular/complex/g This command changes all occurrences on the current line. An address must be specified to direct this command to act upon more than the current line. The following substitution command specifies an address: /regular/s/regular/complex/g This command affects the first line that matches the address in the file. Remember, the first "regular" is an address and the second is a pattern to match for the substitution command. To make it apply to all lines, use the global command, putting g before the address. g/regular/s/regular/complex/g Now the substitution is made everywhere—all occurrences on all lines. Note the different meanings of "g." The "g" at the beginning is the global command that means make the changes on all lines matched by the address. The "g" at the end is a flag that means change each occurrence on a line, not just the first. The address and the pattern need not be the same. g/regular expression/s/regular/complex/g Chapter 2. Understanding Basic Operations sed & awk, 2nd Edition By Arnold Robbins, Dale Dougherty ISBN: 1-56592-225-5 Prepared for Jeffrey Berry, Safari ID: [email protected] Publisher: O'Reilly Media, Inc. Print Publication Date: 1997/03/01 User number: 925461 © 2009 Safari Books Online, LLC. This PDF is made available for personal use only during the relevant subscription term, subject to the Safari Terms of Service. Any other use requires prior written consent from the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved. Chapter 2. Understanding Basic Operations Page 4 Return to Table of Contents On any line that contains the string "regular expression," replace "regular" with "complex." If the address and the pattern are the same, you can tell ed by specifying two consecutive delimiters (//). g/regular/s//complex/g In this example, "regular" is specified as the address and the pattern to be matched for substitution is the same. If it seems that we've covered these commands quickly and that there is a lot to absorb, don't worry.