DOCSLIB.ORG

Sed & Awk, 2Nd Edition

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Sed & Awk, 2Nd Edition Chapter 2. Understanding Basic Operations Table of Contents Understanding Basic Operations ................................................................ 1 Awk, by Sed and Grep, out of Ed .................................................................................................................................................... 1 Command-Line Syntax .................................................................................................................................................................. 6 Using sed ........................................................................................................................................................................................ 8 Using awk ...................................................................................................................................................................................... 12 Using sed and awk Together ......................................................................................................................................................... 15 Chapter 2. Understanding Basic Operations sed & awk, 2nd Edition By Arnold Robbins, Dale Dougherty ISBN: 1-56592-225-5 Prepared for Jeffrey Berry, Safari ID: [email protected] Publisher: O'Reilly Media, Inc. Print Publication Date: 1997/03/01 User number: 925461 © 2009 Safari Books Online, LLC. This PDF is made available for personal use only during the relevant subscription term, subject to the Safari Terms of Service. Any other use requires prior written consent from the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved. Chapter 2. Understanding Basic Operations Page 1 Return to Table of Contents Chapter 2. Understanding Basic Operations If you are starting out to learn sed and awk, you can benefit from looking at how much they have in common. • They are invoked using similar syntax. • They are both stream-oriented, reading input from text files one line at a time and directing the result to standard output. • They use regular expressions for pattern matching. • They allow the user to specify instructions in a script. One reason they have soLicensed much in common is that their by origins can be found in the same line editor, ed. In this chapter, we begin by taking a brief look at ed and show how sed and awk were logical stepsJeffrey towards the creation of Berrya programmable editor. Where sed and awk differ is in the kind of instructions that control the work they do. Make no mistake—this is a major difference, and it affects the kinds of tasks that can best be performed with these programs. This chapter looks at the command-line syntax of sed and awk and the basic structure of scripts. It also offers a tutorial, using a mailing list, that will give you a taste of script writing. It is valuable to see sed and awk scripts side-by-side before you concentrate on either one of them. 2.1. Awk, by Sed and Grep, out of Ed You can trace the lineage of awk to sed and grep, and through those two programs to ed, the original UNIX line editor. Have you ever used a line editor? If so, it will be much easier for you to understand the line orientation of sed and awk. If you have used vi, a full-screen editor, then you are familiar with a number of commands that are derived from its underlying line editor, ex (which in turn is a superset of the features in ed). Let's look at some basic operations using the line editor ed. Don't worry—this is an exercise intended to help you learn sed and awk, not an attempt to convince you of the wonders of line editors. The ed commands that are shown in this exercise are identical to the sed commands you'll learn later on. Feel free to experiment with ed on your own to get a sense of how it works. (If you're already familiar with ed, feel free to skip to the next section.) Chapter 2. Understanding Basic Operations sed & awk, 2nd Edition By Arnold Robbins, Dale Dougherty ISBN: 1-56592-225-5 Prepared for Jeffrey Berry, Safari ID: [email protected] Publisher: O'Reilly Media, Inc. Print Publication Date: 1997/03/01 User number: 925461 © 2009 Safari Books Online, LLC. This PDF is made available for personal use only during the relevant subscription term, subject to the Safari Terms of Service. Any other use requires prior written consent from the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved. Chapter 2. Understanding Basic Operations Page 2 Return to Table of Contents To use a line editor, you work on one line at a time. It is important to know what line you are positioned at in the file. When you open a file using ed, it displays the number of characters in the file and positions you at the last line. $ ed test 339 There is no prompt. If you enter a command that ed does not understand, it prints a question mark as an error message. You can enter the print command, p, to display the current line. p label on the first box. By default, a command affects only the current line. To make an edit, you move to the line that you want to edit and then apply the command. To move to a line, you specify its address. An address might consist of a line number, a symbol indicating a specific position in the file, or a regular expression. You can go to the first line by entering the line number 1. Then you can enter the delete command to remove that line. 1 You might think of a regular expression d Entering "1" makes the first line the current line, displaying it on the screen. The delete command in ed is d and here it deletes the current line. Rather than moving to a line and then editing it, you can prefix an editing command with an address that indicates which line or range of lines is the object of the command. If you enter "1d", the first line would be deleted. You can also specify a regular expression as an address. To delete a line containing the word "regular," you could issue this command: /regular/d where slashes delimit the regular expression and "regular" is the string you want to match. This command deletes the first line containing "regular" and makes the line following it the current line. Make sure you understand that the delete command deletes the whole line. It does not just delete the word "regular" on the line. To delete all the lines that contain the regular expression, you'd prefix the command with the letter g for global. Chapter 2. Understanding Basic Operations sed & awk, 2nd Edition By Arnold Robbins, Dale Dougherty ISBN: 1-56592-225-5 Prepared for Jeffrey Berry, Safari ID: [email protected] Publisher: O'Reilly Media, Inc. Print Publication Date: 1997/03/01 User number: 925461 © 2009 Safari Books Online, LLC. This PDF is made available for personal use only during the relevant subscription term, subject to the Safari Terms of Service. Any other use requires prior written consent from the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved. Chapter 2. Understanding Basic Operations Page 3 Return to Table of Contents g/regular/d The global command makes all lines that match the regular expression the object of the specified command. Deleting text can take you only so far. Substituting text (replacing one bit of text with another) is much more interesting. The substitution command, s, in ed is: [address]s/pattern/replacement/flag pattern is a regular expression that matches a string in the current line to be replaced by replacement. For example, the following command replaces the first occurrence of "regular" with "complex" on the current line. s/regular/complex/ No address is specified, so it affects only the first occurrence on the current line. It is an error if "regular" is not found on the current line. To look for multiple occurrences on the same line, you must specify g as a flag: s/regular/complex/g This command changes all occurrences on the current line. An address must be specified to direct this command to act upon more than the current line. The following substitution command specifies an address: /regular/s/regular/complex/g This command affects the first line that matches the address in the file. Remember, the first "regular" is an address and the second is a pattern to match for the substitution command. To make it apply to all lines, use the global command, putting g before the address. g/regular/s/regular/complex/g Now the substitution is made everywhere—all occurrences on all lines. Note the different meanings of "g." The "g" at the beginning is the global command that means make the changes on all lines matched by the address. The "g" at the end is a flag that means change each occurrence on a line, not just the first. The address and the pattern need not be the same. g/regular expression/s/regular/complex/g Chapter 2. Understanding Basic Operations sed & awk, 2nd Edition By Arnold Robbins, Dale Dougherty ISBN: 1-56592-225-5 Prepared for Jeffrey Berry, Safari ID: [email protected] Publisher: O'Reilly Media, Inc. Print Publication Date: 1997/03/01 User number: 925461 © 2009 Safari Books Online, LLC. This PDF is made available for personal use only during the relevant subscription term, subject to the Safari Terms of Service. Any other use requires prior written consent from the copyright owner. Unauthorized use, reproduction and/or distribution are strictly prohibited and violate applicable laws. All rights reserved. Chapter 2. Understanding Basic Operations Page 4 Return to Table of Contents On any line that contains the string "regular expression," replace "regular" with "complex." If the address and the pattern are the same, you can tell ed by specifying two consecutive delimiters (//). g/regular/s//complex/g In this example, "regular" is specified as the address and the pattern to be matched for substitution is the same. If it seems that we've covered these commands quickly and that there is a lot to absorb, don't worry.
Load more

Recommended publications
  • Learning the Vi Editor
    Learning the vi Editor en.wikibooks.org December 29, 2013 On the 28th of April 2012 the contents of the English as well as German Wikibooks and Wikipedia projects were licensed under Creative Commons Attribution-ShareAlike 3.0 Unported license. A URI to this license is given in the list of figures on page 103. If this document is a derived work from the contents of one of these projects and the content was still licensed by the project under this license at the time of derivation this document has to be licensed under the same, a similar or a compatible license, as stated in section 4b of the license. The list of contributors is included in chapter Contributors on page 101. The licenses GPL, LGPL and GFDL are included in chapter Licenses on page 107, since this book and/or parts of it may or may not be licensed under one or more of these licenses, and thus require inclusion of these licenses. The licenses of the figures are given in the list of figures on page 103. This PDF was generated by the LATEX typesetting software. The LATEX source code is included as an attachment (source.7z.txt) in this PDF file. To extract the source from the PDF file, you can use the pdfdetach tool including in the poppler suite, or the http://www. pdflabs.com/tools/pdftk-the-pdf-toolkit/ utility. Some PDF viewers may also let you save the attachment to a file. After extracting it from the PDF file you have to rename it to source.7z.
    [Show full text]
  • The Linux Command Line
    The Linux Command Line Second Internet Edition William E. Shotts, Jr. A LinuxCommand.org Book Copyright ©2008-2013, William E. Shotts, Jr. This work is licensed under the Creative Commons Attribution-Noncommercial-No De- rivative Works 3.0 United States License. To view a copy of this license, visit the link above or send a letter to Creative Commons, 171 Second Street, Suite 300, San Fran- cisco, California, 94105, USA. Linux® is the registered trademark of Linus Torvalds. All other trademarks belong to their respective owners. This book is part of the LinuxCommand.org project, a site for Linux education and advo- cacy devoted to helping users of legacy operating systems migrate into the future. You may contact the LinuxCommand.org project at http://linuxcommand.org. This book is also available in printed form, published by No Starch Press and may be purchased wherever fine books are sold. No Starch Press also offers this book in elec- tronic formats for most popular e-readers: http://nostarch.com/tlcl.htm Release History Version Date Description 13.07 July 6, 2013 Second Internet Edition. 09.12 December 14, 2009 First Internet Edition. 09.11 November 19, 2009 Fourth draft with almost all reviewer feedback incorporated and edited through chapter 37. 09.10 October 3, 2009 Third draft with revised table formatting, partial application of reviewers feedback and edited through chapter 18. 09.08 August 12, 2009 Second draft incorporating the first editing pass. 09.07 July 18, 2009 Completed first draft. Table of Contents Introduction....................................................................................................xvi
    [Show full text]
  • How Do You Use the Vi Text Editor?
    How do you use the vi text editor? 1.What is the Visual Text Editor (vi)?...................................................................2 2.Configuring your vi session..............................................................................2 2.1.Some of the options available in vi are:................................................3 2.2.Sample of the .exrc file........................................................................3 3.vi Quick Reference Guide.................................................................................4 3.1.Cursor Movement Commands.............................................................4 3.2.Inserting and Deleting Text.................................................................5 3.3.Change Commands.............................................................................6 3.4.File Manipulation...............................................................................7 Institute of Microelectronics, Singapore FAQ for vi text editor E1 1. What is the Visual Text Editor (vi)? The Visual Test Editor, more commonly known as the vi text editor, is a standard “visual” (full screen) editor available for test editing in UNIX. It is a combination of the capabilities of the UNIX ed and ex text editors. It has specific modes for text insertion and deletion as well as command entering. Type view on the command line of the UNIX shell terminal to enter the read- only mode of vi. To switch between the line editor, ex, mode and the full screen wrapmargin vi mode: • To enter the ex mode from the vi mode, type Q • To enter the vi mode from the ex mode, type vi When vi is entered on the command line, you enter vi’s command (entering) mode. Positioning and editing commands can be entered to perform functions in this mode whilst advanced editing commands can be entered following a colon (:) that places you at the bottom of the file. To enter text insertion mode, a vi text insertion command i, o, or a is typed so that text will no longer be interpreted as positioning or editing commands in this mode.
    [Show full text]
  • 1. Using Unix
    CS3214 Spring 2013 Exercise 1 Due: Wednesday, Jan 30, 2013. 11:59pm (no extensions). 1. Using Unix It is crucial that everybody become productive using a Unix command line, even if the computer you are using daily is a Windows machine. Please do the following exercises, then answer the questions below. • Make sure your personal machine has a ssh client and an X11 server installed. On Linux, these programs are usually installed by default; Windows or Mac OSX users may have to install them. The Cygwin/X package for Windows provides both ssh clients and an X11 server. • Set your machine up for public key authentication when logging on to rlogin.cs.vt.edu. Use ssh-keygen to create a key. Find out what ssh -Y does. • Test that you can run X11 client applications, such as xterm, such that they execute on a machine that is part of our rlogin.cs.vt.edu cluster and use your computer as their display. For instance, ‘xclock &’ should open a clock on your computer if set up correctly. • Make sure you can use the command line editing facilities of your shell. For bash users, which most of you are by default, examine the effect of the following keys when editing: ^d, TAB,^a,^e,^r,^k,^w Examine the effect of the following keys when you invoke a program:^c,^s,^q (^x stands for Ctrl-x.) • Make sure you know how to use a command line editor, such as vim, nano, pico, or emacs. • Familiarize yourselves with the following Unix commands if you aren’t already fa- miliar with them.
    [Show full text]
  • Lecture23 Shell Scripting II
    Lecture23 Shell Scripting II Regular Expressions We learnt about RegEx in Perl programming lectures. Regex is a pattern that can be described by a language. For example, we can find all directories within a given directory by typing ls –l | grep ^d The output from ls –l is piped into another process where grep looks for lines that “starts” with d. If you would like to know all the non-directory entries, we can type: ls –l | grep ^[^d] grep can take arguments that are regular expressions (regex) and match them to find patterns in files. sed sed , the stream editor is another tool. Its history is somewhat interesting. Back in "the day", unix's editor was a simple line-editor affectionately known as ed . It was designed for a teletype and, as a consequence, displayed only one line at a time. But, to make things easier, it could do fairly sophisticated edits on that line, including searches and replaces, &c. Eventually, with the advent of CRT terminals, it was extended into the vi sual editor that we all know and love. sed , the stream ed itor, is another one of the original ed 's offspring. When it comes right down to it, sed is a programmable filter. Text is piped into its stdin, gets filtered, and gets pumped back out via stdout in its mutated form. The programming is done via regular expressions. Since sed is a filter, its normal behavior is to output everything that it is given as input, making any changes that it needs to make along the way.
    [Show full text]
  • Getting Started with Linux (Pdf)
    Last Update January 31, 2018 Introduction This Linux tutorial is intended to give you a foundation in how to get the computer to do useful work for you. You won’t be a Linux guru at the end but you will be well on your way and armed with the right knowledge and skills to complete your assignments and programming works. The Command Line Linux has a graphical user interface and it works pretty much like the GUIs on other systems that you are familiar with such as Windows and OSX. This part won’t focus on these as I reckon you can probably figure that part out by yourself. This part will focus instead on the command line. A command line, or terminal, is a text based interface to the system. You are able to enter commands by typing them on the keyboard and feedback will be given to you similarly as text. Opening a Terminal In this part, we will introduce how to open a terminal? If you are on a Linux workstation with GUI, you will easily find the terminal application. But what if you are not in the lab? In that case, you can still work on a lab Linux machine by remote login. SSH is the best choice. To login remotely, you need a ssh client on your local machine. • For Linux and OSX, you don’t need a separate ssh client. Just type ssh commands in your local machine’s terminal. • For Windows, you can use PuTTY or Git Bash to connect a remote machine.
    [Show full text]
  • DOS to UNIX Translation
    DOS to UNIX Translation DOS to... UNIX notes attrib change file chmod attributes cd change cd chdir directories chkdsk display free df disk space cls clear screen clear comp compare files diff copy copy files cp cp to copy files, copy to copy directories, and tar to copy copy files or directories onto floppy disks or tapes. tar date display date date displays the date and time, cal displays the date, system date cal time, and a 3 month calendar. and time del delete files rm Use caution with rm and wildcards, rm * removes all (no undelete option)! deltree delete rm -r Recursively removes the specified directory and its files directory and and subdirectories and files. (no undelete option) files dir list contents ls There are several options to ls including ls -l to see long of a directory lists, ls -c to see a list in columns, and ls -f to see a list indicating file types. diskcomp make a track- diskcmp by-track comparison of two floppy disks diskcopy copy a source diskcp disk to a target disk edit to edit text vi vi is a full-screen text editor with powerful search and files replace functions. edlin line editor ed ed and ex are predecessors of vi. ex fc compare two cmp cmp to compare binary files, diff compares two text files, files diff and diff3 compares three text files. diff3 find find text grep The UNIX find command finds files on the system, within a file and grep (global regular expression parser) finds text within a file.
    [Show full text]
  • Notes on Learning
    The Vicious Interface Benjamin Brewster Except as noted, all images copyrighted with Creative Commons licenses, with attributions given whenever available If It’s So Vicious and Old, Why Study vi? • History • Understanding why it exists and why it was created informs us about the underlying OS (UNIX) and the language it was developed in (C) • Power • There are LOTS of things you can do in vi you can’t do anywhere else • Important for manipulating large data files, repetitive commands, etc. • Ubiquity • Installed on every UNIX and UNIX-like system! • Necessity • Sometimes you’ll have no other options because of the environment Text Editors • There are many text editors available on UNIX • ed (a line editor only) • ex (an extended line editor; vi’s predecessor) • emacs • vi was written by Bill Joy in 1976 for BSD • Added full-screen visibility to ex • Its name comes from the shortest unambiguous abbreviation of visual Script courtesy of Wikipedia of courtesy Script What’s a line editor? $ ed a ed is the standard Unix text editor. This is line number two. 2i . ,l ed is the standard Unix text editor.$ $ This is line number two.$ 3s/two/three/ ,l ed is the standard Unix text editor.$ $ “[ed is] the most user-hostile editor This is line number three.$ w edfile ever created.” 65 -Peter H. Salus, computer historian Q Image by user Chlor~enwiki, CC SA 3.0 $ cat edfile ed is the standard Unix text editor. This is line number three. vi = Vicious Interface • Just because it’s got a full-screen interface, it doesn’t mean it’s easy to use - but it is
    [Show full text]
  • Text Editing with Nano Made Easy
    Text editing with Nano made easy Linix Forum Magazine http://www.linuxformat.com Nano supports syntax highlighting. Nano supports text justification. And yet, Nano is so much easier than Emacs or Vim. Discover the hidden power of this versatile command line text editor - you may never want to go back to the GUI again! Oh, and it's well worth knowing a great CLI text editor too - if you end up at the shell prompt with X not working, you'll need a backup plan. Here are the tricks, tips and shortcuts you need to know... Nano is a great command-line text editor for a number of reasons: • It's small, lightweight and is included in most distributions; • It's easy to use and gives plenty of on-screen feedback; • It includes many power-user features to compete with Vi(m) and Emacs. Nano runs in text mode at the command line; this may prompt you to think: "Why would I want to learn a text mode editor? Kate, Gedit and FooEdit 2000 do everything I need to on my desktop." Well, firstly, all regular Linux users and administrators should be well-versed in a command line editor. If something goes wrong with the X Window System, for example, you'll end up at a CLI prompt and your skills will be essential for editing config files. Secondly, if you set up a Linux machine as a server, it almost certainly won't include X or any kind of graphical tools, so you'll be using a text mode editor frequently.
    [Show full text]
  • The Linux Command Line Command Linux the the Linux Command Line Command Linux the Thethe Linuxlinux
    BANISHBANISH YOURYOUR MOUSEMOUSE THE LINUX COMMAND LINE THE LINUX COMMAND LINE THETHE LINUXLINUX You’ve experienced the shiny, point-and-click surface • Use standard input and output, redirection, and COMMANDCOMMAND LINELINE of your Linux computer—now dive below and explore pipelines its depths with the power of the command line. • Edit files with Vi, the world’s most popular text editor A COMPLETE INTRODUCTION The Linux Command Line takes you from your very first • Write shell scripts to automate common or boring tasks terminal keystrokes to writing full programs in Bash, the most popular Linux shell. Along the way you’ll learn • Slice and dice text files with cut, paste, grep, patch, the timeless skills handed down by generations of and sed WILLIAM E. SHOTTS, JR. gray-bearded, mouse-shunning gurus: file navigation, Once you overcome your initial “shell shock,” you’ll environment configuration, command chaining, pattern find that the command line is a natural and expressive matching with regular expressions, and more. way to communicate with your computer. Just don’t be In addition to that practical knowledge, author William surprised if your mouse starts to gather dust. Shotts reveals the philosophy behind these tools and ABOUT THE AUTHOR the rich heritage that your desktop Linux machine has inherited from Unix supercomputers of yore. William E. Shotts, Jr., has been a software professional and avid Linux user for more than 15 years. He has an As you make your way through the book’s short, easily extensive background in software development, including digestible chapters, you’ll learn how to: technical support, quality assurance, and documentation.
    [Show full text]
  • The Text Editor Sam
    The Text Editor sam Rob Pike [email protected]−labs.com ABSTRACT Sam is an interactive multi-file text editor intended for bitmap displays. A textual command language supplements the mouse-driven, cut-and- paste interface to make complex or repetitive editing tasks easy to spec­ ify. The language is characterized by the composition of regular expres­ sions to describe the structure of the text being modified. The treatment of files as a database, with changes logged as atomic transactions, guides the implementation and makes a general ߢundoߣ mechanism straightfor­ ward. Sam is implemented as two processes connected by a low- bandwidth stream, one process handling the display and the other the editing algorithms. Therefore it can run with the display process in a bit­ map terminal and the editor on a local host, with both processes on a bitmap-equipped host, or with the display process in the terminal and the editor in a remote host. By suppressing the display process, it can even run without a bitmap terminal. This paper is reprinted from SoftwareߞPractice and Experience, Vol 17, number 11, pp. 813-845, November 1987. The paper has not been updated for the Plan 9 manuals. Although Sam has not changed much since the paper was written, the system around it certainly has. Nonethe­ less, the description here still stands as the best introduction to the edi­ tor. Introduction Sam is an interactive text editor that combines cut-and-paste interactive editing with an unusual command language based on the composition of regular expressions. It is writ­ ten as two programs: one, the ߢhost part,ߣ runs on a UNIX system and implements the command language and provides file access; the other, the ߢterminal part,ߣ runs asyn­ chronously on a machine with a mouse and bitmap display and supports the display and interactive editing.
    [Show full text]
  • Multi-Edit Manual
    Multi-Edit 9.10 By Multi Edit Software, Inc. Copyright © 2004 Multi Edit Software, Inc. All rights reserved. Multi-Edit, the Multi-Edit "packaging"and logo, are trademarks of Multi Edit Software, Inc. All other marks and names mentioned herein may be trademarks of their respective companies. Multi Edit Software, Inc. 3370 N. Hayden Rd. #123 pmb 712 Scottsdale, Arizona 85251 This manual was produced using ComponentOne Doc-To-Help.™ Contents Multi-Edit 9.10 1 What's New? .................................................................................................................1 Getting Started 5 Multi-Edit .....................................................................................................................5 What's Installed with Multi-Edit 9.10...........................................................................5 Upgrading From Previous Versions .............................................................................7 Upgrading Existing Multi-Edit Installations....................................................7 Finding Multi-Edit 9 Menus in Multi-Edit 9.10 ..............................................7 New Menu Commands in Multi-Edit 9.10 ......................................................9 Installation Requirements (Hardware/Software) ........................................................10 Add-On Integration.....................................................................................................10 Key / Command Assignments ....................................................................................11
    [Show full text]