Classic Shell Scripting Arnold Robbins and Nelson H. F. Beebe Beijing • Cambridge • Farnham • Köln • Sebastopol • Tokyo Classic Shell Scripting by Arnold Robbins and Nelson H. F. Beebe Copyright © 2005 O’Reilly Media, Inc. All rights reserved. Printed in the United States of America. Published by O’Reilly Media, Inc., 1005 Gravenstein Highway North, Sebastopol, CA 95472. O’Reilly books may be purchased for educational, business, or sales promotional use. Online editions are also available for most titles (safari.oreilly.com). For more information,contact our corporate/insti- tutional sales department: (800) 998-9938 or [email protected]. Editors: Tatiana Apandi Allison Randal Production Editor: Adam Witwer Cover Designer: Emma Colby Interior Designer: David Futato Printing History: May 2005: First Edition. Nutshell Handbook, the Nutshell Handbook logo, and the O’Reilly logo are registered trademarks of O’Reilly Media,Inc. Classic Shell Scripting,the image of a African tent tortoise,and related trade dress are trademarks of O’Reilly Media, Inc. Many of the designations used by manufacturers and sellers to distinguish their products are claimed as trademarks. Where those designations appear in this book, and O’Reilly Media, Inc. was aware of a trademark claim, the designations have been printed in caps or initial caps. While every precaution has been taken in the preparation of this book, the publisher and authors assume no responsibility for errors or omissions, or for damages resulting from the use of the information contained herein. ISBN: 978-0-596-00595-5 [LSI] [2011-03-11] Table of Contents Foreword . ix Preface . xi 1. Background . 1 1.1 Unix History 1 1.2 Software Tools Principles 4 1.3 Summary 6 2. Getting Started . 8 2.1 Scripting Languages Versus Compiled Languages 8 2.2 Why Use a Shell Script? 9 2.3 A Simple Script 9 2.4 Self-Contained Scripts: The #! First Line 10 2.5 Basic Shell Constructs 12 2.6 Accessing Shell Script Arguments 23 2.7 Simple Execution Tracing 24 2.8 Internationalization and Localization 25 2.9 Summary 28 3. Searching and Substitutions . 30 3.1 Searching for Text 30 3.2 Regular Expressions 31 3.3 Working with Fields 56 3.4 Summary 65 iii 4. Text Processing Tools . 67 4.1 Sorting Text 67 4.2 Removing Duplicates 75 4.3 Reformatting Paragraphs 76 4.4 Counting Lines, Words, and Characters 77 4.5 Printing 78 4.6 Extracting the First and Last Lines 83 4.7 Summary 86 5. Pipelines Can Do Amazing Things . 87 5.1 Extracting Data from Structured Text Files 87 5.2 Structured Data for the Web 94 5.3 Cheating at Word Puzzles 100 5.4 Word Lists 102 5.5 Tag Lists 105 5.6 Summary 107 6. Variables, Making Decisions, and Repeating Actions . 109 6.1 Variables and Arithmetic 109 6.2 Exit Statuses 120 6.3 The case Statement 129 6.4 Looping 130 6.5 Functions 135 6.6 Summary 138 7. Input and Output, Files, and Command Evaluation . 140 7.1 Standard Input, Output, and Error 140 7.2 Reading Lines with read 140 7.3 More About Redirections 143 7.4 The Full Story on printf 147 7.5 Tilde Expansion and Wildcards 152 7.6 Command Substitution 155 7.7 Quoting 161 7.8 Evaluation Order and eval 162 7.9 Built-in Commands 168 7.10 Summary 175 iv | Table of Contents 8. Production Scripts . 177 8.1 Path Searching 177 8.2 Automating Software Builds 192 8.3 Summary 222 9. Enough awk to Be Dangerous . 223 9.1 The awk Command Line 224 9.2 The awk Programming Model 225 9.3 Program Elements 226 9.4 Records and Fields 236 9.5 Patterns and Actions 238 9.6 One-Line Programs in awk 240 9.7 Statements 244 9.8 User-Defined Functions 252 9.9 String Functions 255 9.10 Numeric Functions 264 9.11 Summary 266 10. Working with Files . 267 10.1 Listing Files 267 10.2 Updating Modification Times with touch 273 10.3 Creating and Using Temporary Files 274 10.4 Finding Files 279 10.5 Running Commands: xargs 293 10.6 Filesystem Space Information 295 10.7 Comparing Files 299 10.8 Summary 307 11. Extended Example: Merging User Databases . 308 11.1 The Problem 308 11.2 The Password Files 309 11.3 Merging Password Files 310 11.4 Changing File Ownership 317 11.5 Other Real-World Issues 321 11.6 Summary 323 Table of Contents | v 12. Spellchecking . 325 12.1 The spell Program 325 12.2 The Original Unix Spellchecking Prototype 326 12.3 Improving ispell and aspell 327 12.4 A Spellchecker in awk 331 12.5 Summary 350 13. Processes . 352 13.1 Process Creation 353 13.2 Process Listing 354 13.3 Process Control and Deletion 360 13.4 Process System-Call Tracing 368 13.5 Process Accounting 372 13.6 Delayed Scheduling of Processes 373 13.7 The /proc Filesystem 378 13.8 Summary 379 14. Shell Portability Issues and Extensions . 381 14.1 Gotchas 381 14.2 The bash shopt Command 385 14.3 Common Extensions 389 14.4 Download Information 402 14.5 Other Extended Bourne-Style Shells 405 14.6 Shell Versions 405 14.7 Shell Initialization and Termination 406 14.8 Summary 412 15. Secure Shell Scripts: Getting Started . 413 15.1 Tips for Secure Shell Scripts 413 15.2 Restricted Shell 416 15.3 Trojan Horses 418 15.4 Setuid Shell Scripts: A Bad Idea 419 15.5 ksh93 and Privileged Mode 421 15.6 Summary 422 vi | Table of Contents A. Writing Manual Pages . 423 B. Files and Filesystems . 437 C. Important Unix Commands . 473 Bibliography . 478 Glossary . 484 Index . 509 Table of Contents | vii Foreword Surely I haven’t been doing shell scripting for 30 years?!? Well,now that I think about it,I suppose I have,although it was only in a small way at first. (The early Unix shells,before the Bourne shell,were very primitive by modern standards,and writing substantial scripts was difficult. Fortunately, things quickly got better.) In recent years,the shell has been neglected and underappreciated as a scripting lan- guage. But even though it was Unix’s first scripting language,it’s still one of the best. Its combination of extensibility and efficiency remains unique,and the improve- ments made to it over the years have kept it highly competitive with other scripting languages that have gotten a lot more hype. GUIs are more fashionable than com- mand-line shells as user interfaces these days,but scripting languages often provide most of the underpinnings for the fancy screen graphics,and the shell continues to excel in that role. The shell’s dependence on other programs to do most of the work is arguably a defect,but also inarguably a strength: you get the concise notation of a scripting lan- guage plus the speed and efficiency of programs written in C (etc.). Using a com- mon,general-purpose data representation—lines of text—in a large (and extensible) set of tools lets the scripting language plug the tools together in endless combina- tions. The result is far more flexibility and power than any monolithic software pack- age with a built-in menu item for (supposedly) everything you might want. The early success of the shell in taking this approach reinforced the developing Unix philoso- phy of building specialized,single-purpose tools and plugging them together to do the job. The philosophy in turn encouraged improvements in the shell to allow doing more jobs that way. Shell scripts also have an advantage over C programs—and over some of the other scripting languages too (naming no names!)—of generally being fairly easy to read and modify. Even people who are not C programmers,like a good many system administrators these days,typically feel comfortable with shell scripts. This makes shell scripting very important for extending user environments and for customizing software packages. ix This is the Title of the Book, eMatter Edition Indeed,there’s a “wheel of reincarnation” here,which I’ve seen on several software projects. The project puts simple shell scripts in key places,to make it easy for users to customize aspects of the software. However,it’s so much easier for the project to solve problems by working in those shell scripts than in the surrounding C code,that the scripts steadily get more complicated. Eventually they are too complicated for the users to cope with easily (some of the scripts we wrote in the C News project were notorious as stress tests for shells,never mind users!),and a new set of scripts has to be provided for user customization… For a long time,there’s been a conspicuous lack of a good book on shell scripting. Books on the Unix programming environment have touched on it,but only briefly, as one of several topics,and the better books are long out-of-date. There’s reference documentation for the various shells,but what’s wanted is a novice-friendly tutorial, covering the tools as well as the shell,introducing the concepts gently,offering advice on how to get the best results,and paying attention to practical issues like readability. Preferably,it should also discuss how the various shells differ,instead of trying to pretend that only one exists. This book delivers all that,and more. Here,at last,is an up-to-date and painless introduction to the first and best of the Unix scripting languages. It’s illustrated with realistic examples that make useful tools in their own right.