DOCSLIB.ORG

Advanced Bash-Scripting Guide

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Advanced Bash-Scripting Guide Advanced Bash−Scripting Guide An in−depth exploration of the gentle art of shell scripting Mendel Cooper Brindle−Phlogiston Associates [email protected] 13 July 2002 Revision History Revision 0.1 14 June 2000 Revised by: mc Initial release. Revision 0.2 30 October 2000 Revised by: mc Bugs fixed, plus much additional material and more example scripts. Revision 0.3 12 February 2001 Revised by: mc Another major update. Revision 0.4 08 July 2001 Revised by: mc More bugfixes, much more material, more scripts − a complete revision and expansion of the book. Revision 0.5 03 September 2001 Revised by: mc Major update. Bugfixes, material added, chapters and sections reorganized. Revision 1.0 14 October 2001 Revised by: mc Bugfixes, reorganization, material added. Stable release. Revision 1.1 06 January 2002 Revised by: mc Bugfixes, material and scripts added. Revision 1.2 31 March 2002 Revised by: mc Bugfixes, material and scripts added. Revision 1.3 02 June 2002 Revised by: mc 'TANGERINE' release: A few bugfixes, much more material and scripts added. Revision 1.4 16 June 2002 Revised by: mc 'MANGO' release: Quite a number of typos fixed, more material and scripts added. Revision 1.5 13 July 2002 Revised by: mc 'PAPAYA' release: A few bugfixes, much more material and scripts added. This tutorial assumes no previous knowledge of scripting or programming, but progresses rapidly toward an intermediate/advanced level of instruction ...all the while sneaking in little snippets of UNIX wisdom and lore. It serves as a textbook, a manual for self−study, and a reference and source of knowledge on shell scripting techniques. The exercises and heavily−commented examples invite active reader participation, under the premise that the only way to really learn scripting is to write scripts. The latest update of this document, as an archived, bzip2−ed "tarball" including both the SGML source and rendered HTML, may be downloaded from the author's home site. See the change log for a revision history. Dedication For Anita, the source of all the magic Advanced Bash−Scripting Guide Table of Contents Chapter 1. Why Shell Programming?...............................................................................................................1 Chapter 2. Starting Off With a Sha−Bang.......................................................................................................3 2.1. Invoking the script............................................................................................................................5 2.2. Preliminary Exercises.......................................................................................................................6 Part 2. Basics............................................................................................................................................6 Chapter 3. Exit and Exit Status.........................................................................................................................7 Chapter 4. Special Characters...........................................................................................................................9 Chapter 5. Introduction to Variables and Parameters..................................................................................27 5.1. Variable Substitution......................................................................................................................27 5.2. Variable Assignment.......................................................................................................................29 5.3. Bash Variables Are Untyped..........................................................................................................31 5.4. Special Variable Types...................................................................................................................32 Chapter 6. Quoting...........................................................................................................................................36 Chapter 7. Tests................................................................................................................................................42 7.1. Test Constructs...............................................................................................................................42 7.2. File test operators............................................................................................................................48 7.3. Comparison operators (binary).......................................................................................................51 7.4. Nested if/then Condition Tests.......................................................................................................56 7.5. Testing Your Knowledge of Tests..................................................................................................57 Chapter 8. Operations and Related Topics....................................................................................................58 8.1. Operators.........................................................................................................................................58 8.2. Numerical Constants.......................................................................................................................65 Part 3. Beyond the Basics......................................................................................................................66 Chapter 9. Variables Revisited........................................................................................................................67 9.1. Internal Variables............................................................................................................................67 9.2. Manipulating Strings.......................................................................................................................83 9.2.1. Manipulating strings using awk......................................................................................87 9.2.2. Further Discussion..........................................................................................................88 9.3. Parameter Substitution....................................................................................................................88 9.4. Typing variables: declare or typeset..............................................................................................97 9.5. Indirect References to Variables.....................................................................................................98 9.6. $RANDOM: generate random integer..........................................................................................100 9.7. The Double Parentheses Construct...............................................................................................105 Chapter 10. Loops and Branches..................................................................................................................107 10.1. Loops..........................................................................................................................................107 10.2. Nested Loops..............................................................................................................................117 10.3. Loop Control...............................................................................................................................118 10.4. Testing and Branching................................................................................................................120 i Advanced Bash−Scripting Guide Table of Contents Chapter 11. Internal Commands and Builtins.............................................................................................128 11.1. Job Control Commands..............................................................................................................146 Chapter 12. External Filters, Programs and Commands...........................................................................151 12.1. Basic Commands........................................................................................................................151 12.2. Complex Commands...................................................................................................................154 12.3. Time / Date Commands..............................................................................................................161 12.4. Text Processing Commands........................................................................................................164 12.5. File and Archiving Commands...................................................................................................181 12.6. Communications Commands......................................................................................................197 12.7. Terminal Control Commands.....................................................................................................202 12.8. Math Commands.........................................................................................................................203 12.9. Miscellaneous Commands..........................................................................................................212 Chapter 13. System and Administrative Commands..................................................................................222 Chapter 14. Command Substitution.............................................................................................................246
Load more

Recommended publications
  • At—At, Batch—Execute Commands at a Later Time
    at—at, batch—execute commands at a later time at [–csm] [–f script] [–qqueue] time [date] [+ increment] at –l [ job...] at –r job... batch at and batch read commands from standard input to be executed at a later time. at allows you to specify when the commands should be executed, while jobs queued with batch will execute when system load level permits. Executes commands read from stdin or a file at some later time. Unless redirected, the output is mailed to the user. Example A.1 1 at 6:30am Dec 12 < program 2 at noon tomorrow < program 3 at 1945 pm August 9 < program 4 at now + 3 hours < program 5 at 8:30am Jan 4 < program 6 at -r 83883555320.a EXPLANATION 1. At 6:30 in the morning on December 12th, start the job. 2. At noon tomorrow start the job. 3. At 7:45 in the evening on August 9th, start the job. 4. In three hours start the job. 5. At 8:30 in the morning of January 4th, start the job. 6. Removes previously scheduled job 83883555320.a. awk—pattern scanning and processing language awk [ –fprogram–file ] [ –Fc ] [ prog ] [ parameters ] [ filename...] awk scans each input filename for lines that match any of a set of patterns specified in prog. Example A.2 1 awk '{print $1, $2}' file 2 awk '/John/{print $3, $4}' file 3 awk -F: '{print $3}' /etc/passwd 4 date | awk '{print $6}' EXPLANATION 1. Prints the first two fields of file where fields are separated by whitespace. 2. Prints fields 3 and 4 if the pattern John is found.
    [Show full text]
  • Shell Scripting with Bash
    Introduction to Shell Scripting with Bash Charles Jahnke Research Computing Services Information Services & Technology Topics for Today ● Introductions ● Basic Terminology ● How to get help ● Command-line vs. Scripting ● Variables ● Handling Arguments ● Standard I/O, Pipes, and Redirection ● Control Structures (loops and If statements) ● SCC Job Submission Example Research Computing Services Research Computing Services (RCS) A group within Information Services & Technology at Boston University provides computing, storage, and visualization resources and services to support research that has specialized or highly intensive computation, storage, bandwidth, or graphics requirements. Three Primary Services: ● Research Computation ● Research Visualization ● Research Consulting and Training Breadth of Research on the Shared Computing Cluster (SCC) Me ● Research Facilitator and Administrator ● Background in biomedical engineering, bioinformatics, and IT systems ● Offices on both CRC and BUMC ○ Most of our staff on the Charles River Campus, some dedicated to BUMC ● Contact: [email protected] You ● Who has experience programming? ● Using Linux? ● Using the Shared Computing Cluster (SCC)? Basic Terminology The Command-line The line on which commands are typed and passed to the shell. Username Hostname Current Directory [username@scc1 ~]$ Prompt Command Line (input) The Shell ● The interface between the user and the operating system ● Program that interprets and executes input ● Provides: ○ Built-in commands ○ Programming control structures ○ Environment
    [Show full text]
  • The Price of Safety: Evaluating IOMMU Performance Preliminary Results
    The Price of Safety: Evaluating IOMMU Performance Preliminary Results Muli Ben-Yehuda [email protected] IBM Haifa Research Lab The Price of Safety: Evaluating IOMMU Performance, 2007 Spring Xen Summit – p.1/14 Table of Contents IOMMUs Preliminary Performance Results Optimizations and Trade-offs Conclusions The Price of Safety: Evaluating IOMMU Performance, 2007 Spring Xen Summit – p.2/14 IOMMUs IOMMU - IO Memory Management Unit Think MMU for IO devices - separate address spaces! IOMMUs enable virtual machines direct hardware access for PV and FV guests ... while protecting the rest of the system from mis-behaving or malicious virtual machines But at what cost? The Price of Safety: Evaluating IOMMU Performance, 2007 Spring Xen Summit – p.3/14 Setting the Stage Full results to appear at OLS ’07 Calgary (PCI-X) and Calgary (PCI-e) IOMMUs Linux and Xen implementations ... with some corroborating evidence from the DART IOMMU on PPC JS21 blades and the IBM Research Hypervisor Joint work with Jimi Xenidis, Michal Ostrowski, Karl Rister, Alexis Bruemmer and Leendert Van Doorn Utilizing IOMMUs for Virtualization in Linux and Xen, by by M. Ben-Yehuda, J. Mason, O. Krieger, J. Xenidis, L. Van Doorn, A. Mallick, J. Nakajima, and E. Wahlig, OLS ’06 The Price of Safety: Evaluating IOMMU Performance, 2007 Spring Xen Summit – p.4/14 On Comparisons What are we comparing against? not emulation not virtual IO (frontend / backend drivers) Direct hardware access without an IOMMU compared with direct hardware access with an IOMMU on the IO path The Price of Safety:
    [Show full text]
  • Bash Crash Course + Bc + Sed + Awk∗
    Bash Crash course + bc + sed + awk∗ Andrey Lukyanenko, CSE, Aalto University Fall, 2011 There are many Unix shell programs: bash, sh, csh, tcsh, ksh, etc. The comparison of those can be found on-line 1. We will primary focus on the capabilities of bash v.4 shell2. 1. Each bash script can be considered as a text file which starts with #!/bin/bash. It informs the editor or interpretor which tries to open the file, what to do with the file and how should it be treated. The special character set in the beginning #! is a magic number; check man magic and /usr/share/file/magic on existing magic numbers if interested. 2. Each script (assume you created “scriptname.sh file) can be invoked by command <dir>/scriptname.sh in console, where <dir> is absolute or relative path to the script directory, e.g., ./scriptname.sh for current directory. If it has #! as the first line it will be invoked by this command, otherwise it can be called by command bash <dir>/scriptname.sh. Notice: to call script as ./scriptname.sh it has to be executable, i.e., call command chmod 555 scriptname.sh before- hand. 3. Variable in bash can be treated as integers or strings, depending on their value. There are set of operations and rules available for them. For example: #!/bin/bash var1=123 # Assigns value 123 to var1 echo var1 # Prints ’var1’ to output echo $var1 # Prints ’123’ to output var2 =321 # Error (var2: command not found) var2= 321 # Error (321: command not found) var2=321 # Correct var3=$var2 # Assigns value 321 from var2 to var3 echo $var3 # Prints ’321’ to output
    [Show full text]
  • Filesystem Hierarchy Standard
    Filesystem Hierarchy Standard LSB Workgroup, The Linux Foundation Filesystem Hierarchy Standard LSB Workgroup, The Linux Foundation Version 3.0 Publication date March 19, 2015 Copyright © 2015 The Linux Foundation Copyright © 1994-2004 Daniel Quinlan Copyright © 2001-2004 Paul 'Rusty' Russell Copyright © 2003-2004 Christopher Yeoh Abstract This standard consists of a set of requirements and guidelines for file and directory placement under UNIX-like operating systems. The guidelines are intended to support interoperability of applications, system administration tools, development tools, and scripts as well as greater uniformity of documentation for these systems. All trademarks and copyrights are owned by their owners, unless specifically noted otherwise. Use of a term in this document should not be regarded as affecting the validity of any trademark or service mark. Permission is granted to make and distribute verbatim copies of this standard provided the copyright and this permission notice are preserved on all copies. Permission is granted to copy and distribute modified versions of this standard under the conditions for verbatim copying, provided also that the title page is labeled as modified including a reference to the original standard, provided that information on retrieving the original standard is included, and provided that the entire resulting derived work is distributed under the terms of a permission notice identical to this one. Permission is granted to copy and distribute translations of this standard into another language, under the above conditions for modified versions, except that this permission notice may be stated in a translation approved by the copyright holder. Dedication This release is dedicated to the memory of Christopher Yeoh, a long-time friend and colleague, and one of the original editors of the FHS.
    [Show full text]
  • Evaluating UNIX Security Previous Screen Allen B
    84-01-15.1 Evaluating UNIX Security Previous screen Allen B. Lum Payoff The UNIX operating system's basic security features include password protection, access permission, user profiles, shell scripts, and file ownership. Because new and enhanced UNIX security features are continually being added to these features in response to the demands of an increasingly competitive user community, information security professionals must begin with an understanding of the basic UNIX security environment. This article covers the fundamental security features that can be found in most of the currently available versions of the UNIX operating system. Several checklists are included at the end of the article to assist administrators in ensuring the security of UNIX systems. Problems Addressed The UNIX operating system was originally developed for use by programmers within an open systems environment. The adoption of UNIX as a common operating system across several different platforms has increased the need for security beyond its original purpose. As a result, many UNIX installations have less than optimal security. In addition, there are several versions of UNIX on the market today with differing security features. This article discusses basic access controls (e.g., passwords) and directory and file permissions within the UNIX system. The concepts discussed are applicable to all versions of UNIX unless specifically noted otherwise. UNIX History As the majority of UNIX users know, UNIX was developed at AT Bell Laboratories in the late 1960s by Thompson and Ritchie; the name UNIXis a contraction of uni and multics. The original UNIX system software was written in the assembler language to run on the digital PDP-7 computer.
    [Show full text]
  • Chapter 3 Unix Overview
    Chapter 3 Unix Overview Figure 3.1 Unix file system Directory Purpose / The root directory /bin or /sbin Critical executables needed to boot the system /dev Device drivers /etc System configuration files such as passwords, network addresses and names,system startup scripts /home User home directories /lib Shared libraries used by programs /mnt Temporary mount point for file systems /proc Images of currently executing processes on the system /tmp Temporary files /usr A variety of critical system files, including system utilities (/usr/bin), and administration executables (/usr/sbin) /var Stores varying files such as /var/log, /var/mail Table 3.1 Important Directories in the Unix file system Figure 3.2 Unix Architecture Figure 3.3 Relationship between init, inetd, and various network services Sample /etc/inetd.conf file containing services spawned by inetd /etc/inetd.conf file format • Service name (port # defined in /etc/services) • Socket type (stream or dgram) • Protocol (tcp, udp, rpc/tcp, or rpc/udp) • Wait status (wait or nowait) • Username (service run as) • Server program • Server program arguments Use of inetd.conf to create backdoor listeners and attack relays Common Unix Administration Tasks ♦ Vulnerability of using “.” in your search path $PATH ♦ Showing all running processes ps –aux ps –aef ♦ Killing/restarting processes kill –HUP pid killall –HUP inetd ♦ /etc/passwd file ♦ Unix permissions rwxrwxrwx chmod command Common Unix Administration Tasks (cont.) ♦ SetUID programs – Executes with permissions of its owner, not of its
    [Show full text]
  • Shell Script Getopts Example
    Shell Script Getopts Example Gail is constrainedly cryoscopic after delegable Hilbert sag his parcloses illuminatingly. Gonzales often tootle irresistibly when tripersonal Giordano discomposed dissonantly and buffer her Barbarossa. Robert misdraws rompishly. Find that getopts script example we use later will contain whitespace separation with command Instantly share code, notes, and snippets. OPTARG is set to the period character found. They cease in many cases unneeded and brief fact that cartoon always press them its just like matter of personal coding style. File or directory or found. Operator precedence is used when there within five is more arguments. Typically, shell scripts use getopts to parse arguments passed to them. There did many ways to against your remedy environment. Nothing gets printed when no command line options are provided. The return status is zero unless an outcome is encountered while determining the name avid the he directory within an invalid option is supplied. The exit code will be success failure. Log in charge use details from one require these accounts. How does log lumber and cpu usage by an application? Now consider running it bore an unsupported option. No need only pass the positional parameters through to borrow external program. How can I check took a directory exists in a candy shell script? What extent its purpose? When optional, the fashion can buckle on led off additional functionality, as ugly a boolean option. In those cases, it contains a pointer to that parameter. How environment check ride a variable is set to Bash? Connect and deploy knowledge write a single location that is structured and fatigue to search.
    [Show full text]
  • Research Computing and Cyberinfrastructure Team
    Research Computing and CyberInfrastructure team ! Working with the Linux Shell on the CWRU HPC 31 January 2019 7 February 2019 ! Presenter Emily Dragowsky KSL Data Center RCCI Team: Roger Bielefeld, Mike Warfe, Hadrian Djohari! Brian Christian, Emily Dragowsky, Jeremy Fondran, Cal Frye,! Sanjaya Gajurel, Matt Garvey, Theresa Griegger, Cindy Martin, ! Sean Maxwell, Jeno Mozes, Nasir Yilmaz, Lee Zickel Preface: Prepare your environment • User account ! # .bashrc ## cli essentials ! if [ -t 1 ] then bind '"\e[A": history-search-backward' bind '"\e[B": history-search-forward' bind '"\eOA": history-search-backward' bind '"\eOB": history-search-forward' fi ! This is too useful to pass up! Working with Linux • Preamble • Intro Session Linux Review: Finding our way • Files & Directories: Sample work flow • Shell Commands • Pipes & Redirection • Scripting Foundations • Shell & environment variables • Control Structures • Regular expressions & text manipulation • Recap & Look Ahead Rider Cluster Components ! rider.case.edu ondemand.case.edu University Firewall ! Admin Head Nodes SLURM Science Nodes Master DMZ Resource ! Data Manager Transfer Disk Storage Nodes Batch nodes GPU nodes SMP nodes Running a job: where is it? slide from Hadrian Djohari Storing Data on the HPC table from Nasir Yilmaz How data moves across campus • Buildings on campus are each assigned to a zone. Data connections go from every building to the distribution switch at the center of the zone and from there to the data centers at Kelvin Smith Library and Crawford Hall. slide
    [Show full text]
  • Linux Performance Tools
    Linux Performance Tools Brendan Gregg Senior Performance Architect Performance Engineering Team [email protected] @brendangregg This Tutorial • A tour of many Linux performance tools – To show you what can be done – With guidance for how to do it • This includes objectives, discussion, live demos – See the video of this tutorial Observability Benchmarking Tuning Stac Tuning • Massive AWS EC2 Linux cloud – 10s of thousands of cloud instances • FreeBSD for content delivery – ~33% of US Internet traffic at night • Over 50M subscribers – Recently launched in ANZ • Use Linux server tools as needed – After cloud monitoring (Atlas, etc.) and instance monitoring (Vector) tools Agenda • Methodologies • Tools • Tool Types: – Observability – Benchmarking – Tuning – Static • Profiling • Tracing Methodologies Methodologies • Objectives: – Recognize the Streetlight Anti-Method – Perform the Workload Characterization Method – Perform the USE Method – Learn how to start with the questions, before using tools – Be aware of other methodologies My system is slow… DEMO & DISCUSSION Methodologies • There are dozens of performance tools for Linux – Packages: sysstat, procps, coreutils, … – Commercial products • Methodologies can provide guidance for choosing and using tools effectively • A starting point, a process, and an ending point An#-Methodologies • The lack of a deliberate methodology… Street Light An<-Method 1. Pick observability tools that are: – Familiar – Found on the Internet – Found at random 2. Run tools 3. Look for obvious issues Drunk Man An<-Method • Tune things at random until the problem goes away Blame Someone Else An<-Method 1. Find a system or environment component you are not responsible for 2. Hypothesize that the issue is with that component 3. Redirect the issue to the responsible team 4.
    [Show full text]
  • The Complete Freebsd
    The Complete FreeBSD® If you find errors in this book, please report them to Greg Lehey <grog@Free- BSD.org> for inclusion in the errata list. The Complete FreeBSD® Fourth Edition Tenth anniversary version, 24 February 2006 Greg Lehey The Complete FreeBSD® by Greg Lehey <[email protected]> Copyright © 1996, 1997, 1999, 2002, 2003, 2006 by Greg Lehey. This book is licensed under the Creative Commons “Attribution-NonCommercial-ShareAlike 2.5” license. The full text is located at http://creativecommons.org/licenses/by-nc-sa/2.5/legalcode. You are free: • to copy, distribute, display, and perform the work • to make derivative works under the following conditions: • Attribution. You must attribute the work in the manner specified by the author or licensor. • Noncommercial. You may not use this work for commercial purposes. This clause is modified from the original by the provision: You may use this book for commercial purposes if you pay me the sum of USD 20 per copy printed (whether sold or not). You must also agree to allow inspection of printing records and other material necessary to confirm the royalty sums. The purpose of this clause is to make it attractive to negotiate sensible royalties before printing. • Share Alike. If you alter, transform, or build upon this work, you may distribute the resulting work only under a license identical to this one. • For any reuse or distribution, you must make clear to others the license terms of this work. • Any of these conditions can be waived if you get permission from the copyright holder. Your fair use and other rights are in no way affected by the above.
    [Show full text]
  • 10Gb TR-D 10GBAS B/S SFP+ Dxxxl-N00 E-LR/LW Bidi Opti 0
    TR-DXxxL-N00 Rev1.6 10Gb/s SFP+ BiDi Optical Tran sceiver TR-DXxxL-N00, 10Km SMF Application 10GBASE-LR/LW Bi-directional, LC connector Features 10Gb/sserialopticalinterfacecompliantto 802.3ae10GBASE-LR,singleLCconnector forbi-directionalapplication,over10km SMF Electrical interface compliant to SFF-8431 specifications 1270/1330nm DFB transmitter, PINphoto- detector,integratedWDM 2-wireinterfaceformanagement specificationscompliantwithSFF8472 Partnumber(0°Cto70°C): Applications – TR-DX12L-N00,1270TX/1330RX High speed storage area – TR-DX33L-N00,1330TX/1270RX networks Lineside,clientsideloopbackfunction; Computerclustercross-connect Advancedfirmwareallowcustomersystem Customhigh-speeddatapipes encryption information to be stored in transceiver ROHScompliant Figure11: Application in System InnoLight Technology Corp. Page1of13 TR-DXxxL-N00 Rev1.6 1. GENERAL DESCRIPTION This10GigabitSFP+BiDitransceiverisdesignedtotransmitandreceiveopticaldata oversinglemodeopticalfiberforlinklength20km. TheSFP+BiDimoduleelectricalinterfaceiscomplianttoSFIelectricalspecifications. Thetransmitterinputandreceiveroutputimpedanceis100Ohmsdifferential.Data linesareinternally AC coupled. The module provides differential termination and reduce differential to common mode conversion for quality signal termination and lowEMI.SFItypicallyoperatesover200mmofimprovedFR4materialoruptoabout 150mmofstandardFR4withoneconnector. Thetransmitterconverts10Gbit/sserialPECLorCMLelectricaldataintoserialoptical data compliant with the 10GBASE-LR standard. An open collector
    [Show full text]