DOCSLIB.ORG

Advanced Bash-Scripting Guide

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Advanced Bash-Scripting Guide Advanced Bash−Scripting Guide An in−depth exploration of the art of shell scripting Mendel Cooper <[email protected]> 2.5 15 February 2004 Revision History Revision 2.3 03 January 2004 Revised by: mc 'STRAWBERRY' release: Major update. Revision 2.4 25 January 2004 Revised by: mc 'MUSKMELON' release: Bugfixes. Revision 2.5 15 February 2004 Revised by: mc 'STARFRUIT' release: Bugfixes and more material. This tutorial assumes no previous knowledge of scripting or programming, but progresses rapidly toward an intermediate/advanced level of instruction . all the while sneaking in little snippets of Unix® wisdom and lore. It serves as a textbook, a manual for self−study, and a reference and source of knowledge on shell scripting techniques. The exercises and heavily−commented examples invite active reader participation, under the premise that the only way to really learn scripting is to write scripts. This book is suitable for classroom use as a general introduction to programming concepts. The latest update of this document, as an archived, bzip2−ed "tarball" including both the SGML source and rendered HTML, may be downloaded from the author's home site. See the change log for a revision history. Dedication For Anita, the source of all the magic Advanced Bash−Scripting Guide Table of Contents Chapter 1. Why Shell Programming?...............................................................................................................1 Chapter 2. Starting Off With a Sha−Bang.......................................................................................................3 2.1. Invoking the script............................................................................................................................5 2.2. Preliminary Exercises.......................................................................................................................5 Part 2. Basics.......................................................................................................................................................6 Chapter 3. Special Characters...........................................................................................................................7 Chapter 4. Introduction to Variables and Parameters..................................................................................23 4.1. Variable Substitution......................................................................................................................23 4.2. Variable Assignment.......................................................................................................................25 4.3. Bash Variables Are Untyped..........................................................................................................26 4.4. Special Variable Types...................................................................................................................28 Chapter 5. Quoting...........................................................................................................................................32 Chapter 6. Exit and Exit Status.......................................................................................................................38 Chapter 7. Tests................................................................................................................................................40 7.1. Test Constructs...............................................................................................................................40 7.2. File test operators............................................................................................................................46 7.3. Comparison operators (binary).......................................................................................................49 7.4. Nested if/then Condition Tests.......................................................................................................54 7.5. Testing Your Knowledge of Tests..................................................................................................54 Chapter 8. Operations and Related Topics....................................................................................................55 8.1. Operators.........................................................................................................................................55 8.2. Numerical Constants.......................................................................................................................61 Part 3. Beyond the Basics.................................................................................................................................63 Chapter 9. Variables Revisited........................................................................................................................64 9.1. Internal Variables............................................................................................................................64 9.2. Manipulating Strings.......................................................................................................................80 9.2.1. Manipulating strings using awk............................................................................................84 9.2.2. Further Discussion.................................................................................................................85 9.3. Parameter Substitution....................................................................................................................85 9.4. Typing variables: declare or typeset...............................................................................................93 9.5. Indirect References to Variables.....................................................................................................95 9.6. $RANDOM: generate random integer............................................................................................97 9.7. The Double Parentheses Construct...............................................................................................106 Chapter 10. Loops and Branches..................................................................................................................108 10.1. Loops..........................................................................................................................................108 10.2. Nested Loops..............................................................................................................................118 10.3. Loop Control...............................................................................................................................119 i Advanced Bash−Scripting Guide Table of Contents Chapter 10. Loops and Branches 10.4. Testing and Branching................................................................................................................122 Chapter 11. Internal Commands and Builtins.............................................................................................130 11.1. Job Control Commands..............................................................................................................151 Chapter 12. External Filters, Programs and Commands...........................................................................155 12.1. Basic Commands........................................................................................................................155 12.2. Complex Commands...................................................................................................................159 12.3. Time / Date Commands..............................................................................................................167 12.4. Text Processing Commands........................................................................................................170 12.5. File and Archiving Commands...................................................................................................186 12.6. Communications Commands......................................................................................................201 12.7. Terminal Control Commands.....................................................................................................205 12.8. Math Commands.........................................................................................................................206 12.9. Miscellaneous Commands..........................................................................................................215 Chapter 13. System and Administrative Commands..................................................................................225 Chapter 14. Command Substitution.............................................................................................................248 Chapter 15. Arithmetic Expansion................................................................................................................253 Chapter 16. I/O Redirection...........................................................................................................................254 16.1. Using exec...................................................................................................................................256 16.2. Redirecting Code Blocks............................................................................................................259 16.3. Applications................................................................................................................................263
Load more

Recommended publications
  • The Linux Command Line
    The Linux Command Line Fifth Internet Edition William Shotts A LinuxCommand.org Book Copyright ©2008-2019, William E. Shotts, Jr. This work is licensed under the Creative Commons Attribution-Noncommercial-No De- rivative Works 3.0 United States License. To view a copy of this license, visit the link above or send a letter to Creative Commons, PO Box 1866, Mountain View, CA 94042. A version of this book is also available in printed form, published by No Starch Press. Copies may be purchased wherever fine books are sold. No Starch Press also offers elec- tronic formats for popular e-readers. They can be reached at: https://www.nostarch.com. Linux® is the registered trademark of Linus Torvalds. All other trademarks belong to their respective owners. This book is part of the LinuxCommand.org project, a site for Linux education and advo- cacy devoted to helping users of legacy operating systems migrate into the future. You may contact the LinuxCommand.org project at http://linuxcommand.org. Release History Version Date Description 19.01A January 28, 2019 Fifth Internet Edition (Corrected TOC) 19.01 January 17, 2019 Fifth Internet Edition. 17.10 October 19, 2017 Fourth Internet Edition. 16.07 July 28, 2016 Third Internet Edition. 13.07 July 6, 2013 Second Internet Edition. 09.12 December 14, 2009 First Internet Edition. Table of Contents Introduction....................................................................................................xvi Why Use the Command Line?......................................................................................xvi
    [Show full text]
  • Shell Variables
    Shell Using the command line Orna Agmon ladypine at vipe.technion.ac.il Haifux Shell – p. 1/55 TOC Various shells Customizing the shell getting help and information Combining simple and useful commands output redirection lists of commands job control environment variables Remote shell textual editors textual clients references Shell – p. 2/55 What is the shell? The shell is the wrapper around the system: a communication means between the user and the system The shell is the manner in which the user can interact with the system through the terminal. The shell is also a script interpreter. The simplest script is a bunch of shell commands. Shell scripts are used in order to boot the system. The user can also write and execute shell scripts. Shell – p. 3/55 Shell - which shell? There are several kinds of shells. For example, bash (Bourne Again Shell), csh, tcsh, zsh, ksh (Korn Shell). The most important shell is bash, since it is available on almost every free Unix system. The Linux system scripts use bash. The default shell for the user is set in the /etc/passwd file. Here is a line out of this file for example: dana:x:500:500:Dana,,,:/home/dana:/bin/bash This line means that user dana uses bash (located on the system at /bin/bash) as her default shell. Shell – p. 4/55 Starting to work in another shell If Dana wishes to temporarily use another shell, she can simply call this shell from the command line: [dana@granada ˜]$ bash dana@granada:˜$ #In bash now dana@granada:˜$ exit [dana@granada ˜]$ bash dana@granada:˜$ #In bash now, going to hit ctrl D dana@granada:˜$ exit [dana@granada ˜]$ #In original shell now Shell – p.
    [Show full text]
  • UNIX/LINUX Command Reference Sheet
    UNIX/LINUX Command Reference Sheet File Commands Process Management cat – concatenates and displays files at/batch– execute commands at a later time cat > file – places standard input into file bg – lists stopped or background jobs; resume a cd – change current directory stopped job in the background cmp – compares two files cron – clock daemon that executes commands at cp – copies a file or a directory specified dates and times cut – removes columns or fields from the specified crontab – modifies the user's crontab file, which is file(s) used by cron diff – displays line‐by‐line differences between two fg – brings the specified job to foreground text files kill pid – kill process id pid head – displays the first lines of the specified file(s) killall proc – kill all processes named proc1 join – joins two files nice – invoke a command with an altered scheduling jot – print sequential or random data priority ln – make hard or symbolic links to files ps – display your currently active processes ls -al – formatted listing with hidden files renice – alter priority of running processes ls – directory listing top – display all running processes mkdir dir – create a directory dir more/less – output the contents of the specified File & Group Permissions file(s) in pages chmod octal file – change the permissions of mv – rename or move a file file to octal, which can be found separately for nl – put line numbers on a file user, group, and world by adding: od – sequentially copies each file to standard output, • 4 – read (r) displaying it according to the specified arguments; for • 2 – write (w) example, in hexadecimal (often useful in conjunction • 1 – execute (x) with diff) Examples: paste – merge same lines of several files or chmod 777 – read, write, execute for all subsequent lines of one file chmod 755 – rwx for owner, rx for group and world pr – produces a printed listing of one or more files For more options, see man chmod.
    [Show full text]
  • The Linux Command Line
    The Linux Command Line Second Internet Edition William E. Shotts, Jr. A LinuxCommand.org Book Copyright ©2008-2013, William E. Shotts, Jr. This work is licensed under the Creative Commons Attribution-Noncommercial-No De- rivative Works 3.0 United States License. To view a copy of this license, visit the link above or send a letter to Creative Commons, 171 Second Street, Suite 300, San Fran- cisco, California, 94105, USA. Linux® is the registered trademark of Linus Torvalds. All other trademarks belong to their respective owners. This book is part of the LinuxCommand.org project, a site for Linux education and advo- cacy devoted to helping users of legacy operating systems migrate into the future. You may contact the LinuxCommand.org project at http://linuxcommand.org. This book is also available in printed form, published by No Starch Press and may be purchased wherever fine books are sold. No Starch Press also offers this book in elec- tronic formats for most popular e-readers: http://nostarch.com/tlcl.htm Release History Version Date Description 13.07 July 6, 2013 Second Internet Edition. 09.12 December 14, 2009 First Internet Edition. 09.11 November 19, 2009 Fourth draft with almost all reviewer feedback incorporated and edited through chapter 37. 09.10 October 3, 2009 Third draft with revised table formatting, partial application of reviewers feedback and edited through chapter 18. 09.08 August 12, 2009 Second draft incorporating the first editing pass. 09.07 July 18, 2009 Completed first draft. Table of Contents Introduction....................................................................................................xvi
    [Show full text]
  • Basics of UNIX
    Basics of UNIX August 23, 2012 By UNIX, I mean any UNIX-like operating system, including Linux and Mac OS X. On the Mac you can access a UNIX terminal window with the Terminal application (under Applica- tions/Utilities). Most modern scientific computing is done on UNIX-based machines, often by remotely logging in to a UNIX-based server. 1 Connecting to a UNIX machine from {UNIX, Mac, Windows} See the file on bspace on connecting remotely to SCF. In addition, this SCF help page has infor- mation on logging in to remote machines via ssh without having to type your password every time. This can save a lot of time. 2 Getting help from SCF More generally, the department computing FAQs is the place to go for answers to questions about SCF. For questions not answered there, the SCF requests: “please report any problems regarding equipment or system software to the SCF staff by sending mail to ’trouble’ or by reporting the prob- lem directly to room 498/499. For information/questions on the use of application packages (e.g., R, SAS, Matlab), programming languages and libraries send mail to ’consult’. Questions/problems regarding accounts should be sent to ’manager’.” Note that for the purpose of this class, questions about application packages, languages, li- braries, etc. can be directed to me. 1 3 Files and directories 1. Files are stored in directories (aka folders) that are in a (inverted) directory tree, with “/” as the root of the tree 2. Where am I? > pwd 3. What’s in a directory? > ls > ls -a > ls -al 4.
    [Show full text]
  • Discussion Slides
    Discussion Session 03 Project 1 | Framing | Parsing Overview 1. Framing and Parsing 2. Generating HTTP Requests 3. Modular Server Code Calculator Example - Server Calculator Server - 1. Create a socket - socket(PF_INET, SOCK_STREAM, IPPROTO_TCP) 2. Bind the socket - bind(servSock, (struct sockaddr *) &servAddr, sizeof(servAddr) 3. Start Listening - listen(servSock, MAX_PENDING) 4. Loop and Accept Connections - accept(servSock, (struct sockaddr *) &echoClntAddr, &clntLen) Notes: 1. clntLen should be an unsigned int, or you can cast it to (socklen_t *) 2. bind() is also a function in std namespace. Use ::bind() for the sockets bind function Calculator Example - Client Request Format - ADD 10\r\n SUB 5\r\n SET 20\r\n ADD 30\r\n \r\n The server applies these operations on an accumulator which is set to 0 initially Framing Framing - Extracting a complete message based on delimiters Request1\r\n\r\nRequest2\r\n\r\nRequest3\r\n\r\nRequest4 Separate out into Request1, Request2, Request3 and Request4 Generating HTTP Request A sample HTTP request on TritonHTTP Spec - GET / HTTP/1.1<CR><LF> Host: www.cs.ucsd.edu<CR><LF> User-Agent: MyTester v1.0<CR><LF> Cookie: 123<CR><LF> <CR><LF> Here <CR> - Carriage Return (\r), <LF> - Line Feed (\n) Creating HTTP Request Script - unix2dos.sh #!/bin/bash if [ "$#" -ne 2 ]; then echo "Usage: $0 input-file.txt output-file.txt" exit 1 fi if [ ! -f $1 ]; then echo "File $1 not found" exit 2 fi sed 's/$'"/`echo \\\r`/" < $1 > $2 sh unix2dos.sh input_file.txt output_file.txt Creating HTTP Request - Example 1. Create a request in a text editor and save the file 2.
    [Show full text]
  • UNIX Commands Login and Exit Yppasswd Change Password Rlogin
    UNIX Commands Login and Exit yppasswd Change password rlogin machine Log into a remote machine telnet machine Log into a remote machine exit End a shell session Ctrl-D End shell session logout Log out of remote session Help man command Describes the command man -k keyword Search for keyword Display directory listing ls Display directory listing ls -l Display access permissions ls -a Display hidden files ls -d Display directory ls -t Display files sorted by time ls dir Display contents of directory ls file Display file ls ???* Display all files with more than 3 characters in name Change, Create, Remove Directories pwd Display current directory cd Change to your home directory cd .. Change to parent directory cd dir Change to another directory mkdir dir Create a new directory rmdir dir Remove a directory (must be empty) rm -r dir Remove a directory and everything within it Create, Copy, Move, Delete Files touch file Create an empty file cp src-file dst-file Copy a file to another file cp src-file dst-dir Copy a file to a directory cp -r * dst-dir Copy all files and sub-directories to another directory mv src-file dst-file Move a file to another file (renames file) mv src-file dst-dir Move a file to another directory mv src-dir dst-dir Move a directory to another directory Note: filenames can be up to 255 characters long but should not contain special characters, e.g., $ * [ ] & < > Change file permissions chmod o+r file Change file to allow read access to anyone chown username file Change onwer of file to username chgrp new-grp file Change
    [Show full text]
  • Designing Custom Gene Expression Arrays Using Earray
    eArray - custom microarray design web tool Gene Expression design tutorial Page 1 March 2007 Probe Design for Designing Custom Custom GX Arrays GX Arrays Using eArray 5.0 Print what you want when you want it Group/Presentation Title Agilent Confidential Page 2 Month ##, 200X eArray Overview Array Creation Process eArray provides an easy way of managing the array creation process. We are here! Search Agilent Probes Create Create Probe Submit to Microarray Order Arrays Groups Manufacturing Designs Upload Probes Download Design files Page 3 March 2007 Outline • Access your targeted organism’s transcriptome sequence from NCBI or other. •Format the data as needed into a standard FASTA format. •In eArray, select a design methodology and parameters • Uploading the transcriptome file • Download the results from eArray • Review the probe statistic results for possible culling •Create a probe group. •Create the microarray design. Page 4 March 2007 Access the Transcriptome of your Target Organism ex: Anopheles gambiae (African malaria mosquito) UniGene sequences downloaded from NCBI UniGene Build #35: Page 5 March 2007 Retrieve the file Gs.seq.uniq.gz from the NCBI unigene ftp repository (where Gs = Genus species) Page 6 March 2007 There are multiple links towards UniGene FTP site from the NCBI server The most direct way is to paste the link ftp://ftp.ncbi.nih.gov/repository/UniGene/ in your web browser. Alternatively, on the left lower corner of NCBI homepage, http://www.ncbi.nlm.nih.gov/, you will find a link “FTP sites”. Click it, you will be directed to a page with all FTP sites. The link, “UniGene”, leads you to the collection of all available UniGene sets.
    [Show full text]
  • Red Hat Enterprise Linux 7 Migration Planning Guide
    Red Hat Enterprise Linux 7 Migration Planning Guide Key differences between Red Hat Enterprise Linux 6 and Red Hat Enterprise Linux 7 Last Updated: 2021-09-21 Red Hat Enterprise Linux 7 Migration Planning Guide Key differences between Red Hat Enterprise Linux 6 and Red Hat Enterprise Linux 7 Legal Notice Copyright © 2021 Red Hat, Inc. The text of and illustrations in this document are licensed by Red Hat under a Creative Commons Attribution–Share Alike 3.0 Unported license ("CC-BY-SA"). An explanation of CC-BY-SA is available at http://creativecommons.org/licenses/by-sa/3.0/ . In accordance with CC-BY-SA, if you distribute this document or an adaptation of it, you must provide the URL for the original version. Red Hat, as the licensor of this document, waives the right to enforce, and agrees not to assert, Section 4d of CC-BY-SA to the fullest extent permitted by applicable law. Red Hat, Red Hat Enterprise Linux, the Shadowman logo, the Red Hat logo, JBoss, OpenShift, Fedora, the Infinity logo, and RHCE are trademarks of Red Hat, Inc., registered in the United States and other countries. Linux ® is the registered trademark of Linus Torvalds in the United States and other countries. Java ® is a registered trademark of Oracle and/or its affiliates. XFS ® is a trademark of Silicon Graphics International Corp. or its subsidiaries in the United States and/or other countries. MySQL ® is a registered trademark of MySQL AB in the United States, the European Union and other countries. Node.js ® is an official trademark of Joyent.
    [Show full text]
  • UNIX Commands Reference
    Mail Tools Special Command elm Sends and displays mail messages Line Characters Unix mailx Sends and displays mail messages > Redirect standard output to the file; erase mutt Sends and displays mail messages the file first Reference pine Sends and displays mail messages >> Redirect standard output to the file; xbiff Displays a mailbox and flag image on an append to the file X window when mail has arrived < Redirect standard input from the file Card | Pipe the standard output from one Text Editors command to the next A helpful reference to Unix commands on the CECS emacs Text Editor ? Single character wildcard Unix computers. gvim Graphical version of vim * Multiple character wildcard pico Text Editor [...] One of ... wild card vi Text Editor ; Command separator vim vi improved - Text Editor . Current directory xedit Simple Graphical Text Editor .. Parent directory xemacs Graphical version of emacs [gnu] / Directory level separator Text Processing Tools & Run command in the background http://www.cat.pdx.edu $ Begin shell variable name cat Concatenate (list) a file [email protected] = Shell variable assignment cut Displays specified parts from each line of a file dos2unix Convert text file from DOS format to Filename Expansion Program descriptions followed by [packageName] UNIX format Characters indicate which package the program is located in head List the beginning of a file ispell Spelling checker * Matches any characters (0 or more) Packages less Similar to more, but allows backward ? Matches any single character movement [...] matches any character in the enclosed list Packages configure the user's environment based on more List a file page by page or range what software he/she wants to use.
    [Show full text]
  • Intro to Cluster Computing
    Introduction to the Linux Command Line for High-Performance Computing Ken Weiss MSIS Computational Research Consulting Division Roadmap The command shell Navigating the filesystem Basic commands & wildcarding Shell redirection & pipelining Editing text files Permissions Processes Environment variables and customizing your session cja/kgw 2016 2 1/16 Poll How many are here to learn the command line: For HPC, specifically Flux, work? For other computing cluster work? For desktops/other Linux/Unix/Mac OS X? Other? cja 2014 3 10/14 Course Text William E Shotts, Jr., “The Linux Command Line: A Complete Introduction,” No Starch Press, January 2012. Download Creative Commons Licensed version at http://downloads.sourceforge.net/project/linuxcommand /TLCL/13.07/TLCL-13.07.pdf. cja 2014 4 10/14 The command shell cja 2014 5 10/14 What you are used to using kgw 2016 6 1/16 What you will be using kgw 2016 7 1/16 The command shell The command shell is an application that reads command lines from the keyboard and passes them to the Linux operating system to be executed. When you login to a remote Linux system, using a tool like ssh, you will automatically be connected to a shell. Your computing session is kept separate from other user’s computing sessions, because they are “enclosed” in a “shell”. On your desktop, laptop, or tablet, you may have to find and execute a terminal emulator application to bring up a shell in a window. cja 2014 8 10/14 The command shell cja 2014 9 10/14 http://askubuntu.com/questions/161511/are-the-linux-utilities-parts-of-the-kernel-shell
    [Show full text]
  • Jon K. Lærdahl, Structural Bioinforma�Cs
    Jon K. Lærdahl, Structural Bioinforma�cs Summing up Unix, Python, etc. Jon K. Lærdahl, Structural Bioinforma�cs Very important to be able to use basic Unix Make your own Unix scripts is not very central Most likely you will instead use Python for wri�ng small programs/scripts Jon K. Lærdahl, Structural Bioinforma�cs Main points: Demonstrate a prac�cal task where making a script is absolutely necessary Show you that some tasks cannot easily be solved by using «databases and web servers» -­‐ including Galaxy/LifePortal Jon K. Lærdahl, Structural Bioinforma�cs This is most important! Jon K. Lærdahl, Structural Bioinforma�cs Python Steep (or not?) learning curve Day 1 Day 2 Classes, objects and Biopython You are not expected to become experienced programmers The exercise on MSAs and ”the Python script for moving files” illustrates what I expect you to be able to do with Python Jon K. Lærdahl, Structural Bioinforma�cs More Unix stuff You must be able to move files between laptop and freebee.abel.uio.no! You must be able to uncompress and get all files out of a gzipped tarball, for example bigfile.tar.gz – And make this kind of archive file yourself Jon K. Lærdahl, Structural Bioinforma�cs More Unix stuff Three ways to run a Python script 1: In an interac�ve shell Jon K. Lærdahl, Structural Bioinforma�cs More Unix stuff Three ways to run a Python script 2: As a script sent as input to Python Jon K. Lærdahl, Structural Bioinforma�cs More Unix stuff Three ways to run a Python script 3: As a script with #! at first line Jon K.
    [Show full text]