DOCSLIB.ORG

Advanced Bash-Scripting Guide

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Advanced Bash-Scripting Guide Advanced Bash−Scripting Guide An in−depth exploration of the art of shell scripting Mendel Cooper <[email protected]> 5.0 24 June 2007 Revision History Revision 4.2 10 Dec 2006 Revised by: mc 'SPARKLEBERRY' release: Important Update. Revision 4.3 29 Apr 2007 Revised by: mc 'INKBERRY' release: Minor Update. Revision 5.0 24 Jun 2007 Revised by: mc 'SERVICEBERRY' release: Major Update. This tutorial assumes no previous knowledge of scripting or programming, but progresses rapidly toward an intermediate/advanced level of instruction . all the while sneaking in little snippets of UNIX® wisdom and lore. It serves as a textbook, a manual for self−study, and a reference and source of knowledge on shell scripting techniques. The exercises and heavily−commented examples invite active reader participation, under the premise that the only way to really learn scripting is to write scripts. This book is suitable for classroom use as a general introduction to programming concepts. The latest update of this document, as an archived, bzip2−ed "tarball" including both the SGML source and rendered HTML, may be downloaded from the author's home site. A pdf version is also available ( pdf mirror site). See the change log for a revision history. Dedication For Anita, the source of all the magic Advanced Bash−Scripting Guide Table of Contents Chapter 1. Why Shell Programming?...............................................................................................................1 Chapter 2. Starting Off With a Sha−Bang.......................................................................................................3 2.1. Invoking the script............................................................................................................................6 2.2. Preliminary Exercises.......................................................................................................................6 Part 2. Basics.......................................................................................................................................................7 Chapter 3. Special Characters...........................................................................................................................8 Chapter 4. Introduction to Variables and Parameters..................................................................................26 4.1. Variable Substitution......................................................................................................................26 4.2. Variable Assignment.......................................................................................................................29 4.3. Bash Variables Are Untyped..........................................................................................................30 4.4. Special Variable Types...................................................................................................................31 Chapter 5. Quoting...........................................................................................................................................36 5.1. Quoting Variables...........................................................................................................................36 5.2. Escaping..........................................................................................................................................38 Chapter 6. Exit and Exit Status.......................................................................................................................43 Chapter 7. Tests................................................................................................................................................45 7.1. Test Constructs...............................................................................................................................45 7.2. File test operators............................................................................................................................51 7.3. Other Comparison Operators..........................................................................................................54 7.4. Nested if/then Condition Tests.......................................................................................................59 7.5. Testing Your Knowledge of Tests..................................................................................................60 Chapter 8. Operations and Related Topics....................................................................................................61 8.1. Operators.........................................................................................................................................61 8.2. Numerical Constants.......................................................................................................................67 Part 3. Beyond the Basics.................................................................................................................................69 Chapter 9. Variables Revisited........................................................................................................................70 9.1. Internal Variables............................................................................................................................70 9.2. Manipulating Strings.......................................................................................................................87 9.2.1. Manipulating strings using awk............................................................................................93 9.2.2. Further Discussion.................................................................................................................94 9.3. Parameter Substitution....................................................................................................................94 9.4. Typing variables: declare or typeset.............................................................................................103 9.5. Indirect References.......................................................................................................................105 9.6. $RANDOM: generate random integer..........................................................................................108 9.7. The Double Parentheses Construct...............................................................................................117 i Advanced Bash−Scripting Guide Table of Contents Chapter 10. Loops and Branches..................................................................................................................120 10.1. Loops..........................................................................................................................................120 10.2. Nested Loops..............................................................................................................................132 10.3. Loop Control...............................................................................................................................133 10.4. Testing and Branching................................................................................................................136 Chapter 11. Command Substitution.............................................................................................................144 Chapter 12. Arithmetic Expansion................................................................................................................150 Chapter 13. Recess Time................................................................................................................................151 Part 4. Commands..........................................................................................................................................152 Chapter 14. Internal Commands and Builtins.............................................................................................160 14.1. Job Control Commands..............................................................................................................187 Chapter 15. External Filters, Programs and Commands...........................................................................191 15.1. Basic Commands........................................................................................................................191 15.2. Complex Commands...................................................................................................................196 15.3. Time / Date Commands..............................................................................................................206 15.4. Text Processing Commands........................................................................................................209 15.5. File and Archiving Commands...................................................................................................229 15.6. Communications Commands......................................................................................................246 15.7. Terminal Control Commands.....................................................................................................260 15.8. Math Commands.........................................................................................................................261 15.9. Miscellaneous Commands..........................................................................................................270 Chapter 16. System and Administrative Commands..................................................................................283
Load more

Recommended publications
  • Version 7.8-Systemd
    Linux From Scratch Version 7.8-systemd Created by Gerard Beekmans Edited by Douglas R. Reno Linux From Scratch: Version 7.8-systemd by Created by Gerard Beekmans and Edited by Douglas R. Reno Copyright © 1999-2015 Gerard Beekmans Copyright © 1999-2015, Gerard Beekmans All rights reserved. This book is licensed under a Creative Commons License. Computer instructions may be extracted from the book under the MIT License. Linux® is a registered trademark of Linus Torvalds. Linux From Scratch - Version 7.8-systemd Table of Contents Preface .......................................................................................................................................................................... vii i. Foreword ............................................................................................................................................................. vii ii. Audience ............................................................................................................................................................ vii iii. LFS Target Architectures ................................................................................................................................ viii iv. LFS and Standards ............................................................................................................................................ ix v. Rationale for Packages in the Book .................................................................................................................... x vi. Prerequisites
    [Show full text]
  • Powerview Command Reference
    PowerView Command Reference TRACE32 Online Help TRACE32 Directory TRACE32 Index TRACE32 Documents ...................................................................................................................... PowerView User Interface ............................................................................................................ PowerView Command Reference .............................................................................................1 History ...................................................................................................................................... 12 ABORT ...................................................................................................................................... 13 ABORT Abort driver program 13 AREA ........................................................................................................................................ 14 AREA Message windows 14 AREA.CLEAR Clear area 15 AREA.CLOSE Close output file 15 AREA.Create Create or modify message area 16 AREA.Delete Delete message area 17 AREA.List Display a detailed list off all message areas 18 AREA.OPEN Open output file 20 AREA.PIPE Redirect area to stdout 21 AREA.RESet Reset areas 21 AREA.SAVE Save AREA window contents to file 21 AREA.Select Select area 22 AREA.STDERR Redirect area to stderr 23 AREA.STDOUT Redirect area to stdout 23 AREA.view Display message area in AREA window 24 AutoSTOre ..............................................................................................................................
    [Show full text]
  • Rhd256.Bowez Notes.2006-07-20.Txt Thu Jul 20 15:31
    ../rhd256.bowez_notes.2006-07-20.txt Thu Jul 20 15:31:29 2006 1 ==> ./01_intro/notes.txt <== ---------------------------------------------------------------------- ---------------------------------------------------------------------- 2.3 Bash Special Characaters 1.3 Prerequisates ---------------------------------------------------------------------- ---------------------------------------------------------------------- ------------------------------------------------------------------ Familiarity: Redirection (Stream Manipulation) ------------------------------------------------------------------ the classroom setup (server1, station1, etc...) the "pub" directory every process has three default file descriptors (streams): Common Tools: 0: stdin 1: stdout terminal: 2: stderr changing fonts, etc.... - by default, all are connected to the terminal. editors: - In RHL, can be accessed by /dev/std{in,out,err} emacs vi, vim, gvim < redirect descriptor input pico -w, gedit > redirect descriptor output (create or clobber) (less pager) >> redirect descriptor output (create or append) finding information: 2>&1 bind one descriptor to another (i.e., combine stderr and stdout) --help, -h >& bash shortcut for the same man pages (chapters) info pages (info, pinfo, nautilus) examples: /usr/share/doc grep root /etc/* rpm (-qf filename, -ql, -qi) grep root /etc/* > /tmp/out grep root /etc/* 2> /tmp/err ---------------------------------------------------------------------- grep root /etc/* > /tmp/out2 2> /tmp/err2 grep root /etc/* >& /tmp/all 1.4 Procedures
    [Show full text]
  • Computer Security 03
    Computer Security 03. Program Hijacking & Code Injection Paul Krzyzanowski Rutgers University Spring 2019 September 25, 2019 CS 419 © 2019 Paul Krzyzanowski 1 Top vulnerability concerns for 2019 MITRE, a non-profit organization that manages federally-funded research & development centers, publishes a list of top security weaknesses Rank Name Score 1 Improper Restriction of Operations within the Bounds of a Memory Buffer 75.56 2 Cross-site Scripting 45.69 3 Improper Input Validation 43.61 4 Information Exposure 32.12 5 Out-of-bounds Read 26.53 6 SQL Injection 24.54 7 Use After Free 17.94 8 Integer Overflow or Wraparound 17.35 9 Cross-Site Request Forgery (CSRF) 15.54 10 14.10 Improper Limitation of a Pathname to a Restricted Directory ('Path Traversal') https://cwe.mitre.org/top25/archive/2019/2019_cwe_top25.html September 25, 2019 CS 419 © 2019 Paul Krzyzanowski 2 Hijacking Getting software to do something different from what the user or developer expected Examples: • Redirect web browser to a malicious site • Change DNS (IP address lookup) results • Change search engine • Change search paths to load different libraries or have different programs run • Intercept & alter messages Code injection Getting a program to process data in a way that it changes the execution of a program September 25, 2019 CS 419 © 2019 Paul Krzyzanowski 3 Bugs and mistakes • Most attacks are due to – Social engineering: getting a legitimate user to do something – Or bugs: using a program in a way it was not intended • Attacked system may be further weakened because
    [Show full text]
  • 11.6. Tempfile — Generate Temporary Files and Directories — Python V2
    Navigation • index • modules | • next | • previous | • Python v2.6.4 documentation » • The Python Standard Library » • 11. File and Directory Access » 11.6. tempfile — Generate temporary files and directories¶ This module generates temporary files and directories. It works on all supported platforms. In version 2.3 of Python, this module was overhauled for enhanced security. It now provides three new functions, NamedTemporaryFile(), mkstemp(), and mkdtemp(), which should eliminate all remaining need to use the insecure mktemp() function. Temporary file names created by this module no longer contain the process ID; instead a string of six random characters is used. Also, all the user-callable functions now take additional arguments which allow direct control over the location and name of temporary files. It is no longer necessary to use the global tempdir and template variables. To maintain backward compatibility, the argument order is somewhat odd; it is recommended to use keyword arguments for clarity. The module defines the following user-callable functions: tempfile.TemporaryFile([mode='w+b'[, bufsize=-1[, suffix=''[, prefix='tmp'[, dir=None]]]]])¶ Return a file-like object that can be used as a temporary storage area. The file is created using mkstemp(). It will be destroyed as soon as it is closed (including an implicit close when the object is garbage collected). Under Unix, the directory entry for the file is removed immediately after the file is created. Other platforms do not support this; your code should not rely on a temporary file created using this function having or not having a visible name in the file system. The mode parameter defaults to 'w+b' so that the file created can be read and written without being closed.
    [Show full text]
  • GNU Coreutils Cheat Sheet (V1.00) Created by Peteris Krumins ([email protected], -- Good Coders Code, Great Coders Reuse)
    GNU Coreutils Cheat Sheet (v1.00) Created by Peteris Krumins ([email protected], www.catonmat.net -- good coders code, great coders reuse) Utility Description Utility Description arch Print machine hardware name nproc Print the number of processors base64 Base64 encode/decode strings or files od Dump files in octal and other formats basename Strip directory and suffix from file names paste Merge lines of files cat Concatenate files and print on the standard output pathchk Check whether file names are valid or portable chcon Change SELinux context of file pinky Lightweight finger chgrp Change group ownership of files pr Convert text files for printing chmod Change permission modes of files printenv Print all or part of environment chown Change user and group ownership of files printf Format and print data chroot Run command or shell with special root directory ptx Permuted index for GNU, with keywords in their context cksum Print CRC checksum and byte counts pwd Print current directory comm Compare two sorted files line by line readlink Display value of a symbolic link cp Copy files realpath Print the resolved file name csplit Split a file into context-determined pieces rm Delete files cut Remove parts of lines of files rmdir Remove directories date Print or set the system date and time runcon Run command with specified security context dd Convert a file while copying it seq Print sequence of numbers to standard output df Summarize free disk space setuidgid Run a command with the UID and GID of a specified user dir Briefly list directory
    [Show full text]
  • Constraints in Dynamic Symbolic Execution: Bitvectors Or Integers?
    Constraints in Dynamic Symbolic Execution: Bitvectors or Integers? Timotej Kapus, Martin Nowack, and Cristian Cadar Imperial College London, UK ft.kapus,m.nowack,[email protected] Abstract. Dynamic symbolic execution is a technique that analyses programs by gathering mathematical constraints along execution paths. To achieve bit-level precision, one must use the theory of bitvectors. However, other theories might achieve higher performance, justifying in some cases the possible loss of precision. In this paper, we explore the impact of using the theory of integers on the precision and performance of dynamic symbolic execution of C programs. In particular, we compare an implementation of the symbolic executor KLEE using a partial solver based on the theory of integers, with a standard implementation of KLEE using a solver based on the theory of bitvectors, both employing the popular SMT solver Z3. To our surprise, our evaluation on a synthetic sort benchmark, the ECA set of Test-Comp 2019 benchmarks, and GNU Coreutils revealed that for most applications the integer solver did not lead to any loss of precision, but the overall performance difference was rarely significant. 1 Introduction Dynamic symbolic execution is a popular program analysis technique that aims to systematically explore all the paths in a program. It has been very successful in bug finding and test case generation [3, 4]. The research community and industry have produced many tools performing symbolic execution, such as CREST [5], FuzzBALL [9], KLEE [2], PEX [14], and SAGE [6], among others. To illustrate how dynamic symbolic execution works, consider the program shown in Figure 1a.
    [Show full text]
  • NSWI 0138: Advanced Unix Programming
    NSWI 0138: Advanced Unix programming (c) 2011-2016 Vladim´ırKotal (c) 2009-2010 Jan Pechanec, Vladim´ırKotal SISAL MFF UK, Malostransk´en´am.25, 118 00 Praha 1 Charles University Czech Republic Vladim´ırKotal [email protected] March 10, 2016 1 Vladim´ırKotal NSWI 0138 (Advanced Unix programming) Contents 1 Overview 5 1.1 What is this lecture about? . .5 1.2 The lecture will cover... .5 1.3 A few notes on source code files . .6 2 Testing 6 2.1 Why?...........................................6 2.2 When ? . .6 2.3 Types of testing . .7 3 Debugging 8 3.1 Debuging in general . .8 3.2 Observing . .9 3.3 Helper tools . .9 3.3.1 ctags . .9 3.3.2 cscope . 10 3.3.3 OpenGrok . 11 3.3.4 Other tools . 11 3.4 Debugging data . 11 3.4.1 stabs . 11 3.4.2 DWARF . 12 3.4.3 CTF (Compact C Type Format) . 12 3.5 Resource leaks . 13 3.6 libumem . 13 3.6.1 How does libumem work . 14 3.6.2 Using libumem+mdb to find memory leaks . 14 3.6.3 How does ::findleaks work . 16 3.7 watchmalloc . 17 3.8 Call tracing . 18 3.9 Using /proc . 19 3.10 Debugging dynamic libraries . 20 3.11 Debuggers . 20 3.12 Symbol search and interposition . 20 3.13 dtrace . 21 4 Terminals 21 4.1 Terminal I/O Overview . 21 4.2 Terminal I/O Overview (cont.) . 22 4.3 Physical (Hardware) Terminal . 24 4.4 stty(1) command . 24 4.5 TTY Driver Connected To a Phy Terminal .
    [Show full text]
  • GPL-3-Free Replacements of Coreutils 1 Contents
    GPL-3-free replacements of coreutils 1 Contents 2 Coreutils GPLv2 2 3 Alternatives 3 4 uutils-coreutils ............................... 3 5 BSDutils ................................... 4 6 Busybox ................................... 5 7 Nbase .................................... 5 8 FreeBSD ................................... 6 9 Sbase and Ubase .............................. 6 10 Heirloom .................................. 7 11 Replacement: uutils-coreutils 7 12 Testing 9 13 Initial test and results 9 14 Migration 10 15 Due to the nature of Apertis and its target markets there are licensing terms that 1 16 are problematic and that forces the project to look for alternatives packages. 17 The coreutils package is good example of this situation as its license changed 18 to GPLv3 and as result Apertis cannot provide it in the target repositories and 19 images. The current solution of shipping an old version which precedes the 20 license change is not tenable in the long term, as there are no upgrades with 21 bugfixes or new features for such important package. 22 This situation leads to the search for a drop-in replacement of coreutils, which 23 need to provide compatibility with the standard GNU coreutils packages. The 24 reason behind is that many other packages rely on the tools it provides, and 25 failing to do that would lead to hard to debug failures and many custom patches 26 spread all over the archive. In this regard the strict requirement is to support 27 the features needed to boot a target image with ideally no changes in other 28 components. The features currently available in our coreutils-gplv2 fork are a 29 good approximation. 30 Besides these specific requirements, the are general ones common to any Open 31 Source Project, such as maturity and reliability.
    [Show full text]
  • Learning Embedded Linux Through the File System: a Top-Down Approach
    Learning embedded Linux through the file system: A top-down approach Jason Kridner, [email protected] June 8, 2011, 10:10AM ESC Chicago room 34 http://beagleboard.org/linux_education Abstract Typical introductions to embedded Linux focus on complex tasks, like building device drivers, that are highly desired by at least some people working on your Linux product team. What about the rest of us that simply want to build a quick understanding of how to do some of the simple tasks that are trivial in microcontrollers without using operating systems, like accessing GPIO pins and performing I2C or SPI serial communications? This class takes a quick tops-down overview from the perspective of an electrical engineer looking to take advantage of the latest in GUI building technologies, like Android, Qt, and HTML5, without completely losing access to the underlying hardware for simple system integration tasks. Getting started Much to learn ◦ I‟m used to microcontrollers: just give me the datasheet with register definitions and set me free! Training on boot & device drivers useful ◦ Often geared more at system bring-up ◦ What about the everyday user? ◦ Where is that abstraction benefit? Let‟s just walk a working system Kernel.org documentation http://www.kernel.org/doc/ Documentation extracted from the Linux kernel and mirrored on the web where Google can find it: ◦ Documentation - Text files in the kernel source tarball's Documentation subdirectory. ◦ htmldocs - Kernel Documentation maintained in docbook format (output of "make htmldocs"). ◦ Menuconfig - help text for each kernel configuration option (from kconfig source). ◦ README various README files scattered around Linux kernel source ◦ RFC - List of IETF RFCs referred to by kernel source files.
    [Show full text]
  • GNU Coreutils Core GNU Utilities for Version 9.0, 20 September 2021
    GNU Coreutils Core GNU utilities for version 9.0, 20 September 2021 David MacKenzie et al. This manual documents version 9.0 of the GNU core utilities, including the standard pro- grams for text and file manipulation. Copyright c 1994{2021 Free Software Foundation, Inc. Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.3 or any later version published by the Free Software Foundation; with no Invariant Sections, with no Front-Cover Texts, and with no Back-Cover Texts. A copy of the license is included in the section entitled \GNU Free Documentation License". i Short Contents 1 Introduction :::::::::::::::::::::::::::::::::::::::::: 1 2 Common options :::::::::::::::::::::::::::::::::::::: 2 3 Output of entire files :::::::::::::::::::::::::::::::::: 12 4 Formatting file contents ::::::::::::::::::::::::::::::: 22 5 Output of parts of files :::::::::::::::::::::::::::::::: 29 6 Summarizing files :::::::::::::::::::::::::::::::::::: 41 7 Operating on sorted files ::::::::::::::::::::::::::::::: 47 8 Operating on fields ::::::::::::::::::::::::::::::::::: 70 9 Operating on characters ::::::::::::::::::::::::::::::: 80 10 Directory listing:::::::::::::::::::::::::::::::::::::: 87 11 Basic operations::::::::::::::::::::::::::::::::::::: 102 12 Special file types :::::::::::::::::::::::::::::::::::: 125 13 Changing file attributes::::::::::::::::::::::::::::::: 135 14 File space usage ::::::::::::::::::::::::::::::::::::: 143 15 Printing text :::::::::::::::::::::::::::::::::::::::
    [Show full text]
  • Instructions to BASH Shell Scripts
    Instructions to BASH Shell Scripts Jingjing Yang Department of Human Genetics Tips • SSH Login without Password • Local computer: Generate a pair of authentication keys. Do not enter a passphrase. • ssh-keygen -t rsa • Login to cluster: Create a directory ~/.ssh • mkdir -p .ssh • Local computer: Append your local public key to ~/.ssh/authorized_keys • cat .ssh/id_rsa.pub | ssh user@ hgcc.genetics.emory.edu 'cat >> .ssh/authorized_keys’ • More information: http://www.linuxproblem.org/art_9.html • Personalize Your Shell Command • MAC Users: Add your commands to the ~/.bash_profile under your home directory • PC Users: Add your commands to ~/.bashrc or ~/.profile • HGCC: Add your commands to ~/.bashrc.user • Add shortcut for your commands by creating environment variables (e.g., set ll as short cut for ls -l -t -G): export ll=‘ls -l -t -G’ • Or use alias: • alias hgcc='ssh [email protected]' • alias c='clear' • alias e='exit' • One difference between the export and alias is that alias is only a shell feature. Environment variables are inherited by all subprocesses (unless deliberately cleared). Mont Cluster Directory on MAC • Step1: Install SSHFS (https://github.com/osxfuse/sshfs/releases), latest version SSHFS 2.5.0 • Step2: Install FUSE (https://osxfuse.github.io/), Latest version FUSE for macOS 3.7.1 • Step3: After the installation user has to create a folder, mount point, on user’s host machines. Then in terminal execute the command: • sshfs username@server:/path on server/ ~/path to mount point • My example command: sshfs
    [Show full text]