DOCSLIB.ORG

Advanced Bash-Scripting Guide

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Advanced Bash-Scripting Guide Advanced Bash-Scripting Guide Advanced Bash-Scripting Guide Table of Contents Advanced Bash-Scripting Guide.......................................................................................................................1 An in-depth exploration of the art of shell scripting................................................................................1 Mendel Cooper..................................................................................................................................1 Dedication............................................................................................................................................................3 Part 1. Introduction..........................................................................................................................................15 Chapter 1. Shell Programming!.......................................................................................................................17 Notes......................................................................................................................................................18 Chapter 2. Starting Off With a Sha-Bang......................................................................................................21 2.1. Invoking the script.....................................................................................................................................25 Notes......................................................................................................................................................25 2.2. Preliminary Exercises................................................................................................................................27 Part 2. Basics.....................................................................................................................................................29 Chapter 3. Special Characters.........................................................................................................................31 Notes......................................................................................................................................................50 Chapter 4. Introduction to Variables and Parameters..................................................................................51 4.1. Variable Substitution.................................................................................................................................53 Notes......................................................................................................................................................55 4.2. Variable Assignment..................................................................................................................................57 4.3. Bash Variables Are Untyped.....................................................................................................................59 4.4. Special Variable Types..............................................................................................................................61 Notes......................................................................................................................................................65 Chapter 5. Quoting...........................................................................................................................................67 5.1. Quoting Variables......................................................................................................................................69 Notes......................................................................................................................................................70 5.2. Escaping......................................................................................................................................................73 Chapter 6. Exit and Exit Status.......................................................................................................................79 Notes......................................................................................................................................................81 Chapter 7. Tests................................................................................................................................................83 i Advanced Bash-Scripting Guide Table of Contents 7.1. Test Constructs...........................................................................................................................................85 Notes......................................................................................................................................................92 7.2. File test operators.......................................................................................................................................93 Notes......................................................................................................................................................95 7.3. Other Comparison Operators...................................................................................................................97 Notes....................................................................................................................................................102 7.4. Nested if/then Condition Tests................................................................................................................103 7.5. Testing Your Knowledge of Tests...........................................................................................................105 Chapter 8. Operations and Related Topics..................................................................................................107 8.1. Operators..................................................................................................................................................109 Notes....................................................................................................................................................115 8.2. Numerical Constants...............................................................................................................................117 8.3. The Double-Parentheses Construct........................................................................................................119 8.4. Operator Precedence...............................................................................................................................121 Notes....................................................................................................................................................123 Part 3. Beyond the Basics...............................................................................................................................125 Chapter 9. Another Look at Variables.........................................................................................................127 9.1. Internal Variables....................................................................................................................................129 Notes....................................................................................................................................................147 9.2. Typing variables: declare or typeset......................................................................................................149 9.2.1. Another use for declare..............................................................................................................151 Notes..............................................................................................................................................151 9.3. $RANDOM: generate random integer...................................................................................................153 Notes....................................................................................................................................................163 Chapter 10. Manipulating Variables.............................................................................................................165 10.1. Manipulating Strings.............................................................................................................................167 10.1.1. Manipulating strings using awk...............................................................................................174 10.1.2. Further Reference.....................................................................................................................175 Notes..............................................................................................................................................175 ii Advanced Bash-Scripting Guide Table of Contents 10.2. Parameter Substitution.........................................................................................................................177 Notes....................................................................................................................................................186 Chapter 11. Loops and Branches..................................................................................................................187 11.1. Loops.......................................................................................................................................................189 Notes....................................................................................................................................................202
Load more

Recommended publications
  • Linux Tutorial Last Updated: September 29 2021 for Windows Users
    VLAAMS SUPERCOMPUTER Innovative Computing CENTRUM for A Smarter Flanders Linux Tutorial Last updated: September 29 2021 For Windows Users Authors: Jasper Devreker (UGent), Ewan Higgs (UGent), Kenneth Hoste (UGent) Acknowledgement: VSCentrum.be Audience: This document is a hands-on guide for using the Linux command line in the context of the UGent HPC infrastructure. The command line (sometimes called ’shell’) can seems daunting at first, but with a little understanding can be very easy to use. Everything you do startsatthe prompt. Here you have the liberty to type in any commands you want. Soon, you will be able to move past the limited point and click interface and express interesting ideas to the computer using the shell. Gaining an understanding of the fundamentals of Linux will help accelerate your research using the HPC infrastructure. You will learn about commands, managing files, and some scripting basics. Notification: In$ commands this tutorial specific commands are separated from the accompanying text: These should be entered by the reader at a command line in a terminal on the UGent-HPC. They appear in all exercises preceded by a $ and printed in bold. You’ll find those actions ina grey frame. Button are menus, buttons or drop down boxes to be pressed or selected. “Directory” is the notation for directories (called “folders” in Windows terminology) or specific files. (e.g., “/user/home/gent/vsc400/vsc40000”) “Text” Is the notation for text to be entered. Tip: A “Tip” paragraph is used for remarks or tips. They can also be downloaded from the VSC website at https://www.vscentrum.be.
    [Show full text]
  • Linux Fundamentals (GL120) U8583S This Is a Challenging Course That Focuses on the Fundamental Tools and Concepts of Linux and Unix
    Course data sheet Linux Fundamentals (GL120) U8583S This is a challenging course that focuses on the fundamental tools and concepts of Linux and Unix. Students gain HPE course number U8583S proficiency using the command line. Beginners develop a Course length 5 days solid foundation in Unix, while advanced users discover Delivery mode ILT, vILT patterns and fill in gaps in their knowledge. The course View schedule, local material is designed to provide extensive hands-on View now pricing, and register experience. Topics include basic file manipulation; basic and View related courses View now advanced filesystem features; I/O redirection and pipes; text manipulation and regular expressions; managing jobs and processes; vi, the standard Unix editor; automating tasks with Why HPE Education Services? shell scripts; managing software; secure remote • IDC MarketScape leader 5 years running for IT education and training* administration; and more. • Recognized by IDC for leading with global coverage, unmatched technical Prerequisites Supported distributions expertise, and targeted education consulting services* Students should be comfortable with • Red Hat Enterprise Linux 7 • Key partnerships with industry leaders computers. No familiarity with Linux or other OpenStack®, VMware®, Linux®, Microsoft®, • SUSE Linux Enterprise 12 ITIL, PMI, CSA, and SUSE Unix operating systems is required. • Complete continuum of training delivery • Ubuntu 16.04 LTS options—self-paced eLearning, custom education consulting, traditional classroom, video on-demand
    [Show full text]
  • Checkpointing Under Linux with Berkeley Lab Checkpoint/Restart
    N1GE6 Checkpointing and Berkeley Lab Checkpoint/Restart Liang PENG Lip Kian NG N1GE6 Checkpointing and Berkeley Lab Checkpoint/Restart Liang PENG Lip Kian NG APSTC-TB-2004-005 Abstract: N1GE6, formerly known as Sun Grid Engine, is widely used in HPTC environment for efficient utilization of compute resources. As applications in such environment are generally compute intensive, fault tolerance is required to minimize the impact of hardware failure. N1GE6 has several fault tolerance features and in this report, the focus will be on the checkpointing support and the integration of Berkeley Lab Checkpoint/Restart will be used as an example. Keywords: checkpoint, Grid Engine, blcr Email Address: [email protected] [email protected] Revision History Version Date Comments 1.1 Jul 14, 2004 1.2 Dec 28, 2004 Feedback from Reuti (reuti__at__staff.uni-marburg.de) • Transparent interface is user-level (Table 1). • Update to state diagram (Illustration 2). N1GE6 Checkpointing and Berkeley Lab Checkpoint/Restart Liang PENG Lip Kian NG Asia Pacific Science and Technology Center Sun Microsystems Pte Ltd, Singapore Introduction Checkpointing is the process of writing out the state information of a running application to physical storage periodically. With this feature, an application will be able to restart from the last checkpointed state instead of from the beginning which would have been computationally expensive in HPTC environment. In general, checkpointing tools can be classified into 2 different classes: • Kernel-level – Such tools are built into the kernel of the operating system. During a checkpoint, the entire process space (which tends to be huge) is written to physical storage.
    [Show full text]
  • Bash Scripts for Avpres Verify Manifest(1)
    verify_manifest(1) Bash Scripts for AVpres verify_manifest(1) NAME verify_manifest - Verify a checksum manifest of a folder or file SYNOPSIS verify_manifest -i input_path [-m manifest_file] verify_manifest -h | -x DESCRIPTION Bash AVpres is a collection of Bash scripts for audio-visual preservation. One of these small programs is verify_manifest.Itcreates a checksum manifest of a folder or file. Bash version 3.2 is required, but we strongly advise to use the current version 5.1. OPTIONS BASIC OPTIONS -i input_path,--input=input_path path to an input folder or file -m manifest_file,--manifest=manifest_file path to the manifest file If this parameter is not passed, then the script uses for a folder: <input_path>_<algorithm>.txt and for a file: <input_path>_<extension>_<algorithm>.txt ADVA NCED OPTIONS The arguments of the advanced options can be overwritten by the user.Please remember that anystring containing spaces must be quoted, or its spaces must be escaped. --algorithm=(xxh32|xxh64|xxh128|md5|sha1|sha256|sha512|crc32) We advise to use a faster non-cryptographic hash functions, because we consider that, for archival purposes, there is no necessity to apply a more complexunkeyed cryptographic hash function. The algorithm name can be passed in upper or lower case letters. The default algorithm is xxHash 128: --algorithm=xxh128 Note that until end of 2020 the default algorithm was MD5, which has the same checksum size than the xxHash 128 algorithm. Therefore, if you verity files with an MD5 checksum, then you may pass the option --algorithm=md5 in order to speed-up the verification. Also xxHash 32 and CRC-32 have the same checksum size.
    [Show full text]
  • Research Computing and Cyberinfrastructure Team
    Research Computing and CyberInfrastructure team ! Working with the Linux Shell on the CWRU HPC 31 January 2019 7 February 2019 ! Presenter Emily Dragowsky KSL Data Center RCCI Team: Roger Bielefeld, Mike Warfe, Hadrian Djohari! Brian Christian, Emily Dragowsky, Jeremy Fondran, Cal Frye,! Sanjaya Gajurel, Matt Garvey, Theresa Griegger, Cindy Martin, ! Sean Maxwell, Jeno Mozes, Nasir Yilmaz, Lee Zickel Preface: Prepare your environment • User account ! # .bashrc ## cli essentials ! if [ -t 1 ] then bind '"\e[A": history-search-backward' bind '"\e[B": history-search-forward' bind '"\eOA": history-search-backward' bind '"\eOB": history-search-forward' fi ! This is too useful to pass up! Working with Linux • Preamble • Intro Session Linux Review: Finding our way • Files & Directories: Sample work flow • Shell Commands • Pipes & Redirection • Scripting Foundations • Shell & environment variables • Control Structures • Regular expressions & text manipulation • Recap & Look Ahead Rider Cluster Components ! rider.case.edu ondemand.case.edu University Firewall ! Admin Head Nodes SLURM Science Nodes Master DMZ Resource ! Data Manager Transfer Disk Storage Nodes Batch nodes GPU nodes SMP nodes Running a job: where is it? slide from Hadrian Djohari Storing Data on the HPC table from Nasir Yilmaz How data moves across campus • Buildings on campus are each assigned to a zone. Data connections go from every building to the distribution switch at the center of the zone and from there to the data centers at Kelvin Smith Library and Crawford Hall. slide
    [Show full text]
  • Bash Guide for Beginners
    Bash Guide for Beginners Machtelt Garrels Garrels BVBA <tille wants no spam _at_ garrels dot be> Version 1.11 Last updated 20081227 Edition Bash Guide for Beginners Table of Contents Introduction.........................................................................................................................................................1 1. Why this guide?...................................................................................................................................1 2. Who should read this book?.................................................................................................................1 3. New versions, translations and availability.........................................................................................2 4. Revision History..................................................................................................................................2 5. Contributions.......................................................................................................................................3 6. Feedback..............................................................................................................................................3 7. Copyright information.........................................................................................................................3 8. What do you need?...............................................................................................................................4 9. Conventions used in this
    [Show full text]
  • Linux Hardening Techniques Vasudev Baldwa Ubnetdef, Spring 2021 Agenda
    Linux Hardening Techniques Vasudev Baldwa UBNetDef, Spring 2021 Agenda 1. What is Systems Hardening? 2. Basic Principles 3. Updates & Encryption 4. Monitoring 5. Services 6. Firewalls 7. Logging What is System Hardening? ⬡ A collection of tools, techniques, and best practices to reduce vulnerability in technology applications, systems, infrastructure, firmware, and other areas ⬡ 3 major areas: OS vs Software vs Network ⬠ When have we done hardening in this class before? ⬠ This lecture is focusing mostly on OS and software level Why Harden? ⬡ Firewalls can only get us so far, what happens when at attack is inside the network? ⬠ If you have nothing protecting your systems you are in trouble ⬡ We want some kind of secondary protection A Few Cybersecurity Principles ⬡ Zero Trust Security ⬠ Instead of assuming everything behind the firewall is safe, Zero Trust verifies each request as though it originates from an unsecure network ⬡ Principle of Least Privilege ⬠ Only privileges needed to complete a task should be allowed ⬠ Users should not have domain administrator/root privileges ⬡ Principle of Least Common Mechanism ⬠ Mechanisms used to access resources should not be shared in order to avoid the transmission of data. ⬠ Shared resources should not be used to access resources The Threat Model ⬡ A process by which potential threats can be identified and prioritized. ⬠ If you have a web server that feeds input to a mysql database, then protecting against mysql injections would be prioritized in your model. 2 considerations ⬡ *nix like is a very
    [Show full text]
  • Download Instructions—Portal
    Download instructions These instructions are recommended to download big files. How to download and verify files from downloads.gvsig.org • H ow to download files • G NU/Linux Systems • MacO S X Systems • Windows Systems • H ow to validate the downloaded files How to download files The files distributed on this site can be downloaded using different access protocols, the ones currently available are FTP, HTTP and RSYNC. The base URL of the site for the different protocols is: • ftp://gvsig.org/ • http://downloads.gvsig.org/ • r sync://gvsig.org/downloads/ To download files using the first two protocols is recommended to use client programs able to resume partial downloads, as it is usual to have transfer interruptions when downloading big files like DVD images. There are multiple free (and multi platform) programs to download files using different protocols (in our case we are interested in FTP and HTTP), from them we can highlight curl (http://curl.haxx.se/) and wget (http://www.gnu.org/software/wget/) from the command line ones and Free Download Manager from the GUI ones (this one is only for Windows systems). The curl program is included in MacOS X and is available for almost all GNU/Linux distributions. It can be downloaded in source code or in binary form for different operating systems from the project web site. The wget program is also included in almost all GNU/Linux distributions and its source code or binaries of the program for different systems can be downloaded from this page. Next we will explain how to download files from the most usual operating systems using the programs referenced earlier: • G NU/Linux Systems • MacO S X Systems • Windows Systems The use of rsync (available from the URL http://samba.org/rsync/) it is left as an exercise for the reader, we will only said that it is advised to use the --partial option to avoid problems when there transfers are interrupted.
    [Show full text]
  • Linux Cheat Sheet
    1 of 4 ########################################### # 1.1. File Commands. # Name: Bash CheatSheet # # # # A little overlook of the Bash basics # ls # lists your files # # ls -l # lists your files in 'long format' # Usage: A Helpful Guide # ls -a # lists all files, including hidden files # # ln -s <filename> <link> # creates symbolic link to file # Author: J. Le Coupanec # touch <filename> # creates or updates your file # Date: 2014/11/04 # cat > <filename> # places standard input into file # Edited: 2015/8/18 – Michael Stobb # more <filename> # shows the first part of a file (q to quit) ########################################### head <filename> # outputs the first 10 lines of file tail <filename> # outputs the last 10 lines of file (-f too) # 0. Shortcuts. emacs <filename> # lets you create and edit a file mv <filename1> <filename2> # moves a file cp <filename1> <filename2> # copies a file CTRL+A # move to beginning of line rm <filename> # removes a file CTRL+B # moves backward one character diff <filename1> <filename2> # compares files, and shows where differ CTRL+C # halts the current command wc <filename> # tells you how many lines, words there are CTRL+D # deletes one character backward or logs out of current session chmod -options <filename> # lets you change the permissions on files CTRL+E # moves to end of line gzip <filename> # compresses files CTRL+F # moves forward one character gunzip <filename> # uncompresses files compressed by gzip CTRL+G # aborts the current editing command and ring the terminal bell gzcat <filename> #
    [Show full text]
  • SUSE Linux Enterprise Server 11 SP4 System Analysis and Tuning Guide System Analysis and Tuning Guide SUSE Linux Enterprise Server 11 SP4
    SUSE Linux Enterprise Server 11 SP4 System Analysis and Tuning Guide System Analysis and Tuning Guide SUSE Linux Enterprise Server 11 SP4 Publication Date: September 24, 2021 SUSE LLC 1800 South Novell Place Provo, UT 84606 USA https://documentation.suse.com Copyright © 2006– 2021 SUSE LLC and contributors. All rights reserved. Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or (at your option) version 1.3; with the Invariant Section being this copyright notice and license. A copy of the license version 1.2 is included in the section entitled “GNU Free Documentation License”. For SUSE trademarks, see http://www.suse.com/company/legal/ . All other third party trademarks are the property of their respective owners. A trademark symbol (®, ™ etc.) denotes a SUSE or Novell trademark; an asterisk (*) denotes a third party trademark. All information found in this book has been compiled with utmost attention to detail. However, this does not guarantee complete accuracy. Neither SUSE LLC, its aliates, the authors nor the translators shall be held liable for possible errors or the consequences thereof. Contents About This Guide xi 1 Available Documentation xii 2 Feedback xiv 3 Documentation Conventions xv I BASICS 1 1 General Notes on System Tuning 2 1.1 Be Sure What Problem to Solve 2 1.2 Rule Out Common Problems 3 1.3 Finding the Bottleneck 3 1.4 Step-by-step Tuning 4 II SYSTEM MONITORING 5 2 System Monitoring Utilities 6 2.1 Multi-Purpose Tools 6 vmstat 7
    [Show full text]
  • GNU Coreutils Cheat Sheet (V1.00) Created by Peteris Krumins ([email protected], -- Good Coders Code, Great Coders Reuse)
    GNU Coreutils Cheat Sheet (v1.00) Created by Peteris Krumins ([email protected], www.catonmat.net -- good coders code, great coders reuse) Utility Description Utility Description arch Print machine hardware name nproc Print the number of processors base64 Base64 encode/decode strings or files od Dump files in octal and other formats basename Strip directory and suffix from file names paste Merge lines of files cat Concatenate files and print on the standard output pathchk Check whether file names are valid or portable chcon Change SELinux context of file pinky Lightweight finger chgrp Change group ownership of files pr Convert text files for printing chmod Change permission modes of files printenv Print all or part of environment chown Change user and group ownership of files printf Format and print data chroot Run command or shell with special root directory ptx Permuted index for GNU, with keywords in their context cksum Print CRC checksum and byte counts pwd Print current directory comm Compare two sorted files line by line readlink Display value of a symbolic link cp Copy files realpath Print the resolved file name csplit Split a file into context-determined pieces rm Delete files cut Remove parts of lines of files rmdir Remove directories date Print or set the system date and time runcon Run command with specified security context dd Convert a file while copying it seq Print sequence of numbers to standard output df Summarize free disk space setuidgid Run a command with the UID and GID of a specified user dir Briefly list directory
    [Show full text]
  • Cooper (2007) Advanced Bash Scripting Guide
    Advanced Bash-Scripting Guide An in-depth exploration of the art of shell scripting Mendel Cooper <[email protected]> 5.1 10 November 2007 Revision History Revision 4.3 29 Apr 2007 Revised by: mc 'INKBERRY' release: Minor Update. Revision 5.0 24 Jun 2007 Revised by: mc 'SERVICEBERRY' release: Major Update. Revision 5.1 10 Nov 2007 Revised by: mc 'LINGONBERRY' release: Minor Update. This tutorial assumes no previous knowledge of scripting or programming, but progresses rapidly toward an intermediate/advanced level of instruction . all the while sneaking in little snippets of UNIX® wisdom and lore. It serves as a textbook, a manual for self-study, and a reference and source of knowledge on shell scripting techniques. The exercises and heavily-commented examples invite active reader participation, under the premise that the only way to really learn scripting is to write scripts. This book is suitable for classroom use as a general introduction to programming concepts. The latest update of this document, as an archived, bzip2-ed "tarball" including both the SGML source and rendered HTML, may be downloaded from the author's home site. A pdf version is also available ( pdf mirror site). See the change log for a revision history. Dedication For Anita, the source of all the magic Advanced Bash-Scripting Guide Table of Contents Chapter 1. Why Shell Programming?...............................................................................................................1 Chapter 2. Starting Off With a Sha-Bang........................................................................................................3
    [Show full text]