Chapter 10 SHELL Substitution and I/O Operations
Total Page:16
File Type:pdf, Size:1020Kb
Load more
Recommended publications
-
Shell Scripting with Bash
Introduction to Shell Scripting with Bash Charles Jahnke Research Computing Services Information Services & Technology Topics for Today ● Introductions ● Basic Terminology ● How to get help ● Command-line vs. Scripting ● Variables ● Handling Arguments ● Standard I/O, Pipes, and Redirection ● Control Structures (loops and If statements) ● SCC Job Submission Example Research Computing Services Research Computing Services (RCS) A group within Information Services & Technology at Boston University provides computing, storage, and visualization resources and services to support research that has specialized or highly intensive computation, storage, bandwidth, or graphics requirements. Three Primary Services: ● Research Computation ● Research Visualization ● Research Consulting and Training Breadth of Research on the Shared Computing Cluster (SCC) Me ● Research Facilitator and Administrator ● Background in biomedical engineering, bioinformatics, and IT systems ● Offices on both CRC and BUMC ○ Most of our staff on the Charles River Campus, some dedicated to BUMC ● Contact: [email protected] You ● Who has experience programming? ● Using Linux? ● Using the Shared Computing Cluster (SCC)? Basic Terminology The Command-line The line on which commands are typed and passed to the shell. Username Hostname Current Directory [username@scc1 ~]$ Prompt Command Line (input) The Shell ● The interface between the user and the operating system ● Program that interprets and executes input ● Provides: ○ Built-in commands ○ Programming control structures ○ Environment -
Windows Command Prompt Cheatsheet
Windows Command Prompt Cheatsheet - Command line interface (as opposed to a GUI - graphical user interface) - Used to execute programs - Commands are small programs that do something useful - There are many commands already included with Windows, but we will use a few. - A filepath is where you are in the filesystem • C: is the C drive • C:\user\Documents is the Documents folder • C:\user\Documents\hello.c is a file in the Documents folder Command What it Does Usage dir Displays a list of a folder’s files dir (shows current folder) and subfolders dir myfolder cd Displays the name of the current cd filepath chdir directory or changes the current chdir filepath folder. cd .. (goes one directory up) md Creates a folder (directory) md folder-name mkdir mkdir folder-name rm Deletes a folder (directory) rm folder-name rmdir rmdir folder-name rm /s folder-name rmdir /s folder-name Note: if the folder isn’t empty, you must add the /s. copy Copies a file from one location to copy filepath-from filepath-to another move Moves file from one folder to move folder1\file.txt folder2\ another ren Changes the name of a file ren file1 file2 rename del Deletes one or more files del filename exit Exits batch script or current exit command control echo Used to display a message or to echo message turn off/on messages in batch scripts type Displays contents of a text file type myfile.txt fc Compares two files and displays fc file1 file2 the difference between them cls Clears the screen cls help Provides more details about help (lists all commands) DOS/Command Prompt help command commands Source: https://technet.microsoft.com/en-us/library/cc754340.aspx. -
Fortianalyzer
Network diagnose fortilogd Device message rate msgrate-device Network Troubleshooting diagnose fortilogd msgrate-type Message rate for each log type execute ping [host] Ping utility execute traceroute [host] Traceroute utility diag sniffer packet <interface> Packet sniffer Disk <filter> <level> <timestamp> Disk / RAID / Virtual Disk config system fortiview settings Resolve IP address to hostname set resolve-ip enable config system locallog disk setting What happens with oldest logs set diskfull nolog / overwrite Logging diagnose system raid [option] RAID information status, hwinfo, alarms Log Forwarding diagnose system disk [option] Disk information info, health, errors, attributes CHEATSHEET config system log-forward edit log-aggregation <id> For virtual machines: provides a list of Forwarding logs to FortiAnalyzer / execute lvm info available disks aggregation-client set mode Syslog / CEF <realtime, execute lvm extend <disk nr.> For virtual machines: Add disk FORTIANALYZER FOR 6.0 aggregation, disable> config system Configure the FortiAnalyzer that receives © BOLL Engineering AG, FortiAnalyzer Cheat Sheet Version 1.1 / 08.02.2019 log-forward-service logs ADOM set accept-aggregation enable ADOM operation Log Backup config system global ADOM settings set adom-status [en/dis] Enable or disable ADOM mode General execute backup logs config system global Set ADOM mode to normal or advanced / Default device information <device name | all> for VDOMs) <ftp | sftp | scp> <server ip> Backup logs to external storage set adom-mode [normal/advanced] -
Introduction to UNIX at MSI June 23, 2015 Presented by Nancy Rowe
Introduction to UNIX at MSI June 23, 2015 Presented by Nancy Rowe The Minnesota Supercomputing Institute for Advanced Computational Research www.msi.umn.edu/tutorial/ © 2015 Regents of the University of Minnesota. All rights reserved. Supercomputing Institute for Advanced Computational Research Overview • UNIX Overview • Logging into MSI © 2015 Regents of the University of Minnesota. All rights reserved. Supercomputing Institute for Advanced Computational Research Frequently Asked Questions msi.umn.edu > Resources> FAQ Website will be updated soon © 2015 Regents of the University of Minnesota. All rights reserved. Supercomputing Institute for Advanced Computational Research What’s the difference between Linux and UNIX? The terms can be used interchangeably © 2015 Regents of the University of Minnesota. All rights reserved. Supercomputing Institute for Advanced Computational Research UNIX • UNIX is the operating system of choice for engineering and scientific workstations • Originally developed in the late 1960s • Unix is flexible, secure and based on open standards • Programs are often designed “to do one simple thing right” • Unix provides ways for interconnecting these simple programs to work together and perform more complex tasks © 2015 Regents of the University of Minnesota. All rights reserved. Supercomputing Institute for Advanced Computational Research Getting Started • MSI account • Service Units required to access MSI HPC systems • Open a terminal while sitting at the machine • A shell provides an interface for the user to interact with the operating system • BASH is the default shell at MSI © 2015 Regents of the University of Minnesota. All rights reserved. Supercomputing Institute for Advanced Computational Research Bastion Host • login.msi.umn.edu • Connect to bastion host before connecting to HPC systems • Cannot run software on bastion host (login.msi.umn.edu) © 2015 Regents of the University of Minnesota. -
Use Perl Regular Expressions in SAS® Shuguang Zhang, WRDS, Philadelphia, PA
NESUG 2007 Programming Beyond the Basics Use Perl Regular Expressions in SAS® Shuguang Zhang, WRDS, Philadelphia, PA ABSTRACT Regular Expression (Regexp) enhance search and replace operations on text. In SAS®, the INDEX, SCAN and SUBSTR functions along with concatenation (||) can be used for simple search and replace operations on static text. These functions lack flexibility and make searching dynamic text difficult, and involve more function calls. Regexp combines most, if not all, of these steps into one expression. This makes code less error prone, easier to maintain, clearer, and can improve performance. This paper will discuss three ways to use Perl Regular Expression in SAS: 1. Use SAS PRX functions; 2. Use Perl Regular Expression with filename statement through a PIPE such as ‘Filename fileref PIPE 'Perl programm'; 3. Use an X command such as ‘X Perl_program’; Three typical uses of regular expressions will also be discussed and example(s) will be presented for each: 1. Test for a pattern of characters within a string; 2. Replace text; 3. Extract a substring. INTRODUCTION Perl is short for “Practical Extraction and Report Language". Larry Wall Created Perl in mid-1980s when he was trying to produce some reports from a Usenet-Nes-like hierarchy of files. Perl tries to fill the gap between low-level programming and high-level programming and it is easy, nearly unlimited, and fast. A regular expression, often called a pattern in Perl, is a template that either matches or does not match a given string. That is, there are an infinite number of possible text strings. -
A First Course to Openfoam
Basic Shell Scripting Slides from Wei Feinstein HPC User Services LSU HPC & LON [email protected] September 2018 Outline • Introduction to Linux Shell • Shell Scripting Basics • Variables/Special Characters • Arithmetic Operations • Arrays • Beyond Basic Shell Scripting – Flow Control – Functions • Advanced Text Processing Commands (grep, sed, awk) Basic Shell Scripting 2 Linux System Architecture Basic Shell Scripting 3 Linux Shell What is a Shell ▪ An application running on top of the kernel and provides a command line interface to the system ▪ Process user’s commands, gather input from user and execute programs ▪ Types of shell with varied features o sh o csh o ksh o bash o tcsh Basic Shell Scripting 4 Shell Comparison Software sh csh ksh bash tcsh Programming language y y y y y Shell variables y y y y y Command alias n y y y y Command history n y y y y Filename autocompletion n y* y* y y Command line editing n n y* y y Job control n y y y y *: not by default http://www.cis.rit.edu/class/simg211/unixintro/Shell.html Basic Shell Scripting 5 What can you do with a shell? ▪ Check the current shell ▪ echo $SHELL ▪ List available shells on the system ▪ cat /etc/shells ▪ Change to another shell ▪ csh ▪ Date ▪ date ▪ wget: get online files ▪ wget https://ftp.gnu.org/gnu/gcc/gcc-7.1.0/gcc-7.1.0.tar.gz ▪ Compile and run applications ▪ gcc hello.c –o hello ▪ ./hello ▪ What we need to learn today? o Automation of an entire script of commands! o Use the shell script to run jobs – Write job scripts Basic Shell Scripting 6 Shell Scripting ▪ Script: a program written for a software environment to automate execution of tasks ▪ A series of shell commands put together in a file ▪ When the script is executed, those commands will be executed one line at a time automatically ▪ Shell script is interpreted, not compiled. -
“Linux at the Command Line” Don Johnson of BU IS&T We’Ll Start with a Sign in Sheet
“Linux at the Command Line” Don Johnson of BU IS&T We’ll start with a sign in sheet. We’ll end with a class evaluation. We’ll cover as much as we can in the time allowed; if we don’t cover everything, you’ll pick it up as you continue working with Linux. This is a hands-on, lab class; ask questions at any time. Commands for you to type are in BOLD The Most Common O/S Used By BU Researchers When Working on a Server or Computer Cluster Linux is a Unix clone begun in 1991 and written from scratch by Linus Torvalds with assistance from a loosely-knit team of hackers across the Net. 64% of the world’s servers run some variant of Unix or Linux. The Android phone and the Kindle run Linux. a set of small Linux is an O/S core programs written by written by Linus Richard Stallman and Torvalds and others others. They are the AND GNU utilities. http://www.gnu.org/ Network: ssh, scp Shells: BASH, TCSH, clear, history, chsh, echo, set, setenv, xargs System Information: w, whoami, man, info, which, free, echo, date, cal, df, free Command Information: man, info Symbols: |, >, >>, <, ;, ~, ., .. Filters: grep, egrep, more, less, head, tail Hotkeys: <ctrl><c>, <ctrl><d> File System: ls, mkdir, cd, pwd, mv, touch, file, find, diff, cmp, du, chmod, find File Editors: gedit, nedit You need a “xterm” emulation – software that emulates an “X” terminal and that connects using the “SSH” Secure Shell protocol. ◦ Windows Use StarNet “X-Win32:” http://www.bu.edu/tech/support/desktop/ distribution/xwindows/xwin32/ ◦ Mac OS X “Terminal” is already installed Why? Darwin, the system on which Apple's Mac OS X is built, is a derivative of 4.4BSD-Lite2 and FreeBSD. -
Shell Scripting, Scripting Examples
Last Time… on the website Lecture 6 Shell Scripting What is a shell? • The user interface to the operating system • Functionality: – Execute other programs – Manage files – Manage processes • Full programming language • A program like any other – This is why there are so many shells Shell History • There are many choices for shells • Shell features evolved as UNIX grew Most Commonly Used Shells – /bin/csh C shell – /bin/tcsh Enhanced C Shell – /bin/sh The Bourne Shell / POSIX shell – /bin/ksh Korn shell – /bin/bash Korn shell clone, from GNU Ways to use the shell • Interactively – When you log in, you interactively use the shell • Scripting – A set of shell commands that constitute an executable program Review: UNIX Programs • Means of input: – Program arguments [control information] – Environment variables [state information] – Standard input [data] • Means of output: – Return status code [control information] – Standard out [data] – Standard error [error messages] Shell Scripts • A shell script is a regular text file that contains shell or UNIX commands – Before running it, it must have execute permission: • chmod +x filename • A script can be invoked as: – ksh name [ arg … ] – ksh < name [ args … ] – name [ arg …] Shell Scripts • When a script is run, the kernel determines which shell it is written for by examining the first line of the script – If 1st line starts with #!pathname-of-shell, then it invokes pathname and sends the script as an argument to be interpreted – If #! is not specified, the current shell assumes it is a script in its own language • leads to problems Simple Example #!/bin/sh echo Hello World Scripting vs. -
UNIX X Command Tips and Tricks David B
SESUG Paper 122-2019 UNIX X Command Tips and Tricks David B. Horvath, MS, CCP ABSTRACT SAS® provides the ability to execute operating system level commands from within your SAS code – generically known as the “X Command”. This session explores the various commands, the advantages and disadvantages of each, and their alternatives. The focus is on UNIX/Linux but much of the same applies to Windows as well. Under SAS EG, any issued commands execute on the SAS engine, not necessarily on the PC. X %sysexec Call system Systask command Filename pipe &SYSRC Waitfor Alternatives will also be addressed – how to handle when NOXCMD is the default for your installation, saving results, and error checking. INTRODUCTION In this paper I will be covering some of the basics of the functionality within SAS that allows you to execute operating system commands from within your program. There are multiple ways you can do so – external to data steps, within data steps, and within macros. All of these, along with error checking, will be covered. RELEVANT OPTIONS Execution of any of the SAS System command execution commands depends on one option's setting: XCMD Enables the X command in SAS. Which can only be set at startup: options xcmd; ____ 30 WARNING 30-12: SAS option XCMD is valid only at startup of the SAS System. The SAS option is ignored. Unfortunately, ff NOXCMD is set at startup time, you're out of luck. Sorry! You might want to have a conversation with your system administrators to determine why and if you can get it changed. -
Unix (And Linux)
AWK....................................................................................................................................4 BC .....................................................................................................................................11 CHGRP .............................................................................................................................16 CHMOD.............................................................................................................................19 CHOWN ............................................................................................................................26 CP .....................................................................................................................................29 CRON................................................................................................................................34 CSH...................................................................................................................................36 CUT...................................................................................................................................71 DATE ................................................................................................................................75 DF .....................................................................................................................................79 DIFF ..................................................................................................................................84 -
The Linux Command Line Presentation to Linux Users of Victoria
The Linux Command Line Presentation to Linux Users of Victoria Beginners Workshop August 18, 2012 http://levlafayette.com What Is The Command Line? 1.1 A text-based user interface that provides an environment to access the shell, which interfaces with the kernel, which is the lowest abstraction layer to system resources (e.g., processors, i/o). Examples would include CP/M, MS-DOS, various UNIX command line interfaces. 1.2 Linux is the kernel; GNU is a typical suite of commands, utilities, and applications. The Linux kernel may be accessed by many different shells e.g., the original UNIX shell (sh), the TENEX C shell (tcsh), Korn shell (ksh), and explored in this presentation, the Bourne-Again Shell (bash). 1.3 The command line interface can be contrasted with the graphic user interface (GUI). A GUI interface typically consists of window, icon, menu, pointing-device (WIMP) suite, which is popular among casual users. Examples include MS-Windows, or the X- Window system. 1.4 A critical difference worth noting is that in UNIX-derived systems (such as Linux and Mac OS), the GUI interface is an application launched from the command-line interface, whereas with operating systems like contemporary versions of MS-Windows, the GUI is core and the command prompt is a native MS-Windows application. Why Use The Command Line? 2.1 The command line uses significantly less resources to carry out the same task; it requires less processor power, less memory, less hard-disk etc. Thus, it is preferred on systems where performance is considered critical e.g., supercomputers and embedded systems. -
Bash Guide for Beginners
Bash Guide for Beginners Machtelt Garrels Garrels BVBA <tille wants no spam _at_ garrels dot be> Version 1.11 Last updated 20081227 Edition Bash Guide for Beginners Table of Contents Introduction.........................................................................................................................................................1 1. Why this guide?...................................................................................................................................1 2. Who should read this book?.................................................................................................................1 3. New versions, translations and availability.........................................................................................2 4. Revision History..................................................................................................................................2 5. Contributions.......................................................................................................................................3 6. Feedback..............................................................................................................................................3 7. Copyright information.........................................................................................................................3 8. What do you need?...............................................................................................................................4 9. Conventions used in this