Intermediate Linux
th st !!***May subroutine 30 mpi_utils_step_parallel_edge– 31 , 2016 implicit none integer ele, ierr do ele = 1, nfaces Lecturerscall mpi_isend(commvec(ele)%field1(commvec(ele)%out_i1, (in alphabetical order): & & commvec(ele)%out_j1, & Jaakko& Leinonen commvec(ele)%out_k1), & Tomasz& 1, commvec(ele)%mpi_type_out1,Malkiewicz & & commvec(ele)%to_id, commvec(ele)%to_id, & Thomas& MPI_COMM_WORLD, Zwinger send_reqs(ele), ierr) if(ierr /= MPI_SUCCESS) then call pio_abort(ierr) end if call mpi_isend(commvec(ele)%field2(commvec(ele)%out_i2, & & commvec(ele)%out_j2, & & commvec(ele)%out_k2), & & 1, commvec(ele)%mpi_type_out2, & & commvec(ele)%to_id, commvec(ele)%to_id+tag_offset, & & MPI_COMM_WORLD, send_reqs(nfaces+ele), ierr) if(ierr /= MPI_SUCCESS) then call pio_abort(ierr) end if end do #ifdef NONBLOCK do ele = 1, nfaces All material (C) 2016 by the authors. call mpi_irecv(commvec(ele)%field1(commvec(ele)%in_i1, & This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 3.0 Unported & commvec(ele)%in_j1, & 1 License, http://creativecommons.org/licenses/by-nc-sa/3.0/ 2 & commvec(ele)%in_k1), & & 1, commvec(ele)%mpi_type_in1, & Program, May 30th
09:30 – 10:00 Morning coffee & registration 10:00 – 10:15 Introduction to the course (whereabouts, etc.) 10:15 – 10:45 Recapping Introduction to Linux (to refresh memory) 10:45 – 11:30 Basic concepts of shells 11:30 – 12:30 Lunch 12:30 – 14:30 Working with the Bash shell 14:30 – 15:00 Coffee 15:00 – 15:45 Bash job management and signals About This Course 15:45 – 16:30 Hands-on exercises & Troubleshooter: Interactive session to deal with open questions and specific problems
3 4
Program, May 31st How We Teach
09:00 – 10:00 Pipes, named pipes and redirecting All topics are presented with interactive demonstrations. 10:00 – 10:30 Coffee – Please, indicate immediately, if pace is too fast. We want to have 10:30 – 12:00 Shell commands and utilities everyone with us all the time. 12:00 – 13:00 Lunch Additionally, exercises to each of the sections will be provided. 13:00 – 14:00 Pattern matching and regular expressions 14:00 – 14:30 Coffee The Troubleshooter section is meant for personal interaction and is 14:30 – 15:00 Software and revision control (SVN & Git) (with a time-limit to 16:30 or 16:00) kept in an open end style. 15:00 – 16:00 Hands-on exercises & Troubleshooter: Interactive session to deal with open questions and specific problems
5 6 Around CSC Practicalities B1 (102,103) Kamppi Keep the name tag visible. B2 (194/5,503/4/… Pasila,… Lunch is served in the same building. – Room will be locked during lunch but lobby is open, use the lockers. Toilets are in the lobby. B2 Restaurant You are Network: here – Wi-Fi: eduroam, HAKA authentication (Training – Ethernet cables on the tables CSC room) – CSC-Guest accounts upon request B1 Transportation: – Bus stops to Kamppi/Center are located at the other side of the street (102, 103). – Bus stops to Center/Pasila are on this side, towards the bridge (194, 195, 503-6). Restaurant – To arrive at CSC the stops are at the same positions, just on opposite sides on the street. – If you came by car: parking is being monitored – ask for a temporary parking permit from the reception (tell which workshop you’re participating). Visiting outside: the doors by the reception desks are open. Username and password for workstations: given on-site. (K4 Salad bar) (THINK restaurant)
7 8 What Is a Shell?
"A shell in computing provides a user interface for access to an operating system’s kernel services." (Wikipedia) Remote login: – Normally no GUI (Graphical User Interface) – Text shell: Terminal with a set of commands Different flavours: – bash (default), tcsh (old default), zsh, corn-shell, … The basics (recapping earlier course)
9 10
What Is a Command? Listing of Directories
A command is a small program provided by the shell Print contents of a directory or information on a file The over-all structure of a command is: Detailed list of directory: command -option [optional input] $ ls –lthr /etc/ Example:”$” is not part of the command, but depicts the command prompt – -l displays additional information (detailed list in Windows) $ ls –lsh /etc/init.d (we will see later) – -h displays size in human readable format Case sensitive? Try: – -t orders by date (use -r to reverse order, i.e., oldest first) $ Ls –lsh /etc/init.d – -d keeps from going into sub-directories How to find a command? Only print directory/filenames matching a wildcard expression: $ apropos list $ ls -d /etc/*.d How to find all options? Only print directory/filenames with a 4 char suffix: $ man ls $ ls –l /etc/*.????
11 12 Moving Around in Directories Creating and (Re-)Moving Directories
Change directory: $ cd /etc/ Make a new directory: $ mkdir test1 Print work directory: $ pwd – Recursively: $ mkdir -p test2/anotherone Go to subdirectory: $ cd ./init.d Moving a directory: $ mv test2 test3 Relative path: $ cd ../ Removing a directory: $ rmdir test2/anotherone Absolute path: $ cd /etc/init.d – Recursively: $ rmdir -p test3/anotherone Combination: $ cd ../../usr Where is home: $ cd or cd ~/
13 14
Creating/Copying/(Re-)Moving Files File Permissions
Copy a file: $ cp mytest.txt othertest2.txt UNIX distinguishes between users, groups and others – Recursively: $ cp -r test test2 – Check your groups: $ groups Move a file (renaming): $ mv mytest.txt othertest3.txt Each user belongs to at least one group Remove file(s): $ rm –f mytest.txt To displays the attributes of a file or directory: – Recursively: $ rm –r test2 $ ls -l name
-rw-r--r-- 1 userid groupid 0 Jan 29 11:04 name
group
user others Size of file (usually in bytes) Number of references from file system
r = read, w = write, x = execute
The above configuration means: user can read + write, group and all others only read
15 16 File Permissions (cont'd) Finding Stuff
Changing permissions with chmod: The hard way: cd yourself through the tree and ls $ ls -l lala The elegant way: -rw-r--r-- 1 userid groupid 0 Jan 29 11:04 lala $ find /etc -name "*.conf" -print $ chmod o-r,g+w,u+x lala $ ls –l lala – Finds all config files in the /etc tree -rwxrw---- 1 userid groupid 0 Jan 29 11:04 lala The alternative: $ chmod u-xrw lala $ locate *.conf $ less lala
17 18
Finding Stuff (cont'd) Managing Space
Finding expressions inside files: How much space is left on my filesystem? – For instance, we want to know all files in the directory /etc/init.d that $ df -h contain keyword "network": Filesystem Size Used Avail Use% Mounted on /dev/sda5 22G 20G 903M 96% / $ grep network /etc/init.d/* /dev/sda1 447M 27M 396M 7% /boot .host:/ 12G 8.0G 4.1G 66% /mnt/hgfs – Or recursively: $ grep -r network /etc – Getting rid of noise: What are the sub-directories that consume the most disk-space? $ grep -r network /etc 2 >/dev/null $ du -sh ./* 1.4M bin Piping of output: 6.3M core 44K Desktop – Instead of re-directing into files, output can be piped in a chain of 696M Documents 1.2G Downloads commands: … $ grep –r network /etc 2> /dev/null| grep start| less
19 20 Login Remote Copying
Only secure connections (no telnet, rlogin) are recommended scp is like cp, but used for remote transfer Quotes are Secure Shell (SSH): $ scp lala [email protected]:'$HOME' important here $ ssh [email protected] -X rsync works local as well as remotely and helps to keep two – Option -X tunnels the graphical output or alternatively -Y, e.g. (remote) directories in sync: $ ssh [email protected] -Y $ mv lala test $ rsync –avt test/ test2 – This syncs everything in test with test2 – Important: Do not drop trailing / – Remotely: $ rsync –avt test [email protected]:'$HOME'
21 22
(De-)compressing files Archives of Files
GNU zip: Most common: tar (tape archive) – To inflate: – Take whole sub-tree and make a single compressed file $ gunzip hello-2.7.tar.gz $ tar cvzf myfirsttarfile.tar.gz /etc/init.d – To compress: . c create new archive $ gzip hello-2.7.tar . v be verbose BZIP (Bzip2): bzip2, bunzip2 . z gzip simultaneously ZIP: . f target file – To compress: $ zip myvar.zip myvariables.txt – Check contents (and simultaneously gunzip) $ tar tvzf hello-2.7.tar.gz – Listing: $ unzip –l myvar.zip – Unpack (and simultaneously gunzip): – To inflate: $ unzip myvar.zip $ tar xvzf hello-2.7.tar.gz
23 24 What is shell? Command interpreter between machine and human, a basic user interface to a computer running Unix Different flavors – bash – tcsh – zsh – corn-shell, … Basic concepts of shells
25 26
Shell vs Terminal / Console Shell concept A shell is a program which provides the traditional, text- Shell is mandatory by POSIX specification only user interface for Linux (and other Unix like systems) Reads in a typed command terminated by Enter and then parses it Shell’s primary function is to read commands that are typed into a console or terminal window and then execute them Commands may be built-ins or executable applications Commands may contain options and/or arguments Terminal - a wrapper program which runs a shell
Console – special sort of terminal, used for direct communication at a low level with the operating system
27 28 Shell commands Commands cont.
Structure of a command: A command is an instruction given by a user telling a computer to do something, e.g.: command -option [optional input] – run a single program – run a group of linked programs Examples: – whoami Commands are generally issued by typing them in at the command line and then pressing the ENTER key, which – ls –l passes them to the shell – file skype
29 30
Programs Bash
Command-line programs De-facto standard shell on most Unix-like operating systems Examples: Can generally be found everywhere – gnuplot – cal GNU project's Bourne Again SHell Open source – Bash implements essentially the same grammar, parameter and – gimp variable expansion, redirection, and quoting as the Bourne Shell – gnumeric Closed source – Main differences between Bash and Bourne: – skype . Bash Reference Manual: Major Differences From The Bourne Shell
31 32 Setting Shell and Environmental Variables (cont’d) Demoting and Un-setting Variables
Demoting an environmental variable back into a shell variable is Setting shell variable: done by: $ TEST_VAR=var $ export -n $TEST_VAR – Invoking: – No longer an environmental variable: $ echo $TEST_VAR $ printenv | grep TEST_VAR Setting environmental variable: – But still a shell variable: $ export TEST_VAR="Testing export" $ set | grep TEST_VAR Accessing the value of environmental variable is done by preceding Completely unset a variable, either shell or environmental, with the it with a $ sign, alike for shell variable: unset command: $ echo $TEST_VAR $ unset TEST_VAR
33 34
Setting Shell and Environmental Variables at Login Shell Modes
Motivation: not to set important variables up every time we start a Each shell session is classified as either login or non-login and new shell session interactive or non-interactive: Shell initialization files are the way to persist common shell – A login shell is a shell session that begins by authenticating the user. If configuration, such as: you start a new shell session from within your authenticated session, a – $PATH and other environment variables non-login shell session is started. – – shell tab-completion An interactive shell session is a shell session that is attached to a terminal. A non-interactive shell session is one that is not attached to a – aliases, functions terminal session. – key bindings
35 36 Common Operations and Shell Modes Shell Initialization
Operation Shell modes 1. A session started as a login session will read configuration details log in to a remote system via SSH: $ ssh user@host login, interactive from the /etc/profile file first execute a script remotely: $ ssh user@host 'echo $PWD' non-login, non-interactive 2. Then it reads the first file that it can find out of ~/.bash_profile, execute a script remotely and request a terminal: non-login, interactive ~/.bash_login, and ~/.profile. $ ssh user@host -t 'echo $PWD' start a new shell process: $ bash non-login, interactive A session defined as a non-login shell will read /etc/bash.bashrc run a script: $ bash test_script.sh non-login, non-interactive (/etc/bashrc in Red Hat) and then the user-specific ~/.bashrc file to run an executable with #!/usr/bin/env bash cal non-login, non-interactive build its environment. open a new graphical terminal window/tab on Unix/Linux non-login, interactive Non-interactive shells read the environmental variable called BASH_ENV and read the file specified to define the new environment.
37 38
The Aliases Aliases
An alias is a (usually short) name that the shell translates into An alias is a (usually short) name that the shell translates into another (usually longer) name or command. another (usually longer) name or command. Aliases allow you to define new commands by substituting a string They are typically placed in the ~/.bashrc file so that they are for the first token of a simple command. available to interactive subshells. They are typically placed in the ~/.bashrc file so that they are The general syntax for the alias command varies somewhat available to interactive subshells. according to the shell. In the case of the bash shell it is alias [name="value"] When used with no arguments, alias provides a list of aliases that are in effect for the current user: $ alias
39 40 Listing and Creating Aliases Listing and Creating Aliases (cont'd)
Nesting aliases: An example of an alias: $ alias lsf="ls -F" $ alias ll=“ls -lrt" $ alias lm="lsf | more" An alias can be created with the same name as the core name of a command; it is the alias that is called, rather than the command: $ alias ls="ls --color=auto -F" Use the unalias built-in to remove an alias: – Such an alias can be disabled temporarily by preceding it with a $ unalias lm lsf backslash: $ \ls Aliases are disabled for non-interactive shells (i.e., shell scripts); you have to use the actual commands instead.
41 42
Example: setting and unsetting aliases Tcsh Stands for Tenex C-shell $ cat .bashrc Enhanced version of the Berkeley UNIX C shell (csh) # .bashrc set +o noclobber Includes: PS1='\h:\W\$ ' – command-line editor export PS1 – programmable word completion alias ls='ls --color=auto -F' – spelling correction unalias ll 2> /dev/null – history mechanism – job control function ll() { ls --color -laF "$@" | more; } – C-like syntax
43 44 bash and tcsh comparison
bash tcsh invoking bash output tcsh output
which $SHELL; Which shell bash tcsh /bin/bash /bin/tcsh which $shell
Shell variables n=5 set n = 5 echo $n 5 5
Env. variables export env_n=10 setenv env_n 10 echo $env_n 10 10
echo $path; - /a /b PATH export PATH=/a:/b set path=(/a /b) echo $PATH /a:/b /a:/b Aliases alias ls="ls -l" alias ls "ls -l" ls same as ls -l same as ls –l Command PS1=abc- set prompt=abc- [ENTER] abc- abc- prompt
prog > ofile 2> (prog > ofile) >& stdout → ofile stdout → ofile Redirection [ENTER] efile efile stderr → efile stderr → efile
45 What Is a Shell?
A Unix shell is both a command interpreter and a programming language. – As a command interpreter, the shell provides the user interface to the computer. – The programming language features allow these utilities to be combined. This is called scripting. . While executing commands is essential, most of the power (and complexity) of shells is due to their embedded programming languages. Like any high-level language, the shell provides variables, flow control Working with the Bash Shell constructs, quoting, and functions. Shells offer features geared specifically for interactive use. – These features include job control, command line editing, command history and aliases.
46 47
Bash Bare Essentials Auto-Complete
In order to enter characters into the line, simply type them. If you The bash shell supports command completion, typically bound to mistype a character, you can use your Backspace key to back up and the TAB key, which allows you to complete the names of commands delete the mistyped character. stored upon your $PATH, file names, or directory names: To move around you use the LEFT and RIGHT arrow keys (or C-b and $ ls Pic
48 49 History Shell Operation
The shell provides access to the command history, the list of Basically, the shell does the following: commands previously typed. 1. Reads its input from a file, from a string supplied as an argument to You can view (and modify) the history with the history command: the -c invocation option, or from the user's terminal. $ history 2. Breaks the input into words and operators, called tokens. To access the history, use UP and DOWN arrow keys. Or, you can run 3. Parses the tokens into simple and compound commands. a command from the history by typing e.g.: 4. Performs the various shell expansions, breaking the expanded $ !h tokens into lists of filenames and commands and arguments. $ !435 5. Performs any necessary redirections. You can even interactively search the history with C-r: 6. Executes the command. $
Quoting Shell Commands
Quoting is used to remove the special meaning of certain characters A simple shell command consists of the command itself followed by or words to the shell. arguments, separated by spaces: Enclosing characters in single quotes (') preserves the literal value of $ echo Hello world $ ls /usr/bin/b* each character within the quotes. $ echo 'Hello!' More complex shell commands are composed of simple commands $ echo '$SHELL' arranged together in a variety of ways: Enclosing characters in double quotes (") preserves the literal value – Pipelines – Connecting the input and output of several commands. of all characters within the quotes, except for $, ', \, and ! – Lists – How to execute commands sequentially. $ echo "Hello!" – Compound commands – Shell commands for control flow. $ echo "$SHELL" – Coprocesses – Two-way communication between commands. Escape character (\) preserves the literal value of the next character . Coprocesses are not discussed in this course; see man bash, section that follows. Coprocesses if you would like to learn them. $ echo Hello\!; echo \$SHELL
52 53 Pipelines Lists of Commands
A pipeline is a sequence of simple commands separated by one of A list is a sequence of one or more pipelines separated by one of the control operators "|" or "|&": the operators ";", "&", "&&", or "||": $ ls | sort | more $ mkdir folder; cd folder The output of each command in the pipeline is connected via a pipe $ mkdir folder && cd folder $ mkdir folder || echo "Can't do it" to the input of the next command. That is, each command reads the previous command's output. Commands separated by a ";" are executed sequentially; the shell waits for each command to terminate in turn. – If '|&' is used, command1's standard error, in addition to its standard output, is connected to command2's standard input through the pipe. If a command is terminated by the control operator "&", the shell Each command in a pipeline is executed in its own subshell. executes the command asynchronously in a subshell. $ ls & echo 'Boo!' The exit status of a pipeline is the exit status of the last command in – This is known as executing the command in the background. the pipeline.
54 55
Compound Commands Looping Constructs
Compound commands are the shell programming constructs. until test-commands; do consequent-commands; done – Each construct begins with a reserved word or control operator and is – Execute consequent-commands as long as test-commands has an exit terminated by a corresponding reserved word or operator. status which is not zero. Bash provides while test-commands; do consequent-commands; done – Execute consequent-commands as long as test-commands has an exit – looping constructs, e.g.: $ for i in a b c d e f; do echo $i; done status which is zero. – conditional commands, e.g.: for name [ [in [words …] ] ; ] do commands; done $ for i in a b c d e f; do echo $i; done $ if [ "$SHELL" = "/bin/bash" ]; then echo Hurrah; fi – Expand words, and execute commands once for each member in the – mechanisms to group commands and execute them as a unit: name $ ( TESTVAR="I'm a test var"; echo $TESTVAR ); echo $TESTVAR resultant list, with bound to the current member. $ { TESTVAR="I'm a test var"; echo $TESTVAR; }; echo $TESTVAR for (( expr1 ; expr2 ; expr3 )) ; do commands ; done $ for (( i=0; i<5; i++ )); do echo $i; done – Similar to for-loop in the C programming language.
56 57 Conditional Constructs Conditional Expressions
if test-commands; then Conditional expressions are used by the [[ compound command and consequent-commands; the test and [ builtin commands. [elif more-test-commands; then more-consequents;] The syntax is [[ expression ]] and it returns a status of 0 for true or 1 [else alternate-consequents;] for false depending on the evaluation of expression. fi $ if [ "$SHELL" = "/bin/bash" ]; then echo Hurrah; fi $ if [ "$SHELL" = "/bin/bash" ]; then echo Hurrah; fi $ test $SHELL = "/bin/bash" && echo Hurrah – The test-commands list is executed, and if its return status is zero, the $ [[ $SHELL = "/bin/bash" ]] && echo Hurrah consequent-commands list is executed. If test-commands returns a non-zero status, each elif list is executed in turn, and if its exit status $ [[ $TESTVAR ]] && echo "Yes, we have a test var" || echo Nope more-consequents is zero, the corresponding is executed and the $ [[ -e abc ]] || echo "File abc does not exist" command completes. If 'else alternate-consequents' is present, and the final command in the final if or elif clause has a non-zero exit status, then alternate-consequents is executed.
58 59
Conditional Expressions (cont'd) Shell Functions
-e f True if file exists. f1 -nt f2 True if file1 is newer than Shell functions are a way to group commands for later execution -d f True if file exists and is a file2, or if file1 exists and using a single name for the group. They are executed just like a directory. file2 does not. "regular" command. -f f True if file exists and is a s1 = s2 True if the strings are equal. – When the name of a shell function is used as a simple command regular file. s1 != s2 True if the strings are not name, the list of commands associated with that function name is -h f True if file exists and is a equal. executed. symbolic link. i -eq j Arithmetic equals, = Functions are declared using this syntax: -r f True if file exists and is i -ne j Arithmetic not equals, ≠ name () compound-command [ redirections ] readable. i -lt j Arithmetic less than, < or -v v True if the shell variable v is set. i -le j Arithmetic less than or function name [()] compound-command [ redirections ] -w f True if file exists and is writable. equal, ≤ -x f True if file exists and is i -gt j Arithmetic greater than, > executable.
60 61 Shell Functions (cont'd) Shell Parameters
Thus, all of the following are valid function declarations and do A parameter is an entity that stores values. It can be a name, a exactly the same: number, or one of the special characters. $ function myfunc () { if [ "$SHELL" = "/bin/bash" ]; then echo Hurrah; A parameter is set if it has been assigned a value. fi; } $ function myfunc { if [ "$SHELL" = "/bin/bash" ]; then echo Hurrah; fi; Terms parameter and variable seem to serve the same purpose and } they are interchangeable in common speak. $ myfunc () { if [ "$SHELL" = "/bin/bash" ]; then echo Hurrah; fi; } – In shell grammar they mean two very different things but they are $ myfunc () if [ "$SHELL" = "/bin/bash" ]; then echo Hurrah; fi $ myfunc() ( [[ $SHELL == "/bin/bash" ]] && echo Hurrah ) accessed and used the same way. A function definition may be deleted using the -f option to the unset Positional parameters: The shell's command-line arguments. $ somecommand -a -b filename built-in: $ unset -f myfunc First parameter: $0 $1 $2 $3 Special parameters: Parameters denoted by special characters.
62 63
Special Parameters Parameters Example
$0 expands to the name of the shell or shell script. echo "I am $0" To demonstrate special $# expands to the number of positional parameters in decimal. echo "I have $# parameters" parameters we need to for p in "$@"; do echo " $p"; done $@ expands to the positional parameters, starting from one, i.e. $@ echo "My PID is $$" create a small shell script. is equivalent to $1 $2… – Type the text on the left $$ expands to the process ID of the shell. with your favourite text editor and save it with $? expands to the exit status of the most recently executed name testscript.sh foreground pipeline. Run the script with: $! expands to the process ID of the job most recently placed into $ bash testscript.sh a b c the background Run the script again with: $- expands to the current option flags as specified upon invocation, $ source testscript.sh a b c by the set builtin command, or those set by the shell itself.
64 65 Closing in, not slowing down Wait, what?
What we have covered so far: Shell input: 1. Reads its input from a file, from a string supplied as an argument to moving around, auto-complete, history the -c invocation option, or from the user's terminal. Pipelines: 2. Breaks the input into words and operators, called tokens. $ ls | sort | more 3. Parses the tokens into simple and compound commands. Lists of commands: 4. Performs the various shell expansions, breaking the expanded $ mkdir folder || echo "Can't do it" tokens into lists of filenames and commands and arguments. Compound commands: 5. Performs any necessary redirections. $ for i in a b c d e f; do echo $i; done 6. Executes the command. Functions: $ myfunc () if [ "$SHELL" = "/bin/bash" ]; then echo Hurrah; fi 7. Optionally waits for the command to complete and collects its exit status. Positional and special parameters: $ echo "I am "$0" and my parameters are $@"
66 67
Shell Expansions Brace Expansion
Each time you type a command line and press the enter key, bash Brace expansion is a mechanism by which arbitrary strings may be performs several processes upon the text before it carries out your generated. command. $ echo beg{i,a,u}n $ echo b{i,ou,ou}nd – We have seen a couple of cases of how a simple character sequence, $ echo {a..z} {1..9} {10..100..10} for example "*", can have a lot of meaning to the shell. $ touch a{b,c,d,e,f}g The process that makes this happen is called expansion. This construct is typically used as shorthand when the common With expansion, you type something and it is expanded into prefix of the strings to be generated is longer than in the above something else before the shell acts upon it. examples: $ mkdir -p very/long/path/to/{old,new,dist,bugs}
68 69 Filename Expansion Tilde Expansion
Bash scans each word for the characters '*', '?', and '['. If one of If a word begins with a tilde character ('~'), all of the characters up these characters appears, then the word is regarded as a pattern, to the first slash (or all characters, if there is no slash) are and replaced with an alphabetically sorted list of filenames considered a tilde-prefix. matching the pattern. $ ls ~ (The value of $HOME) $ ls /usr/bin/b* $ ls ~/Documents ($HOME/Documents) $ ls /usr/bin/b? $ ls ~fred ($HOME of user fred) $ ls /usr/bin/b[itz]* $ ls ~+ ($PWD) $ ls s[ao]me/*/???.txt $ ls ~- ($OLDPWD) $ cd /tmp; cp ~-/testscript.sh .
70 71
Shell Parameter Expansion Command Substitution
The dollar ('$') character introduces parameter expansion, Command substitution allows the output of a command to replace command substitution, or arithmetic expansion. the command itself. – The parameter name or symbol to be expanded may be enclosed in $ cat $(ls *.txt) braces, which are optional but serve to protect the variable to be $ a=$(echo "10*2+5" | bc -l); echo $a expanded from characters immediately following it which could be Old format is `command` and its use is discouraged. interpreted as part of the name. $ echo $SHELL is great. $ echo ${LINES}th line $ echo ${CSHELL:-bash} is still great. $ echo ${SHELL:5} is great. $ echo ${SHELL##/bin/} is great. See man bash for all possible shell parameter expansions (there are lots of them!).
72 73 Arithmetic Expansion It wasn't all that bad, or was it now?
Arithmetic expansion allows the evaluation of an arithmetic What we have covered so far: expression and the substitution of the result. The format for 1. Reads its input from a file, from a string supplied as an argument to arithmetic expansion is: $(( expression )) the -c invocation option, or from the user's terminal. $ a=10; b=20 2. Breaks the input into words and operators, called tokens. $ echo $(( a+b )) $ echo $(( (a+b)/(b-a) )) 3. Parses the tokens into simple and compound commands. $ (( a++ )); echo $a 4. Performs the various shell expansions, breaking the expanded $ echo $(( a == b )) tokens into lists of filenames and commands and arguments. $ echo $(( a < b )) 5. Performs any necessary redirections. $[expression] In older bash versions was used instead of 6. Executes the command. $((expression)), avoid it! 7. Optionally waits for the command to complete and collects its exit You can also replace $(( expression )) with let expression: status. $ let c=a+b; echo $c
74 75
Builtin Commands Some Builtins
A shell builtin is a command or a function, called from a shell, that Replace the shell and execute: is executed directly in the shell itself, instead of an external $ exec less aeg.txt executable program which the shell would load and execute. Fix command: – Shell builtins work significantly faster than external programs, $ fc 453 because there is no program loading overhead. Show and set resource limits: – Shell builtins are usually used for simple, almost trivial, functions, $ ulimit -a such as text output. $ ulimit -c 0 . Some examples from bash include cd, echo, and history. The help Set default file permissions: bash-builtin command will list all builtins. $ umask -S $ umask 022
76 77 Content PUTTING PROGRAMS TO BACKGROUND SENDING SIGNALS TO RUNNING PROGRAMS SCREEN
Bash job management and signals
78 79
Job control Putting processes to background
• Linux supports executing multiple processes in parallel or in series • Put process/program in the background: • Selectively stop (suspend) the execution of processes and continue – command & (resume) their execution at a later point – Example: • Usually via interactive interface supplied jointly by the operating $ gedit & system kernel's terminal driver and shell (e.g. bash) • The & operator puts command in the background and free up terminal • Commands typed at a shell prompt one by one – When bash starts a job asynchronously (in the background), it prints a • Launched program takes control of screen, when it is done, the shell line that looks like: prompt to type a new set of commands [1] 2565 • Sometimes a need to carry out tasks in background and use the – Job sends a message to the terminal upon completion: terminal for other purposes [1]+ Done gedit
80 81 Background jobs Manipulating the jobs • The command which runs in background is called a job • In bash, typing the suspend character: (typically ^Z, Ctrl-Z) while a process is running causes that process to be stopped and returns • Background jobs are those whose process group ID differs from the control to bash terminal's • Operating system on which bash is running must support job control • Other commands can be typed while background command is running • Delayed suspend character (typically ^Y, Ctrl-Y) causes the process to be stopped when it attempts to read input from the terminal • Background processes are immune to keyboard-generated signals • Manipulate the state of the job: • Only foreground processes are allowed to read from or write to the terminal - $ bg – continue in the background • List active jobs: - $ fg – continue in the foreground $ jobs - $ kill
82 83
kill command Screen
• Kill a process (by default) • Multiple shell windows from a single SSH session. • Sends a signal to a process • Keep a shell active even through network disruptions – Default signal: TERM (terminate) • Disconnect and re-connect to a shell sessions from multiple locations – (typically ^Z, Ctrl-C) can be used to terminate a running job • Run a long running process without maintaining an active shell session • If program does not exit when given the TERM signal, we can escalate the signal by passing the KILL signal – If not available, one needs to install screen – Example: $ kill -KILL
84 85 Screen usage • Launching: Screen usage (cont’d) $ screen • Switching between windows: • Terminating: Ctrl-a – n - next window $ exit Ctrl-a – p - previous window • • Control command inside a screen: Example: Ctrl-a Ctrl-a - 1 - screen • Example: • Detaching from screen: Ctrl-a – ? - screen help page Ctrl-a – d • Creating windows: • Resuming (re-attaching) screen: Ctrl-a – c $ screen –r (screen –r
86 87 Forward, forward, there is no turning back!
What we have covered so far: 1. Reads its input from a file, from a string supplied as an argument to the -c invocation option, or from the user's terminal. 2. Breaks the input into words and operators, called tokens. 3. Parses the tokens into simple and compound commands. 4. Performs the various shell expansions, breaking the expanded tokens into lists of filenames and commands and arguments. 5. Performs any necessary redirections. Pipes, Named Pipes and Redirecting 6. Executes the command. 7. Optionally waits for the command to complete and collects its exit status.
88 89
File Descriptors Redirections
A file descriptor "is an abstract indicator used to access a file or Before a command is executed, its input and output may be other input/output resource, such as a pipe or network redirected using redirection operators |, <, > and/or >>. connection." (Wikipedia) – The pipeline is a simple redirection where output (stdout) of each A file descriptor is a non-negative integer number, which is used as command in the pipeline is connected via a pipe to the input (stdin) a reference pointer to a file or other input/output resource. of the next command. $ ls | wc -l There are three standard file descriptors: The redirection operators may precede or appear anywhere within – Standard input, stdin, 0 a simple command or may follow a command. Redirections are – Standard output, stdout, 1 processed in the order they appear, from left to right. – Standard error, stderr, 2 By default, the shell assigns stdin into your terminal's input and both stdout and stderr are connected to your terminal's output.
90 91 Redirecting Output Appending Redirected Output
The general format for redirecting output is: [n]>file Unless the shell option noclobber has been set redirecting to a file Redirection of output causes the file to be opened for writing on file always overwrites an existing file. descriptor n, or the standard output if n is not specified. If you need to append to a file instead of overwriting it, use the >> $ ls >output operator instead of the > operator: $ grep adg * 2>error $ ls >>output Multiple redirections are also possible using file descriptor $ grep adg * >>output 2>&1 duplication, e.g. to redirect both stdout and stderr to a file: To prevent accidental overwriting set the noclobber variable: $ grep adg * >output 2>error $ set -o noclobber $ grep adg * >output 2>&1 Redirecting to a pipe is also possible: $ grep adg * 2>&1 | less
92 93
Some Special Files Redirecting Input
Unix has three special, so-called pseudo device files, which are The general format for redirecting input is: [n]
94 95 A Bizarre Experiment Here Documents
You can also open a file for both input and output: The format of here-documents is: $ echo 1234567890 >weirdfile <<[-]word $ exec 3<>weirdfile here-document $ read -n 4 <&3 word $ echo -n . >&3 A here-document is an input redirection using source data specified $ exec 3>&- $ cat weirdfile directly at the command line until a line containing only word is seen. – All of the lines read up to that point are then used as the standard input for a command.
96 97
Here Documents (cont’d) Here Documents (cont’d)
Here-documents are commonly used in scripts for printing out A variant of here documents, the format is: << 98 99 Named Pipes Creating a Named Pipe A named pipe (FIFO) works much like a regular pipe, but does have A named pipe is created with mkfifo command: some noticeable differences: $ mkfifo -m a=rw MYFIFO – Named pipes exist as a device special file in the file system. FIFO files can be quickly identified in a physical file system by the – Processes of different ancestry can share data through a named pipe. "p" indicator seen here in a long directory listing: $ ls -lF MYFIFO – When all I/O is done by sharing processes, the named pipe remains in prw-rw-rw- 1 jaakko staff 0 Oct 15 13:16 MYFIFO| the file system for later use. Also notice the vertical bar ("pipe sign") located directly after the file name. 100 101 Using a Named Pipe We're coming to the end of our road 1. On your terminal: 2. Open another terminal and What we have covered so far: $ cat < 102 103 Command line text/file utilities HOW TO ELEGANTLY MANIPULATE FILES Shell commands and utilities 104 105 paste – add files side-by-side cut – trim your output paste [-d del -s] file1 file2 ... cut [-b -c -f -d del -s] file1 file2 ... – merge lines of files separated by tabs – remove sections from each line of files -d del use other delimiter than tabs -b byte list for selection -s sequential (if more than two input files) processing: [(file1 + file2) + -c character list for selection file 3] + file4 -f select fields $ paste count.txt sheep.txt > sheep_tab.txt -d del use other delimiter than tabulator that merges count and sheep using tabs as delimiters -s skip lines not containing delimiters – same, but with space as delimiters $ cut -f 1 sheep_tab.txt $ paste -d ' ' count.txt sheep.txt > sheep_space.txt $ cut -d ' ' -f 1 sheep_space.txt – prints 1st column (i.e, original count.txt file contents) 106 107 wc (word count), counting lines [and sheep] cat (curiosity killed the …) wc [-l -w -c -m] filename cat [-n -E -v -T] file1 file2 … – l for counting lines: – concatenates files and prints to stdout $ wc -l sheep_space.txt -n numbering output lines (e.g., source-code listing) how many (lines in) sheep? -E indicate ends with a $ -v show non-printing -w counts the words -T indicate tabs -c counts the bits $ cat -n sheep_space.txt > sheep_lines.txt -m counts the characters numbers the lines in sheep_space.txt and adds the column to the new file $ cat -T -E sheep_tab.txt 108 109 head and tail sort – bring things in order head [-n N] filename sort [-d -f -g ] file1 file2 ... – output head of a file: $ head -n 4 sheep_space.txt – sort lines of text files -n N for display N first lines -d dictionary (alphanumeric) order -f ignore upper/lower case tail [-n N -f --pid pid] filename -g general numeric – output the end of a file: $ tail -n 4 sheep_space.txt – Spot the difference: -n N for display N last lines $ sort -d sheep_line.txt -f continuously display file while being appended (very useful if following log-files) $ sort -g sheep_line.txt -pid pid terminate tail after process with ID pid has ended 110 111 uniq – removing redundancy touch – all about timestamps uniq [-c -f -s -w ] file1 file2 ... touch [-a -m --reference=reffile] file1 … – filter adjacent matching lines -a changes access time (usually not shown) -c prefix lines by number of their occurrence -m changes modification time -f N avoid comparing the first N fields --reference=reffile uses file’s timestamps -s N avoid comparing the first N characters $ ls -ltr *.* → → → → → → → → → → → → -rwxrwxrwx 1 root root 76 Oct 6 14:33 sheep.txt -rwxrwxrwx 1 root root 80 Oct 6 14:33 count.txt $ touch -m sheep.txt -w N compare not more than N characters/line $ ls -ltr *.* → → → → → → → → → → → → -rwxrwxrwx 1 root root 80 Oct 6 14:33 count.txt -rwxrwxrwx 1 root root 76 Oct 8 09:20 sheep.txt $ uniq -c -f 1 -w 10 sheep_line.txt $ touch -a count.txt – skips the first column (the previously inserted numbers) and matches $ ls -ltr -u *.* → → → → → → → → → → → → -rwxrwxrwx 1 root root 76 Oct 8 09:11 sheep.txt $ touch --reference=count.txt sheep.txt -rwxrwxrwx 1 root root 80 Oct 8 09:24 count.txt in max. 10 characters (i.e., avoiding the later columns) and prefixes the $ ls –ltr *.* → → → → → → → → → → → → -rwxrwxrwx 1 root root 76 Oct 6 14:33 sheep.txt number of occurrence (Hint: try with –f 2) -rwxrwxrwx 1 root root 80 Oct 6 14:33 count.txt 112 113 which – how to find executable files which executablename – Find the directory of an executable file in the system Command line process utilities $ which ElmerSolver HOW TO MAKE YOUR WORK MORE EFICIENT – The filename, naturally, has to be in the environment variable $PATH echo contents – The contents can be anything from an environment variable $ echo $PATH – to a string $ echo “hello world” 114 115 stat + file – identifying your files Real world workflow example stat filename We are now going to demonstrate some practicalities using a real – display file or file system status world simulation problem $ stat $(which ElmerSolver) – We will use the Finite Element package Elmer (which is installed on – $(…) puts the output of the expression inside the brackets as input to that virtual appliance) to do some test simulations with a toy-model stat for a glacier file filename – Apologies, if this particular example doesn’t fall into your field of – determine file type expertise (the probability of a second Glaciologist joining this course is close to zero) or interest (but perhaps you want to learn something $ file $(which ElmerSolver) about glacier flow) $ file $(which ElmerSolver_mpi) 116 117 Real world workflow example Real world workflow example $ cd Work What if I launch the case, but leave the session I launched from? $ tar xvzf testglacier.tar.gz – Open a second terminal: $ top Unpacking our test – Launch again, but now redirect the output into a file and send to bg: $ cd TestGlacier case $ ElmerSolver Stokes_prognostic.sif > out1 & $ ElmerGrid 14 2 testglacier.msh –autoclean – You BTW can check the output with: $ ElmerSolver Stokes_prognostic.sif $ tail -f out1 $ tail -f out| grep -i Time takes Don’t worry, this is just to get the mesh – Before you reach the end (500 steps), log out (head symbol lower about 2 right) Minutes $ paraview & – Log back in, launch a terminal 118 119 Real world workflow example Real world workflow example Can I guarantee a job to continue, if I leave the shell I launched Now, lets launch a lot of jobs. Lets write a script … from? $ gedit submit.sh – Open a new terminal and go to: $ cd Work/TestGlacier #!/bin/bash – Launch with: for i in {1..9} do $ nohup ElmerSolver Stokes_prognostic.sif > out2 & ElmerSolver Stokes_prognostic.sif > out.$i & – Before you reach the end (500 steps), log out (head symbol lower right) done – Log back in, launch a terminal – Make it executable: $ chmod u+x submit.sh Log back in and compare out1 and out2 – Open a second terminal: $ top – Launch the jobs: $ ./submit.sh 120 121 Real world workflow example Real world workflow example The 9 jobs share about the same amount of resources $ nice ElmerSolver Stokes_prognostic.sif > out.10 & top - 20:51:47 up 2:30, 3 users, load average: 7.74, 2.75, 1.07 top - 20:51:47 up 2:30, 3 users, load average: 7.74, 2.75, 1.07 Tasks: 161 total, 12 running, 149 sleeping, 0 stopped, 0 zombie Tasks: 161 total, 12 running, 149 sleeping, 0 stopped, 0 zombie %Cpu(s): 93.4 us, 5.5 sy, 1.2 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st %Cpu(s): 93.4 us, 5.5 sy, 1.2 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st KiB Mem: 1017780 total, 931040 used, 86740 free, 14668 buffers KiB Mem: 1017780 total, 931040 used, 86740 free, 14668 buffers KiB Swap: 1046524 total, 14496 used, 1032028 free. 359032 cached Mem KiB Swap: 1046524 total, 14496 used, 1032028 free. 359032 cached Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 4228 elmerus+ 20 0 203120 12524 5928 R 23.6 1.2 0:21.40 ElmerSolver 4228 elmerus+ 20 0 203120 12524 5928 R 23.6 1.2 0:21.40 ElmerSolver 4232 elmerus+ 20 0 203120 12524 5928 R 23.6 1.2 0:20.00 ElmerSolver 4232 elmerus+ 20 0 203120 12524 5928 R 23.6 1.2 0:20.00 ElmerSolver 4227 elmerus+ 20 0 203120 12524 5928 R 23.3 1.2 0:21.32 ElmerSolver 4227 elmerus+ 20 0 203120 12524 5928 R 23.3 1.2 0:21.32 ElmerSolver 4225 elmerus+ 20 0 203120 12524 5928 R 23.0 1.2 0:19.65 ElmerSolver 4225 elmerus+ 20 0 203120 12524 5928 R 23.0 1.2 0:19.65 ElmerSolver 4226 elmerus+ 20 0 203120 12524 5928 R 20.3 1.2 0:19.13 ElmerSolver 4226 elmerus+ 20 0 203120 12524 5928 R 20.3 1.2 0:19.13 ElmerSolver 4224 elmerus+ 20 0 203120 12524 5928 R 20.0 1.2 0:19.04 ElmerSolver 4224 elmerus+ 20 0 203120 12524 5928 R 20.0 1.2 0:19.04 ElmerSolver 4229 elmerus+ 20 0 203120 12524 5928 R 20.0 1.2 0:20.43 ElmerSolver 4229 elmerus+ 20 0 203120 12524 5928 R 20.0 1.2 0:20.43 ElmerSolver 4230 elmerus+ 20 0 203120 12524 5928 R 20.0 1.2 0:19.57 ElmerSolver 4230 elmerus+ 20 0 203120 12524 5928 R 20.0 1.2 0:19.57 ElmerSolver 4231 elmerus+ 20 0 203120 12524 5928 R 20.0 1.2 0:19.48 ElmerSolver 4231 elmerus+ 20 0 203120 12524 5928 R 20.0 1.2 0:19.48 ElmerSolver 4278 elmerus+ 30 10 202988 12024 5616 R 2.3 1.2 0:00.27 ElmerSolver 122 123 pgrep and pkill – task management The ordinary way: find the process ID and $ kill -9 PID Easy way to find all process ID of one application: $ pgrep ElmerSolver Easy way to kill all processes of one application (use with care): $ pkill ElmerSolver Check your top window – all Elmer processes are gone 124 Pattern matching with more and less • less filename1 filename2 – less is more than more – h for help screen – Searching: /pattern or ?pattern – Continue search: n, N or ESC-N – Matching lines: &pattern Pattern matching and regular expressions – Goto line number: linenumber g – Multiple files: :n (next) :p (previous) 125 126 The grep-family The grep-family • egrep = grep -E • grep: matches patterns in a file/stdout and outputs the corresponding line – -E interprets pattern as extended regural expression, e.g.: – We already had a fair amount of examples $ ls -d /dev/*|grep -E '(a$|sd)' – Some options: – searches for files, which either end with an a or contain sd in their names • -i ignores upper/lowercase • fgrep = grep -F • -v invert match, e.g., – Interprets pattern as string (so no regular expressions). Spot a difference: $ ls -d /dev/*|grep -v 'sd' $ ls /etc|grep ..conf . all filenames not containing sd $ ls /etc|grep -F ..conf • -f patternfile obtain the pattern from a file • -w only lines that contain the pattern as whole words Note: Direct invocation as either egrep or fgrep is deprecated, but is provided $ ls -d /dev/*|grep -w 'sda' to allow historical applications that rely on them to run unmodified 127 128 The grep-family Regular Expressions • For searching (e.g., {e,f,r}grep) • rgrep = grep -r • Any word is a regular expression: e.g., – -r option for recursive search, e.g., $ ls -d /dev/*|grep 'sda' $ rgrep network /etc/init.d – Will find any (non-hidden) files/directories in /dev that contain sda – searches recursively for file containing the pattern network in their name . grep would not find a single occurrence, as all files are in sub- • Brackets [] describes a character list: directories of /etc/init.d $ ls -d /dev/*|grep '[1_]' – Note: Direct invocation as either egrep, fgrep is deprecated, – Will find any (non-hidden) files/directories in /dev that contain either 1 or _ in their name 129 130 Regular Expressions Regular Expressions • Beginning of the line ^: • Single character wildcard: $ ls -d /dev/*|grep '^/dev/s' $ ls -d /dev/*|grep '.da' – Will find any files/directories in /dev that start with a s – Will find any files in /dev that contain a 3 character expression • End of the line $: which ends with da (e.g., sda1, hda1, ...) $ ls -d /dev/*|grep 'a$' • The ^ operator excludes (in [] brackets): – Will find any files/directories in /dev that end with an a (e.g., $ ls -d /dev/*|grep tty|grep '[^S]' /dev/sda) • * preceding character 1 or more times: – Finds files in /dev including sequence tty but only highlights everything but S $ ls -d /dev/*|grep '[o*]' – Displays all files in /dev containing o, oo, ... 131 132 Regular Expressions • Character classes: Expression Purpose [:alpha:] [:digit] Any alphabetic/digit character [:lower] [:upper:] Anny lower/upper case character [:space:] Any white-space character (space, tab, ...) [:print:] [:cntrl:] Any printable/non-printable character $ cat file|grep '[[:upper:]]' – displays all lines of file with upper-case letters 133 Revision/Version control systems? Modern software is developed by teams Version control: – any action that provides control over changes in source code (can be a person, tar-archives and patches, nowadays usually software) – Repository (may it be distributed or central): – Updates can be made by everyone/any time and have to be consistent – File history has to be trace-/revertable Software and revision control Is this important to my daily computing activities? – Generally not, if you are not a developer! – Yes, in cases where you pick code from such repositories! 134 135 Revision/Version control systems? Concurrent Versions System (CVS) Two principal approaches to source code management: CVS is hardly used nowadays, but you never know 1. Client/Server: developers use a single shared repository and work Client/Server model = single central repository on local copies they commit to this repository The repository usually is defined in environment $CVSROOT 2. Distributed: developers have their own repository and changes to $ cvs checkout project retrieve project from repository the sources are copied between them in an extra step $ cvs update download updated/new files from repository $ cvs add file add file (from your own copy) to repository $ cvs commit -m “message” [file] commit changes to repository $ cvs status print status Graphs from http://git-scm.com 136 137 Apache Subversion (SVN) GIT In large parts very similar to CVS Distributed revision control system = each user is running her own Client/Server model = single central repository repository Allows for branching/merging – Initiated by Linux Torvalds for use in Linux kernel development $ svn checkout URL retrieve project from repository at URL – A lot of modern software projects rely on git $ svn update download updated/new files from repository – Supports branches $ svn add file add file (from your own copy) to repository Popular software distribution sites run git-servers, such as $ svn commit -m “message” [file] commit changes to repository SourceForge and GitHub $ svn status print status e.g.: $ svn checkout svn://svn.code.sf.net/p/keepass/code/trunk keepass-code 138 139 GIT GIT Adding a file: Cloning a repository: $ cd ReleaseNotes $ cd /tmp $ echo “New release” > Release_Oct15.txt $ git clone https://github.com/ElmerCSC/elmerfem.git $ git add test.txt On branch release Check, which branch: Now the file is staged Your branch is up-to-date with 'origin/release'. $ cd elmerfem Cloning into 'elmerfem'... Changes to be committed: remote: Counting objects: 19601, done. $ git status (use "git reset HEAD 140 141 Version control on Windows, OS X GUI, Linux? Yes – also neatly integrated in your WinExplorer SVN: TortoiseSVN (http://sourceforge.net/projects/tortoisesvn/) GIT: Git for Windows, Mac (https://git-scm.com) – TortoiseGIT (https://tortoisegit.org/) On Linux: svn, cvs and git able to be installed as packages. E.g., – Debian/Ubuntu: $ sudo apt-get install git – Fedore/RedHat: $ sudo yum install git – Also as GUI: e.g., gitk 142