● Log into the Moodle site ● Enter the “Lecture 11” area (button 12) ● At 14:00, choose “Daily Quiz 9” ● Answer the multiple choice quiz (you have 10 min to finish) Automating the command line interface with scripts: programming basics J. M. P. Alves Laboratory of Genomics & Bioinformatics in Parasitology Department of Parasitology, ICB, USP Repetitive actions ● It is often the case that you frequently run a certain set of individual commands to perform a bigger task ● For example: maybe you have a large amount or files and you need to run several commands on each and every file ● Of course, being a programmer’s system, a Unix-like system such as Linux makes it easy to automate such tasks ● The automation is performed by saving commands in a file that we usually call a script or program ● Except for the simplest (but still very useful) scripts, certain techniques from computer programming can be used J.M.P. Alves 3 / 53 BMP0260 / ICB5765 / IBI5765 Computer programming ● Obviously, computer programming is a vast subject and we could spend a semester studying it and still not cover most things ● So, in one lecture only, we will see the basics of the basics, which are still very useful for performing some amazing feats on the command-line interface that are impossible (or at least very difficult to achieve) in a graphical shell and/or very boring/error-prone to do manually (in the CLI or otherwise) ● But… what is a computer program anyway? J.M.P. Alves 4 / 53 BMP0260 / ICB5765 / IBI5765 Computer programming ● A computer program is a set of instructions designed to perform a certain task… using a computer ● Programs are written in formally defined programming languages ● Initially, the program is written by the human as source code; later, it gets translated to machine code (zeros and ones) specific to the kind of computer processor and operating system one is using J.M.P. Alves 5 / 53 BMP0260 / ICB5765 / IBI5765 Programming languages ● There are tons of programming languages available: Java, C/C++, Python, Perl, Fortran, COBOL, assembly, Ada etc. etc. etc. ● Some are better suited to some circumstances, while others are more efficient in other situations ● But, overall, if a language is Turing-complete (or computationally universal), then it can do whatever any other such language can do J.M.P. Alves 6 / 53 BMP0260 / ICB5765 / IBI5765 Programming languages ● In practice, people use whatever they already know ● The shell is special in that it is an interface as well as a scripting language interpreter ● Huh? Scripting language? Interpreter? ● Well, there are 10 (in binary) kinds of languages… ● Interpreted or compiled (although the distinction can be somewhat arbitrary and fuzzy in some cases…) J.M.P. Alves 7 / 53 BMP0260 / ICB5765 / IBI5765 Compiled x interpreted ● Compiling is the act of transforming source code into (usually) machine code ● A language like C++, for example, is compiled: one cannot run the program right after writing the source code file –it is first necessary to compile the program, and then run the generated executable file ● A language like Python, on the other hand, is interpreted: one runs the text file with the source code directly, without the need of first generating a compiled file ● Of course, I am simplifying brutally here… But in practice, from the programmer’s point of view, that is a reasonable approximation J.M.P. Alves 8 / 53 BMP0260 / ICB5765 / IBI5765 The shell language ● As mentioned above, the shell language is interpreted ● Commands are written in a file (called a shell script) and the shell reads the file and executes each command it finds ● How to write and run a script: 1. Write the script 2.Make the script executable 3.Put the script somewhere the shell can find it ● Steps 2 and 3 are optional, since there are ways to run the script without doing them, but they are a big help if you need to run the script frequently J.M.P. Alves 9 / 53 BMP0260 / ICB5765 / IBI5765 The shell language ● Pretty much anything that can be run on the CLI can also be run from a shell script ● Besides the regular commands we have learned so far, the shell also uses, among other thing, control constructs and variables ● Script file format: #!/bin/bash # This is my beautiful script echo 'Hello, World!' # yeah! ● This script follows an ancient tradition of computer programming: the “hello, world” first program J.M.P. Alves 10 / 53 BMP0260 / ICB5765 / IBI5765 Remember the three steps ● How to write a script: 1. Write the script 2.Make the script executable 3.Put the script somewhere the shell can find it ● Here it goes again: #!/bin/bash # This is my beautiful script echo 'Hello, World!' # yeah! J.M.P. Alves 11 / 53 BMP0260 / ICB5765 / IBI5765 Remember the three steps ● If you want to run the script without making it executable and moving it elsewhere (usually some directory in the $PATH), you can run: bash script_0 ● Let’s dissect our first script #!/bin/bash : tells the shell which interpreter to use to execute the script; the #! part works as a magic number and is called shebang; for a Python script this line could be something like #!/usr/bin/python J.M.P. Alves 12 / 53 BMP0260 / ICB5765 / IBI5765 Remember the three steps ● Lines starting with # (except for the shebang) are comments and are completely ignored; that is why you did not see anything about “beautiful script” in the output of the program ● Comments can appear at the end of a line too, as seen in the last line: echo 'Hello, World!' # yeah! (the comment, as expected, did not appear in the output) J.M.P. Alves 13 / 53 BMP0260 / ICB5765 / IBI5765 Edit the script ● Add other commands to the script and run it again! ● For example: cd /usr/local/lib ls -l cd ~ echo echo "Disk space report for $HOME:" df -h . echo echo 'Done!' J.M.P. Alves 14 / 53 BMP0260 / ICB5765 / IBI5765 Variation is a good thing ● Even if all that shell scripts could do was run simple commands, they would still be very useful ● But, as with any programming language, the Bash shell can use variables too ● We have seen variables before: $PATH, $HOME, $USER etc. ● Again, think of variables as little boxes where you can store data to use later J.M.P. Alves 15 / 53 BMP0260 / ICB5765 / IBI5765 Variation is nice ● Normal shell variables can contain any kind of data, but only one piece (a string) at a time; that is what is called a scalar variable ● To create a new variable, just give it an initial value ● For example: ● my_var="blah blah" : now, there is a variable called $my_var, containing the text “blah blah” J.M.P. Alves 16 / 53 BMP0260 / ICB5765 / IBI5765 It is good to vary ● Notice that when using the variable, it must start with $ ● To create the variable, it must NOT start with $ – a bit confusing, but that’s life ● Example, modifying our first script: #!/bin/bash # This is my beautiful script echo 'Hello, World!' # yeah! var='blah blah' echo '1. And I now say $var' echo "2. And I now say $var" ● Notice that the first echo uses single quotes and the second one uses double quotes! J.M.P. Alves 17 / 53 BMP0260 / ICB5765 / IBI5765 Important! ● See that the variable was (in one case) expanded and its value used in the command ● Variable expansion only happens with double quotes ● An empty variable expands into nothing (which can lead to errors if, for example, it is used in a command that expects something) ● Variable names can contain letters, digits, and the underscore sign; no other kind of character, including space, is allowed J.M.P. Alves 18 / 53 BMP0260 / ICB5765 / IBI5765 Important! ● Variable names cannot start with digits though ● It is traditional to use all uppercase letters for variables that are not intended to change their value, i.e., constants ● For example: PI=3.14159265359 J.M.P. Alves 19 / 53 BMP0260 / ICB5765 / IBI5765 More on variables ● Let’s say you have a script with the following commands, which try to rename files by adding “1” or “b” to the end of the names, respectively: mv $file $file1 mv $file $fileb ● The shell has no way, in those commands, to tell that the second instance of $file in each command is the same as the first, i.e., we mean them to be the same variable ● The shell will “think” that $file1 and $fileb are new variables and that will lead to errors J.M.P. Alves 20 / 53 BMP0260 / ICB5765 / IBI5765 More on variables ● To make a variable name unambiguous, it can be surrounded by {} ● Those commands can be corrected like this: mv $file ${file}1 mv $file ${file}b ● Now, the shell can tell where the variable name starts and ends J.M.P. Alves 21 / 53 BMP0260 / ICB5765 / IBI5765 Quiz time! Go to the course page and choose Quiz 33 J.M.P. Alves 22 / 53 BMP0260 / ICB5765 / IBI5765 Controlling the flow ● Programming is not just about commands and variables though! ● Its true power comes from controlling the flow of the commands ● In the scripts we have played with so far, the flow has been linear, one command executed right after the other, from start to finish ● But any program that is useful will need to make decisions and repeat actions a certain number of times J.M.P. Alves 23 / 53 BMP0260 / ICB5765 / IBI5765 Controlling the flow ● Flow can be changed in two main ways: ● Branching ● Looping ● In branching, the program chooses which parts to run based on a condition (e.g., is a file a directory? or is x greater than y? did the user type “yes”?) ● In looping, a set of statements is repeated a certain number of times… or forever! The number of repetitions can be fixed or it can depend on a condition J.M.P.