Introduction to Linux - June 2010 Introduction to Linux Outline • An Overview of Linux • Accessing Pawsey Supercomputers • The Linux Command Line Interface • The vi Editor I. OVERVIEW OF LINUX About Linux • It is an Operating System • Free and open source • Runs on just about anything • Is used on supercomputers and data stores • Is used in mobile phones and tablets (Android) • Has a graphical user interface (GUI) and command line interface (CLI) • Can be accessed via remote terminal windows • MobaXterm on PC, Terminal on Mac Linux Operating System • Linux is the kernel, or basic operating system. • On top of this is a software stack including utilities, libraries, compilers, office suites, email clients, web browsers etc. • Linux itself is very robust and secure. Apps are usually the problem. Linux Distributions • Different organisations package Linux with various user applications. These are “Distributions”. • They include package managers, to simplify application and dependency management. • Common distributions are: • CentOS, SuSE, SLES – used on Pawsey supercomputers • Ubuntu – used on NeCTAR • Android is based on Linux II. LINUX STORAGE LAYOUT Linux Filesystem Linux has a single hierarchical structure of directories. Everything starts from the root directory “/”. A file can be specified by the absolute path through the directory tree from the root directory. For example: /etc/motd or by its relative path to where you are. For example: ../motd Directories have a list of files in the directory as well as their properties Typical Linux Layout / bin dev etc home lib opt mnt usr var bash sda hostname user1 cdrom lib log cp sdb motd user2 usb include Linux Directory Structure /bin contains essential programs for booting and maintaining the system; e.g. bash, chmod, chown, cp, grep, kill, ps, rm, tar /dev special files pointing to system devices /etc configuration files for various applications /home contains directories for all users a system has e.g. /home/username /usr contains almost all the programs which are not located in /bin or /sbin, manual pages /var files like logs, emails User Interface from Iron Man 2 III. USING LINUX Shells A shell is a command-line interpreter. Popular shells include: • bash Bourne-Again Shell, default shell on most Linux systems and at Pawsey • sh Bourne Shell, an older shell, superseded by Bash • csh C Shell, with a syntax that resembles the C programming language syntax • tcsh an improved version of the C Shell • ksh Korn Shell Remote Access Access to Pawsey machines is via ssh • ssh creates an encrypted tunnel to the remote host. By default it starts a remote shell. • ssh can also tunnel graphical programs. • ssh uses fingerprints to identify computers … so you don't give your password to someone pretending to be the remote host magnus: 02:2e:b8:f5:0c:e8:f2:ab:7b:66:33:41:39:5a:13:12 Command line SSH Within a terminal window, type: ssh [email protected] • Common terminal programs: • Windows, use MobaXterm (download required) • Linux, use xterm (preinstalled) • OS X, use Terminal (preinstalled) or xterm (download required) Graphical Interfaces • Remote graphical interface for some tasks available. • Graphical debuggers, text editors, remote visualisation. • Low performance over long distances. • Very easy to set up for Linux, Mac and MobaXterm clients. Add -X flag to ssh. ssh -X [email protected] • For higher performance remote GUI, use VNC. See portal.pawsey.org.au/docs/Supercomputers/Remote_Visualisation Basic Commands Most Linux commands follow the general form: command [option(s)] [filename(s)] Examples: ls ls -la ls / ls -la /home/cou000 Basic Commands cd change directory ls list directory contents pwd print name of current/working directory mkdir make directories rm remove files or directories mv move (rename) files cp copy (replicate) files cat concatenate files and print on the standard output more / less view files a page at a time Note: Transferring files between Pawsey machines scp mymotd3 [email protected]:/home/username Exercise 1 Connect to magnus.pawsey.org.au using SSH: ssh [email protected] Enter the following commands: pwd cd /usr/share ls ls –l pwd cd pwd Exercise 2 Create some files and directories mkdir test1 test2 test3 ls -la cp /etc/motd test1 ls test1 cat test1/motd (use tab to autocomplete filenames) Move into and out of directories pwd cd test1 pwd cd .. pwd Deleting files and directories: rmdir test1 rm test1/motd rmdir test1 (or rm -r for recursive delete - use with care!) Inbuilt Help Information Most commands have help command line options, typically: -h --help There are also manual pages for most commands: man ls (press q to exit manual pages) Exercise 3 Run the word count utility on the motd file wc /etc/motd Read the manual page to find out what the output values refer to man wc What option would just output the word count? Command History The command line interface keeps a history of your commands, which you can view with the history command You can reuse commands from your history using the exclamation mark (also referred to as bang or shriek) and the line number (eg !42) Alternatively the up and down arrow keys can scroll through previous commands, and Ctrl-r can let you search for the matching command The exclamation mark can also be used to shortcut to the last command (!!) or last argument of the last command (!$) Piping & Grep Pipes are a UNIX feature which allows you to connect several commands together in one line and pass data from one to the next. • Most UNIX commands get input from stdin and pass output to stdout • The pipe symbol “|” directs UNIX to connect stdout from the first command to the stdin of the second command. • > will redirect the output to a file, >> will append to an existing file, < will redirect input from a file grep is a command line text search utility. It gets its name from global / regular expression / print. • grep Pawsey /etc/motd 23 Exercise 4 More handy commands history tail /etc/motd tail -5 /etc/motd grep Pawsey !$ history !! !<number> (to repeat history item <number>) What’s in the Magnus queue? squeue squeue | less (press space for next page) squeue | wc squeue | grep <username> File Permissions There are three sets of permissions. Each of these is via numerical (bitwise) addition. User Group Others read-write-execute read-write-execute read-write-execute 4-2-1 4-2-1 4-2-1 E.g. 750 = 4+2+1,4+1,0 = rwxr-x--- • Use the chmod command to set permissions: chmod 750 myfile chmod u=rwx,g=rx,o= myfile File Permissions User Group Others read-write-execute read-write-execute read-write-execute 4-2-1 4-2-1 4-2-1 • The group applies to the group associated with the file/directory. • Change the group of a file with the chgrp command. Exercise 5 Create a file and change its permissions: cd ~ cat /etc/motd > mymotd ls -l chmod 700 mymotd ls -l chmod 644 mymotd ls -l IV. THE VI EDITOR vi Concepts Two modes: • Command Mode: Enter commands to delete, cut, paste, save, quit, etc. • Text Entry Mode: Type characters in file • If you can’t tell which mode you’re in, press ESC and you will then be in command mode • vi is case-sensitive: J and j do very different things! • To use vi: • at command prompt type vi filename to edit/create file filename • type vi without filename to open new buffer Vi Commands See vi commands handout Getting into text entry mode: type i a o or O in command mode . redo most recent command u undo most recent command $ go to end of line 0 (zero) go to beginning of line More useful vi commands ➡ vi <filename> Moving around: ➡ hjkl or cursor keys to move one char/line ➡ w to move a word at a time ➡ ^ and $ for start and end of line Editing text: (two modes - move and edit, ESC to return to move) ➡ i or I for inserting text, ESC to finish ➡ a or A to append text, ESC to finish ➡ dd to delete a line ➡ cw to change a word, ESC to finish ➡ u to undo last command Saving/closing files: ➡ :w <enter> to save ➡ :wq <enter> save and quit (or ZZ) ➡ :q! <enter> to quit without saving Exercise 6 ➡ cd ~ ➡ vi mymotd Edit some text: ➡ Type “i” to insert text, then type: ➡ Today I’ve learnt a bit about Linux which should help me to use supercomputers and is just a cool thing to know! ➡ Press “esc” to exit editing mode Changing text and moving around: ➡ Use hjkl or cursor keys to move to the L in Linux and type “i” to insert the word “using”, then press “esc” ➡ Type w to move a word at a time across to the start of the word “should”, then type “cw” to change the word... type “will” and press “esc” ➡ Move the cursor to the word cool, then type $ to go to the end of line, type “a” to append and put a few more exclamation marks in Saving/closing files: ➡ :wq <enter> save and quit (or ZZ) Exercise 7 Creating a file to edit: ➡ cd ~ ➡ vi myscript Edit some text: ➡ Type “i” to insert text, then type: #!/bin/bash echo echo "Hello World!" echo ➡ Press “esc” to exit editing mode Saving/closing file: ➡ :wq <enter> save and quit (or ZZ) Changing permission on the file: ➡ At the prompt, type: chmod 700 myscript Running the script: ➡ At the prompt, type: ./myscript Exercise 8 - bonus Ideas for another script: #!/bin/bash cd STR=”Experiment run" echo $STR $1 $(date) # mkdir $1 cd $1 touch testdata # could be a wget, cp, scp for i in `seq 1 10`; do mkdir $1$i copy testdata $1$i done # For i in $( ls ); do echo Processing item: $i done cd OF=~/backups-$!-$(date +%Y%m%d).tgz Tar –cZf $OF ~/$1 Things you can do now… • Remotely log in to Pawsey machines • Transfer files to and from Pawsey machines • Run commands • Change file permissions to share with colleagues • Change file permissions to make scripts executable • Organise files into directories • Edit text files, including code and scripts • Write scripts to automate repetitive tasks Further Help Unix / Linux Tutorial for Beginners http://www.ee.surrey.ac.uk/Teaching/Unix/ UNIX Command Summary http://linuxconfig.org/linux-commands Do a Google image search for “vi cheat sheet”.