Introduction to UNIX Today’s Class • What is a cluster? • History of UNIX • Why is UNIX the operating system of scientific computing? • UNIX basics – Working from the command-line • Practice Problems Connecting to the UH cluster • Connect via ssh (“secure shell”) • Installed by default on any UNIX-based machine (Mac/Linux) • Mac (Applications->Utilities) • Linux Machine (Terminal) • Windows – PuTTY, a free ssh software • Need: Check out a node to use in interactive • A user account mode: • IP address !$ qsub –I –l walltime=04:00:00 ! • Format: !$ exit! • $ ssh username@IP_address! Do not run jobs on the • For example: login nodes!! $ ssh [email protected]! $ ssh [email protected]! Using a cluster • Log onto the “login” node Loosely connected or tightly connected computers that • Home directory – store your files work together so that in many • Do not run processes on the login nodes!!! respects they can be viewed as a single system. • http://en.wikipedia.org/wiki/Computer_cluster To use the “compute” nodes • Submit a job (“qsub –I” ; “exit” when done) • Check out many nodes at once using a job script - Connected through fast LANs – high speed data transfer - Each “node” (or server) runs its own instance of the operating system - User logs into a single “login” node, where they have a home directory to store files - Request to use multiple nodes via a job scheduler software - Job scheduler assigns nodes to the particular user, runs the job, and returns the results. - Check out single nodes for interactive use Beowulf Cluster Off-the-shelf computers connected to one another via a network One host and multiple clients (“master” node and “slave” nodes) Easy to grow and update – just add more computers Two ways of using a cluster: (1) One big job, distributed across the nodes “parallel computing” (2) Many individual jobs, each run on its own computer “batch” computing Any given run on a node is probably no better/faster than your computer. History of UNIX • Created in 1969 as part of AT&T’s Bell Labs • Multi-user, multi-tasking operating system – lets many programmers simultaneously work on the same computer • OS coordinates the use of the computer’s resources (real-time sharing) • Each user is unaware of the activities of other users • First attempt called MULTICS • Key Features: • Multi-user • Multi-tasking • System Portability • UNIX tools (modular design - many small programs rather than one monolithic program) Source: http://www.bell-labs.com/history/unix/tutorial.html Ken Thompson and Dennis Ritchie porting UNIX to the PDP-11 History of UNIX • 1976-1977 Ken Thompson took a sabbatical at UC Berkeley – taught a course on UNIX • Students and professors at Berkeley continued to develop the system – led to the development of Berkeley Software Distribution (BSD) UNIX • UNIX eventually became licensed/trademarked, and ‘true’ UNIX systems follow the UNIX specification • Many UNIX operating systems are not official UNIX, but “Unix-like” – these include Linux and MacOSX Evolution of UNIX and UNIX-like Systems http://en.wikipedia.org/wiki/Unix GNU + Linux = Open source UNIX Richard Stallman, Free Linus Torvalds Software Foundation GNU is not UNIX! GNU: Unix shell, utilities Linux: kernel GNU Public License (GPL) • Most widely used free software license Users are free to use, modify, and redistribute the software However, the same terms and conditions apply to the derivative work. GPL is a “copyleft” license. Copyrights limit use, modification and redistribution. “Copyleft” does the opposite – ensures that the code will never become proprietary. Three Levels of the UNIX Operating System • Kernel – core of the operating system; controls the hardware • Shell – acts as the interpreter between the user and the computer • Applications/Utilities UNIX directory structure /bin = GNU/Linux utilities (e.g., ls, cd) /home = user accounts /lib = libraries /usr = where additional/optional binaries (/usr/bin) or libraries (/usr/lib) are installed “which” or “whereis” can you figure out where a given program or library is installed User has RWX (read-write-execute) permissions only for their only home directory (and sub-directories) “root” (administrator) access is required to install at the system level “su” (superuser) or “sudo” (in front of a command) will execute that command as root UNIX directory structure • Files are organized into directories. • Everything is a file (including devices, executables, scripts, links) • File suffixes are for convention only (but encouraged!) • Directories are organized in a hierarchical manner • Pathnames show the location of the file ~ Home directory • UNIX uses forward slashes (“/”) to . This directory separate directories .. Up one level in the • Pathnames can be absolute or relative directory tree • The terminal “/” when navigating to a directory is optional “cd /home/ostrow“ is the same as “cd /home/ostrow/”, which is the same as “cd ~“ “cd /home/ostrow” == “cd /home/ostrow/” == “cd ~” Navigating on UNIX machine • Home directory • List contents (ls) • Change directory (cd) • Print working directory (pwd) - cd .! ls! - cd /! ls –l! - cd ~! ls –la! - cd ./! ls -lrt! - cd ..! - cd /project! - cd /home/ostrow! - pwd! - locate! “Root” (= “Administrator”) • Dangerous for any user to be able to modify files on a multi-user UNIX cluster • A typical user has “write” privileges that are limited to his or her own directory (and its subdirectories) • Will have broad read/execute privileges, so can run applications • Need to be a superuser (“root”) to be able to write at the system level • Ask sys admin to install software at the system level OR do a “local” install (i.e., install in your own directory). Three ways to have root privileges 1. Log in as root ssh [email protected]! 2. Go into “superuser” mode: $ su! (You must be on the superuser list.) 3. Issue a command as “superuser” $ sudo [your command here]! http://xkcd.com/149/ A Note to Mac Users • Although Macs are UNIX-based, many of the standard utilities are not installed by default • Download the Apple Command-Line Tools Two ways: Install Xcode (free from Apple website) OR register as an Apple Developer and use downloads site • Consider installing a package manager • (fink, MacPorts, Homebrew) • With command-line tools, you will have most UNIX utilities installed and ready to use • Windows machine – Try a Linux virtual machine or Cygwin UNIX command line • General format of UNIX commands: command [-options] [target] For example: ls! ls ~! ls /users/eaostrow! ls -altr /users/eaostrow! ! less countATGC.rb ! mv countATGC.rb countAGCT.rb! cut -f1 myfile.dat! wc -l snpfile! ! Useful Commands Consult the manpages if you can’t remember how Navigation: the syntax! ls (list files)! e.g., “man ls”, “info ls” pwd (print working directory)! cd path_to_dir (change directory)! ! Creating Files: touch filename! ! Renaming or Moving Files: cp filename destination_dir! scp filename user@hostname:path_to_dest! mv filename new_filename! rm filename #DANGEROUS! ! mkdir my_directory! rmdir my_directory! rm –rf ./mydir #DANGEROUS! ! ! Useful Commands (cont’d) Reading files: less [-S] filename ! Keystrokes!! in less:! more filename! q to quit; ! cat filename! SPACE to scroll down; ! / to search; Finding text in files: grep pattern filename! sort filename! cut [-f] filenames ! wc [-l] filename (=word count) Grep and sort have many powerful options! Printing lines: head filename ! grep:! tail filename! -c, -n, -A, -B, -v! ! Redirect: sort:! > (redirect to file, replace)! -k –n -r! >> (redirect to file, append) ! ! ! ! UNIX command line (cont’d) • Power of the UNIX command line is ability to build complex tasks by chaining commands togethers: Using “pipes”, the output of one command becomes the input to the next Example: $ less QS73_m15q338.pileup | grep -c DDB0169550 (counts number of mitochondrial sites) • Re-direct output to file (>, >>) $ less QS73_m15q338.pileup | grep -c DDB0232428 | cut –f2,4 > chr1.cover.dat! ! • By chaining, grepping, and redirecting, you can efficiently filter and customize data sets. Ctrl-A Beginning of the line Keystrokes Ctrl-B Moves cursor backwards Ctrl-C Cancel currently running • Tab – auto-completes command • (Tab again to see completion Ctrl-E End of the line options) Ctrl-F Moves cursor forward • Up-arrow / Down-arrow Ctrl-H Delete / Backspace Cycle through previous Ctrl-K Delete forward from cursor commands Ctrl-L Redraws the screen (same as clear) Ctrl-P Pastes previous line !!! Executes the (same as up-arrow) previous command Ctrl-U Delete backwards from reset! Reseats the terminal cursor if not displaying Ctrl-W Deletes last word typed properly Ctrl-Z Suspends a running clear! Clears the screen process Killing a process • Cancel a job you’ve started on the cluster: • $ qstat –u username • $ canceljob <jobid> • If your computer becomes unresponsive: Ctrl-C • Open a second terminal - Type “kill <psid>” , where <psid> is your process ID - Find your psid: type “ps aux” or “ps aux | grep <etc>” For Monday’s Quiz • Know how to navigate up and down the UNIX directory structure • Get back to your home directory (efficiently) • Print your working directory • List directory contents • Create, copy, remove, move, and rename directories and files (touch, cp, rm, mv, mkdir, rmdir) • Difference between a relative and an absolute path; how to find and specify both. For example: “cd ../” “cd ./” “cd ~” “pwd”, “less ../myfile.dat”, “less ~/myfile.dat” !etc.) • Log on remotely (ssh); copy files from one machine to another (scp) • How to obtain a node for an interactive job (qsub) • How to exit from a shell (exit); how to kill a process (Ctrl-C) • Keystrokes to type efficiently on the command line; tab completion • How to access, interpret, and use manpages • How to read the contents of a file (less, cat, more, echo, >, >>, head, tail) .