Basics of UNIX August 27, 2011 1 Connecting to a UNIX machine from {UNIX, Mac, Windows} See the file on bspace on connecting remotely to SCF. More generally, the department computing FAQs is the place to go for answers to questions about SCF. For questions not answered there, the SCF help page requests: “please report any problems regarding equipment or system software to the SCF staff by sending mail to ’trouble’ or by re- porting the problem directly to room 498/499. For information/questions on the use of application packages (e.g., R, SAS, Matlab), programming languages and libraries send mail to ’consult’. Questions/problems regarding accounts should be sent to ’manager’.” Note that for the purpose of this class, questions about application packages, languages, li- braries, etc. can be directed to me (not consult). 2 Files and directories 1. Files are stored in directories (aka folders) that are in a (inverted) directory tree, with “/” as the root of the tree 2. Where am I? > pwd 3. What’s in a directory? > ls > ls -a > ls -al 1 4. Moving around > cd /home/paciorek/teaching/243 > cd ~paciorek/teaching/243 > cd ~/teaching/243 > cd 243 # provided I am in ’teaching’ > cd .. > cd - 5. Copying and removing > cp > cp -r > cp -rp # preserves timestamps and other metainfo (VERY handy for tracing your workflows if you move files between machines) > mkdir > rm > rm -r > rm -rf # CAREFUL! To copy between machines, we can use scp, which has similar options to cp: > scp file.txt [email protected]:~/research/. > scp [email protected]:/data/file.txt ~/research/renamed.txt 6. File permissions: ls -al will show the permissions for the ’user’, ’group’, ’other’ • to allow a file to be executed as a program: > chmod ugo+x myProg # myProg should be compiled code or a shell script • to allow read and write access to all: > chmod ugo+rw code.q • to prevent write access: > chmod go-w myThesisCode.q 7. Compressing files 2 • the zip utility compresses in a format compatible with zip files for Windows: > zip files.zip a.txt b.txt c.txt • gzip is the standard in UNIX: > gzip a.txt b.txt c.txt # will create a.txt.gz, b.txt.gz, c.txt.gz • tar will nicely wrap up entire directories: > tar cvf files.tar myDirectory > tar cvfz files.tgz myDirectory • To unwrap a tarball > tar xvf files.tar > tar xvfz files.tgz • To peek into a zipped (text) file: > gzip -cd file.gz | less > zcat file.zip | less 3 A variety of UNIX tools/capabilities Many UNIX programs are small programs (tools, utilities) that can be combined to do complicated things. 1. For help on a UNIX program, including command-line utilities like ls, cp, etc. > man cp 2. What’s the path of an executable? > which R 3. Tools for remotely mounting the filesystem of a remote UNIX machine/filesystem • Samba protocol - see “How can I mount my home directory” on SCF Help Desk FAQs • MacFUSE • Linux: > cd; mkdir scf # create a directory as a mountpoint > sshfs bilbo.berkeley.edu: /home/paciorek/scf # mount the remote filesystem > fusermount -u ~/scf # to unmount 3 4. Cloud storage: Dropbox and other services will mirror directories on multiple machines and on their servers 5. Doing something at the UNIX command line from within R: > system(“ls -al”) 6. Editors • traditional UNIX: emacs, vi • Windows: WinEdt • Mac: Aquamacs Emacs • Be careful in Windows - file suffixes are often hidden • For statistical computing, we need an editor, not a word processor, because we’re going to be operating on files of code and data files, for which word processing formatting gets in the way. 7. Basic emacs: • emacs has special modes for different types of files: R code files, C code files, Latex files – it’s worth your time to figure out how to set this up on your machine for the kinds of files you often work on – For working with R, ESS (emacs speaks statistics) mode is helpful. This is built in to Aquamacs emacs. • To open emacs in the terminal window rather than as a new window, which is handy when tunneling the graphical emacs window is slow or impossible: > emacs -nw file.txt 8. Files that provide info about a UNIX machine: • /proc/meminfo • /proc/cpuinfo • /etc/issue • Example: how do I find out how many processors a machine has: > grep processor /proc/cpuinfo 9. There are (free) tools in UNIX to convert files between lots of formats (pdf, ps, html, latex, jpg). This is particularly handy when preparing figures for a publication. My computing tips page lists a number of these. 4 Table 1. Helpful emacs control sequences. Sequence Result C-x,C-c Close the file C-x,C-s Save the file C-x,C-w Save with a new name C-s Search ESC Get out of command buffer at bottom of screen C-a Go to beginning of line C-e Go to end of line C-k Delete the rest of the line from cursor forward C-space, then move to end of block Highlight a block of text C-w Remove the highlighted block, putting it in the kill buffer C-y (after using C-k or C-w) Paste from kill buffer (’y’ is for ’yank’) 5 The bash shell and UNIX utilities September 7, 2011 Sections with more advanced material that are not critical for those of you just getting started with UNIX are denoted (***). Note that it can be difficult to distinguish what is shell-specific and what is just part of UNIX. Some of the material here is not bash-specific but general to UNIX. Reference: Newham and Rosenblatt, Learning the bash Shell, 2nd ed. 1 Shell basics The shell is the interface between you and the UNIX operating system. When you are working in a terminal window (i.e., a window with the command line interface), you’re interacting with a shell. There are multiple shells (sh, bash, csh, tcsh, ksh). We’ll assume usage of bash, as this is the default for Mac OSX and on the SCF machines and is very common for Linux. 1. What shell am I using? > echo $SHELL 2. To change to bash on a one-time basis: > bash 3. To make it your default: > chsh /bin/bash /bin/bash should be whatever the path to the bash shell is, which you can figure out using which bash Shell commands can be saved in a file (with extension .sh) and this file can be executed as if it were a program. To run a shell script called file.sh, you would type ./file.sh. Note that if you just typed file.sh, the shell will generally have trouble finding the script and recognizing that it is executable. 1 2 Tab completion When working in the shell, it is often unnecessary to type out an entire command or file name, because of a feature known as tab completion. When you are entering a command or filename in the shell, you can, at any time, hit the tab key, and the shell will try to figure out how to complete the name of the command or filename you are typing. If there is only one command in the search path and you’re using tab completion with the first token of a line, then the shell will display its value and the cursor will be one space past the completed name. If there are multiple commands that match the partial name, the shell will display as much as it can. In this case, hitting tab twice will display a list of choices, and redisplay the partial command line for further editing. Similar behavior with regard to filenames occurs when tab completion is used on anything other than the first token of a command. Note that R does tab completion for objects (including functions) and filenames. 3 Command history By using the up and down arrows, you can scroll through commands that you have entered previ- ously. So if you want to rerun the same command, or fix a typo in a command you entered, just scroll up to it and hit enter to run it or edit the line and then hit enter. Note that you can use emacs-like control sequences (C-a, C-e, C-k) to navigate and delete characters, just as you can at the prompt in the shell usually. You can also rerun previous commands as follows: > !-n # runs the nth previous command > !xt # runs the last command that started with ’xt’ If you’re not sure what command you’re going to recall, you can append :p at the end of the text you type to do the recall, and the result will be printed, but not executed. For example: > !xt:p You can then use the up arrow key to bring back that statement for editing or execution. You can also search for commands by doing C-r and typing a string of characters to search for in the search history. You can hit return to submit, C-c to get out, or ESC to put the result on the regular command line for editing. 4 Basic UNIX utilities Table 1 shows some basic UNIX programs, which are sometimes referred to as filters. The general syntax for a UNIX program is 2 Table 1. UNIX utilities. Name What it does tail shows last few lines of a file less shows a file one screen at a time cat writes file to screen wc counts words and lines in a file grep finds patterns in files wget downloads files from the web sort sorts a file by line nl numbers lines in a file diff compares two files uniq removes repeated (sequential) rows cut extracts fields (columns) from a file > command -options argument1 argument2 ..