Unix Essentials (Pdf)

Unix Essentials

Bingbing Yuan

Next Hot Topics: Unix – Beyond Basics (Mon Oct 20th at 1pm) 1

Objectives

• Unix Overview • Whitehead Resources • Unix Commands • BaRC Resources • LSF

2 Objectives: Hands-on

• Parsing Human Body Index (HBI) array data Goal: Process a large data file to get important information such as genes of interest, sorting expression values, and subset the data for further investigation.

3 Advantages of Unix • Processing files with thousands, or millions, of lines How many reads are in my fastq file? Sort by gene name or expression values • Many programs run on Unix only  Command-line tools • Automate repetitive tasks or commands Scripting • Other software, such as Excel, are not able to handle large files efficiently • Open Source 4 Scientific computing resources

5 Shared packages/programs

https://tak.wi.mit.edu

Request new packages/programs

Installed packages/programs

6 Login • Requesting a tak account http://iona.wi.mit.edu/bio/software/unix/bioinfoaccount.php

• Windows  PuTTY or Cygwin  Xming: setup X-windows for graphical display

• Macs Access through Terminal

7 Connecting to tak for Windows

Command Prompt user@tak ~$ 8 Log in to tak for Mac

ssh –Y [email protected]

9 Unix Commands

• General syntax  Command  Options or switches (zero or more)  Arguments (zero or more) Example: uniq –c myFile.txt

command options arguments  Options can be combined ls –l –a or ls –la

• Manual (man) page man uniq • One line description whatis ls

10 Unix Directory Structure

root /

home dev bin nfs lab . . .

jdoe BaRC_Public solexa_public solexa_lodish

page

/home/jdoe /lab/page

11 Accessing Shared Resources at Whitehead • Unix  /nfs/BaRC_Public  /lab/solexa_public  /lab/page • Windows (access using Start Menu  Search)  \\wi-files1\BaRC_Public  \\wi-files1\fink_lab  \\wi-files2\page  \\wi-htdata\solexa_public • Macs (access using Go  Connect to Server…)  cifs://wi-files1/BaRC_Public  cifs://wi-htdata/solexa_public

Where’s my lab’s share? • http://wi-inside.wi.mit.edu/departments/it/services/filestorage/labshares

12 Directory Contents

• List files/directories ls lists the contents of a directory ls –l includes additional info (eg. permissions, time stamp) Options: -l long listing -h human readable

thiruvil@tak /nfs/BaRC_Public$ ls -l total 4740 drwxrwxr-x 5 gbell barc 4096 2012-03-16 15:56 apps/ drwxrwxr-x 4 gbell barc 4096 2011-10-18 09:48 BaRC_code/ drwxrwxrwx 5 gbell barc 4096 2012-09-17 15:03 Bartel_Lab/ drwxrwsrwx 3 gbell barc 4096 2012-05-04 16:17 Cheeseman_Lab/ drwxrwsrwx 3 byuan barc 4096 2010-11-23 14:22 chip_seq/ drwxrwsrwx 2 gbell barc 4096 2012-02-21 16:26 CMT/ -rw-r--r-- 1 gbell barc 192568 2012-10-10 10:14 du.20121010a.txt

Permissions Owner Group Size (bytes) Time Stamp File or directory

13 Permissions

drwxrwxr-x

r  read Type: directory (d) User Group Others w  write symbolic link(l) x  execute • Use chmod to change permissions  user(u), group(g), others(o), all(a)  chmod u+x foo.pl (user can execute)  chmod g-w foo.pl (group can’t write)  permission denied error thiruvil@tak /nfs/BaRC_Public$ ls -l myFile.txt -rw-r--r-- 1 thiruvil barc 0 2012-10-10 13:32 myFile.txt thiruvil@tak /nfs/BaRC_Public$ chmod g+w myFile.txt thiruvil@tak /nfs/BaRC_Public$ ll myFile.txt -rw-rw-r-- 1 thiruvil barc 0 2012-10-10 13:32 myFile.txt 14 Navigating in Unix • pwd print working directory byuan@tak ~$ pwd /home/byuan • cd change directory cd fink_lab # if you are in /lab cd to home directory cd ~ cd to directory above cd .. cd to a specific directory cd /nfs/BaRC_Public • No such file or directory error

15 Organizing Files and Directories • Commands mkdir make a directory mkdir my_foo rmdir remove a directory (must be empty) rmdir my_foo mv move or rename a file/directory mv myOldFile myNewFile cp copy a file cp myOldFile myNewFile rm remove or delete a file rm myFile

Unix Tips

• Use to reuse previous commands

• Ctrl-c: stop a process that is running

• Tab-completion: – Complete commands/file names

• Unix is case-sensitive

17 Getting Files

• Getting files or directories Files wget http://www.broadinstitute.org/igv/projects/downloads/IGV_2.1.17.zip

Directories from (outside) servers scp -r [email protected]:/broad/lab/works .

18 (Un)Compressing Files

• .gz file Compress: gzip expression.txt > expression.gz Uncompress: gunzip expression.gz • .tar.gz file

Compress: tar –czvf myFiles.tar.gz myFiles Uncompress: tar –xzvf myFiles.tar.gz

Options -c create an archive (files to archive, archive from files) -x extract an archive (archive to files, files from archive) -f FILE name of archive -v be verbose, list all files being archived/extracted -z create/extract archive with gzip/gunzip • View compressed files using: – zmore,zgrep 19 Editing a File

• Command-line editors  pico  nano  emacs (emacs –nw)  vi • Graphical editors (Windows users need an X-windows emulator) Note: may not be part of standard installation  nedit  gedit  xemacs • Put an & at the end of command line to run it in the background when using a graphical editor so that you can continue to use the terminal window eg. gedit myFile.txt&

20 Viewing a File

• Display page-by-page basis more myFile.txt

 Use: to scroll, space for next page and q to quit

• Display first 15 lines of a file head -15 myFile.txt • Display last 15 lines of a file tail -15 myFile.txt • Show all contents of a file cat myFile.txt  Show hidden characters (^M or carriage return) cat –A myFile.txt • Display number of lines in a file wc –l myFile.txt 21 Output Redirection and Piping

• Write output of a command to file  Write to output file • sort myFile.txt > myFile_sorted.txt  Replace to output file • sort myFile.txt >| myFile_sorted.txt  Append to output file • sort myFile.txt >> myFile_sorted.txt

• Piping “|”: use output of one command as input for another command  sort myFile.txt | more

22 Parsing a File: cut

• Select columns of interest cut –f 9,12-15 myGeneValues.txt > col_9.12to15.txt Options: -f output only these fields -d field delimiter

23 Parsing a File: sort and uniq

• Sort on column(s) sort -k 3,3 myGeneExpression.txt | more Options: -n numerical sort -r reverse -k pos1,pos2 start a key at pos1, end it at pos2 • Get only unique entries  ensure file is sorted before running uniq uniq mySortedGenes.txt > myUniqGenes.txt Options: -c count entries -d duplicate counts

24 Regular Expressions

• Pattern matching • Easier to search • Commonly used regular expressions Regular Expression Matches . All characters * Zero or more; wildcard ^ Beginning of a line $ End of a line Example: list all txt files ls *.txt

25 Searching Within a File

• grep (global regular expression print) • Find words, or patterns, occurring in lines of a file grep TMEM geneList.txt TMEM131 TMEM9B TMEM14C TMEM66 TMEM49

Options: -v select non-matching lines -i ignore case -n print line number Example: get TMEM that does not end with 9 grep TMEM geneList.txt | grep -v "TMEM14C" | more

26 BaRC Resources • jura.wi.mit.edu

27 BaRC SOP

http://barcwiki.wi.mit.edu/wiki/SOPs

28 BaRC Scripts

29 Running Scripts on Unix

• Perl bed2gff.pl • R run_rma_customCDF.R • Python myScript.py • Matlab matlab -nodesktop -nosplash myScript.m • Java Archive (JAR) java -Xmx1000m -jar /usr/local/share/IGVTools/igv.jar

30 Running Programs/Tools on Unix • bedtools bedtools intersect -a myGenes1.bed –b myGenes2.bed Other utilities: http://code.google.com/p/bedtools/wiki/Usage • samtools samtools view myFile.bam Other utilities: http://samtools.sourceforge.net/samtools.shtml • Fastx toolkit fastx_quality_stats -i mySeq.fastq -o fastxStats_mySeq • FastQC fastqc mySeq.fastq • BLAST blastp –task blastp -db myProtDB.fa –q myProt.fa –out out.txt 31

Commonly Used Data Locations at Whitehead

Location Description /nfs/genomes Genome data: gff, gtf, fasta, bowtie indexed files, blat indexed file, etc. for several organisms /nfs/seq/Data Sequence data, including blast databases, for several organisms /nfs/BaRC_datasets Large (array/NGS) datasets: HBI, HBM 2.0

32 Scientific computing resources

33 LSF cluster jobs

https://tak.wi.mit.edu

34 Load Sharing Facility (LSF) Cluster • More computing power • Multiple jobs running at the same time

35 LSF Commands

• bsub to submit jobs bsub wc –l reads.fq bsub “sort foo.txt > sorted.txt” Options: -e error file -o standard out file -m machine -u email address • bjobs to view your jobs bjobs • bkill to kill a job bkill 237878