Useful Shell Commands Head/Tail, Cut, Sort, Uniq

Total Page:16

File Type:pdf, Size:1020Kb

Useful Shell Commands Head/Tail, Cut, Sort, Uniq Useful shell commands head/tail, cut, sort, uniq Virginie Orgogozo March 2011 cat = shows the content of a file less = shows the content of a file Head and tail cd pcfb/examples/ $head ctd.txt shows the first 10 lines $head -n 2 *.pdb shows the first 2 lines $history | tail -n 15 shows the 15 most recent items in your command history $tail -n +2 Thalas*.txt shows from the second line to the end $head -n -1 Thalas*.txt shows from the second line to the 10 th line Cut cd pcfb/examples/ $cut -f 1,3 Thal*.txt returns columns 1 and 3 delimited by tabs $cut -f 1-3 Thal*.txt returns columns 1 to 3 delimited by tabs $cut -c 16-20,30 Thal*.txt returns characters 16 to 20 and 30 from each line $grep ">" FPexamples.fta | cut -c 2-11 prints out the gene names $head ctd.txt | cut -f 5,7 -d "," returns columns 5 and 7. These are delimited by , in the original file and in the output. Be careful when space is used between columns because there are sometimes two spaces instead of one. Sort $grep ">" FPexcerpt.fta | sort lines are sorted by alphabetical order Options -n sorts by numerical value rather than alphabetically -f Make all lines uppercase before sorting -r sorts in reverse order -k 3 sorts lines based on column 3 , with columns delimited by space or tab $head Thal.txt | sort -k 2 -t "," uses commas for delimiters -u returns a unique representative of repeated items The ASCII list Uniq Removes identical lines that are in immediate succession and keeps a single line. Options -c counts the number of occurrence of each unique line and write it before each unique line $cut -c 12-21 ctd.txt | uniq -c -f 4 ignores the first 4 fields (columns delimited by any number of spaces) in determining uniqueness -i ignore case when determining uniqueness .
Recommended publications
  • DC Console Using DC Console Application Design Software
    DC Console Using DC Console Application Design Software DC Console is easy-to-use, application design software developed specifically to work in conjunction with AML’s DC Suite. Create. Distribute. Collect. Every LDX10 handheld computer comes with DC Suite, which includes seven (7) pre-developed applications for common data collection tasks. Now LDX10 users can use DC Console to modify these applications, or create their own from scratch. AML 800.648.4452 Made in USA www.amltd.com Introduction This document briefly covers how to use DC Console and the features and settings. Be sure to read this document in its entirety before attempting to use AML’s DC Console with a DC Suite compatible device. What is the difference between an “App” and a “Suite”? “Apps” are single applications running on the device used to collect and store data. In most cases, multiple apps would be utilized to handle various operations. For example, the ‘Item_Quantity’ app is one of the most widely used apps and the most direct means to take a basic inventory count, it produces a data file showing what items are in stock, the relative quantities, and requires minimal input from the mobile worker(s). Other operations will require additional input, for example, if you also need to know the specific location for each item in inventory, the ‘Item_Lot_Quantity’ app would be a better fit. Apps can be used in a variety of ways and provide the LDX10 the flexibility to handle virtually any data collection operation. “Suite” files are simply collections of individual apps. Suite files allow you to easily manage and edit multiple apps from within a single ‘store-house’ file and provide an effortless means for device deployment.
    [Show full text]
  • UNIX Cheat Sheet – Sarah Medland Help on Any Unix Command List a Directory Change to Directory Make a New Directory Remove A
    THE 2013 INTERNATIONAL WORKSHOP ON STATISTICAL METHODOLOGY FOR HUMAN GENOMIC STUDIES UNIX cheat sheet – Sarah Medland Help on any Unix command man {command} Type man ls to read the manual for the ls command. which {command} Find out where a program is installed whatis {command} Give short description of command. List a directory ls {path} ls -l {path} Long listing, with date, size and permisions. ls -R {path} Recursive listing, with all subdirs. Change to directory cd {dirname} There must be a space between. cd ~ Go back to home directory, useful if you're lost. cd .. Go back one directory. Make a new directory mkdir {dirname} Remove a directory/file rmdir {dirname} Only works if {dirname} is empty. rm {filespec} ? and * wildcards work like DOS should. "?" is any character; "*" is any string of characters. Print working directory pwd Show where you are as full path. Copy a file or directory cp {file1} {file2} cp -r {dir1} {dir2} Recursive, copy directory and all subdirs. cat {newfile} >> {oldfile} Append newfile to end of oldfile. Move (or rename) a file mv {oldfile} {newfile} Moving a file and renaming it are the same thing. View a text file more {filename} View file one screen at a time. less {filename} Like more , with extra features. cat {filename} View file, but it scrolls. page {filename} Very handy with ncftp . nano {filename} Use text editor. head {filename} show first 10 lines tail {filename} show last 10 lines Compare two files diff {file1} {file2} Show the differences. sdiff {file1} {file2} Show files side by side. Other text commands grep '{pattern}' {file} Find regular expression in file.
    [Show full text]
  • Introduction to Linux – Part 1
    Introduction to Linux – Part 1 Brett Milash and Wim Cardoen Center for High Performance Computing May 22, 2018 ssh Login or Interactive Node kingspeak.chpc.utah.edu Batch queue system … kp001 kp002 …. kpxxx FastX ● https://www.chpc.utah.edu/documentation/software/fastx2.php ● Remote graphical sessions in much more efficient and effective way than simple X forwarding ● Persistence - can be disconnected from without closing the session, allowing users to resume their sessions from other devices. ● Licensed by CHPC ● Desktop clients exist for windows, mac, and linux ● Web based client option ● Server installed on all CHPC interactive nodes and the frisco nodes. Windows – alternatives to FastX ● Need ssh client - PuTTY ● http://www.chiark.greenend.org.uk/~sgtatham/putty/download.html - XShell ● http://www.netsarang.com/download/down_xsh.html ● For X applications also need X-forwarding tool - Xming (use Mesa version as needed for some apps) ● http://www.straightrunning.com/XmingNotes/ - Make sure X forwarding enabled in your ssh client Linux or Mac Desktop ● Just need to open up a terminal or console ● When running applications with graphical interfaces, use ssh –Y or ssh –X Getting Started - Login ● Download and install FastX if you like (required on windows unless you already have PuTTY or Xshell installed) ● If you have a CHPC account: - ssh [email protected] ● If not get a username and password: - ssh [email protected] Shell Basics q A Shell is a program that is the interface between you and the operating system
    [Show full text]
  • The Ifplatform Package
    The ifplatform package Original code by Johannes Große Package by Will Robertson http://github.com/wspr/ifplatform v0.4a∗ 2017/10/13 1 Main features and usage This package provides the three following conditionals to test which operating system is being used to run TEX: \ifwindows \iflinux \ifmacosx \ifcygwin If you only wish to detect \ifwindows, then it does not matter how you load this package. Note then that use of (Linux or Mac OS X or Cygwin) can then be detected with \ifwindows\else. If you also wish to determine the difference between which Unix-variant you are using (i.e., also detect \iflinux, \ifmacosx, and \ifcygwin) then shell escape must be enabled. This is achieved by using the -shell-escape command line option when executing LATEX. If shell escape is not enabled, \iflinux, \ifmacosx, and \ifcygwin will all return false. A warning will be printed in the console output to remind you in this case. ∗Thanks to Ken Brown, Joseph Wright, Zebb Prime, and others for testing this package. 1 2 Auxiliary features \ifshellescape is provided as a conditional to test whether shell escape is active or not. (Note: new versions of pdfTEX allow you to query shell escape with \ifnum\pdfshellescape>0 , and the pdftexcmds package provides the wrapper \pdf@shellescape which works with X TE EX, pdfTEX, and LuaTEX.) Also, the \platformname command is defined to expand to a macro that represents the operating system. Default definitions are (respectively): \windowsname ! ‘Windows’ \notwindowsname ! ‘*NIX’ (when shell escape is disabled) \linuxname ! ‘Linux’ \macosxname ! ‘Mac OS X’ \cygwinname ! ‘Cygwin’ \unknownplatform ! whatever is returned by uname E.g., if \ifwindows is true then \platformname expands to \windowsname, which expands to ‘Windows’.
    [Show full text]
  • HEP Computing Part I Intro to UNIX/LINUX Adrian Bevan
    HEP Computing Part I Intro to UNIX/LINUX Adrian Bevan Lectures 1,2,3 [email protected] 1 Lecture 1 • Files and directories. • Introduce a number of simple UNIX commands for manipulation of files and directories. • communicating with remote machines [email protected] 2 What is LINUX • LINUX is the operating system (OS) kernel. • Sitting on top of the LINUX OS are a lot of utilities that help you do stuff. • You get a ‘LINUX distribution’ installed on your desktop/laptop. This is a sloppy way of saying you get the OS bundled with lots of useful utilities/applications. • Use LINUX to mean anything from the OS to the distribution we are using. • UNIX is an operating system that is very similar to LINUX (same command names, sometimes slightly different functionalities of commands etc). – There are usually enough subtle differences between LINUX and UNIX versions to keep you on your toes (e.g. Solaris and LINUX) when running applications on multiple platforms …be mindful of this if you use other UNIX flavours. – Mac OS X is based on a UNIX distribution. [email protected] 3 Accessing a machine • You need a user account you should all have one by now • can then log in at the terminal (i.e. sit in front of a machine and type in your user name and password to log in to it). • you can also log in remotely to a machine somewhere else RAL SLAC CERN London FNAL in2p3 [email protected] 4 The command line • A user interfaces with Linux by typing commands into a shell.
    [Show full text]
  • Modern Programming Languages CS508 Virtual University of Pakistan
    Modern Programming Languages (CS508) VU Modern Programming Languages CS508 Virtual University of Pakistan Leaders in Education Technology 1 © Copyright Virtual University of Pakistan Modern Programming Languages (CS508) VU TABLE of CONTENTS Course Objectives...........................................................................................................................4 Introduction and Historical Background (Lecture 1-8)..............................................................5 Language Evaluation Criterion.....................................................................................................6 Language Evaluation Criterion...................................................................................................15 An Introduction to SNOBOL (Lecture 9-12).............................................................................32 Ada Programming Language: An Introduction (Lecture 13-17).............................................45 LISP Programming Language: An Introduction (Lecture 18-21)...........................................63 PROLOG - Programming in Logic (Lecture 22-26) .................................................................77 Java Programming Language (Lecture 27-30)..........................................................................92 C# Programming Language (Lecture 31-34) ...........................................................................111 PHP – Personal Home Page PHP: Hypertext Preprocessor (Lecture 35-37)........................129 Modern Programming Languages-JavaScript
    [Show full text]
  • BIMM 143 Introduction to UNIX
    BIMM 143 Introduction to UNIX Barry Grant http://thegrantlab.org/bimm143 Do it Yourself! Lets get started… Mac Terminal PC Git Bash SideNote: Terminal vs Shell • Shell: A command-line interface that allows a user to Setting Upinteract with the operating system by typing commands. • Terminal [emulator]: A graphical interface to the shell (i.e. • Mac users: openthe a window Terminal you get when you launch Git Bash/iTerm/etc.). • Windows users: install MobaXterm and then open a terminal Shell prompt Introduction To Barry Grant Introduction To Shell Barry Grant Do it Yourself! Print Working Directory: a.k.a. where the hell am I? This is a comment line pwd This is our first UNIX command :-) Don’t type the “>” bit it is the “shell prompt”! List out the files and directories where you are ls Q. What do you see after each command? Q. Does it make sense if you compare to your Mac: Finder or Windows: File Explorer? On Mac only :( open . Note the [SPACE] is important Download any file to your current directory/folder curl -O https://bioboot.github.io/bggn213_S18/class-material/bggn213_01_unix.zip curl -O https://bioboot.github.io/bggn213_S18/class-material/bggn213_01_unix.zip ls unzip bggn213_01_unix.zip Q. Does what you see at each step make sense if you compare to your Mac: Finder or Windows: File Explorer? Download any file to your current directory/folder curl -O https://bioboot.github.io/bggn213_S18/class-material/bggn213_01_unix.zip List out the files and directories where you are (NB: Use TAB for auto-complete) ls bggn213_01_unix.zip Un-zip your downloaded file unzip bggn213_01_unix.zip curlChange -O https://bioboot.github.io/bggn213_S18/class-material/bggn213_01_unix.zip directory (i.e.
    [Show full text]
  • GNU Grep: Print Lines That Match Patterns Version 3.7, 8 August 2021
    GNU Grep: Print lines that match patterns version 3.7, 8 August 2021 Alain Magloire et al. This manual is for grep, a pattern matching engine. Copyright c 1999{2002, 2005, 2008{2021 Free Software Foundation, Inc. Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.3 or any later version published by the Free Software Foundation; with no Invariant Sections, with no Front-Cover Texts, and with no Back-Cover Texts. A copy of the license is included in the section entitled \GNU Free Documentation License". i Table of Contents 1 Introduction ::::::::::::::::::::::::::::::::::::: 1 2 Invoking grep :::::::::::::::::::::::::::::::::::: 2 2.1 Command-line Options ::::::::::::::::::::::::::::::::::::::::: 2 2.1.1 Generic Program Information :::::::::::::::::::::::::::::: 2 2.1.2 Matching Control :::::::::::::::::::::::::::::::::::::::::: 2 2.1.3 General Output Control ::::::::::::::::::::::::::::::::::: 3 2.1.4 Output Line Prefix Control :::::::::::::::::::::::::::::::: 5 2.1.5 Context Line Control :::::::::::::::::::::::::::::::::::::: 6 2.1.6 File and Directory Selection:::::::::::::::::::::::::::::::: 7 2.1.7 Other Options ::::::::::::::::::::::::::::::::::::::::::::: 9 2.2 Environment Variables:::::::::::::::::::::::::::::::::::::::::: 9 2.3 Exit Status :::::::::::::::::::::::::::::::::::::::::::::::::::: 12 2.4 grep Programs :::::::::::::::::::::::::::::::::::::::::::::::: 13 3 Regular Expressions ::::::::::::::::::::::::::: 14 3.1 Fundamental Structure ::::::::::::::::::::::::::::::::::::::::
    [Show full text]
  • 101 Useful Linux Commands - Haydenjames.Io
    101 Useful Linux Commands - haydenjames.io Some of these commands require elevated permissions (sudo) to run. Enjoy! 1. Execute the previous command used: !! 2. Execute a previous command starting with a specific letter. Example: !s 3. Short way to copy or backup a file before you edit it. For example, copy nginx.conf cp nginx.conf{,.bak} 4. Toggle between current directory and last directory cd - 5. Move to parent (higher level) directory. Note the space! cd .. 6. Go to home directory cd ~ 7. Go to home directory cd $HOME 8. Go to home directory (when used alone) cd 9. Set permissions to 755. Corresponds to these permissions: (-rwx-r-x-r-x), arranged in this sequence: (owner-group-other) chmod 755 <filename> 10. Add execute permission to all users. chmod a+x <filename> 11. Changes ownership of a file or directory to . chown <username> 12. Make a backup copy of a file (named file.backup) cp <file> <file>.backup 13. Copy file1, use it to create file2 cp <file1> <file2> 14. Copy directory1 and all its contents (recursively) into directory2 cp -r <directory1> <directory2>/ 15. Display date date 16. Zero the sdb drive. You may want to use GParted to format the drive afterward. You need elevated permissions to run this (sudo). dd if=/dev/zero of=/dev/sdb 17. Display disk space usage df -h 18. Take detailed messages from OS and input to text file dmesg>dmesg.txt 19. Display a LOT of system information. I usually pipe output to less. You need elevated permissions to run this (sudo).
    [Show full text]
  • Useful Tai Ls Dino
    SCIENCE & NATURE Useful Tails Materials Pictures of a possum, horse, lizard, rattlesnake, peacock, fish, bird, and beaver What to do 1. Display the animal pictures so the children can see them. 2. Say the following sentences. Ask the children to guess the animal by the usefulness of its tail. I use my tail for hanging upside down. (possum) I use my tail as a fly swatter. (horse) When my tail breaks off, I grow a new one. (lizard) I shake my noisy tail when I am about to strike. (rattlesnake) My tail opens like a beautiful fan. (peacock) I use my tail as a propeller. I cannot swim without it. (fish) I can’t fly without my tail. (bird) I use my powerful tail for building. (beaver) More to do Ask the children if they can name other animals that have tails. Ask them how these animals’Downloaded tails might by [email protected] useful. from Games: Cut out the tailsProFilePlanner.com of each of the animals. Encourage the children to pin the tails on the pictures (like “Pin the Tail on the Donkey”). Dotti Enderle, Richmond, TX Dino Dig Materials Plastic or rubber dinosaurs or bones Sand Wide-tip, medium-sized paintbrushes Plastic sand shovels Small plastic buckets Clipboards Paper Pencil or pens 508 The GIANT Encyclopedia of Preschool Activities for Four-Year-Olds Downloaded by [email protected] from ProFilePlanner.com SCIENCE & NATURE What to do 1. Beforehand, hide plastic or rubber dinosaurs or bones in the sand. 2. Give each child a paintbrush, shovel, and bucket. 3.
    [Show full text]
  • “Linux at the Command Line” Don Johnson of BU IS&T  We’Ll Start with a Sign in Sheet
    “Linux at the Command Line” Don Johnson of BU IS&T We’ll start with a sign in sheet. We’ll end with a class evaluation. We’ll cover as much as we can in the time allowed; if we don’t cover everything, you’ll pick it up as you continue working with Linux. This is a hands-on, lab class; ask questions at any time. Commands for you to type are in BOLD The Most Common O/S Used By BU Researchers When Working on a Server or Computer Cluster Linux is a Unix clone begun in 1991 and written from scratch by Linus Torvalds with assistance from a loosely-knit team of hackers across the Net. 64% of the world’s servers run some variant of Unix or Linux. The Android phone and the Kindle run Linux. a set of small Linux is an O/S core programs written by written by Linus Richard Stallman and Torvalds and others others. They are the AND GNU utilities. http://www.gnu.org/ Network: ssh, scp Shells: BASH, TCSH, clear, history, chsh, echo, set, setenv, xargs System Information: w, whoami, man, info, which, free, echo, date, cal, df, free Command Information: man, info Symbols: |, >, >>, <, ;, ~, ., .. Filters: grep, egrep, more, less, head, tail Hotkeys: <ctrl><c>, <ctrl><d> File System: ls, mkdir, cd, pwd, mv, touch, file, find, diff, cmp, du, chmod, find File Editors: gedit, nedit You need a “xterm” emulation – software that emulates an “X” terminal and that connects using the “SSH” Secure Shell protocol. ◦ Windows Use StarNet “X-Win32:” http://www.bu.edu/tech/support/desktop/ distribution/xwindows/xwin32/ ◦ Mac OS X “Terminal” is already installed Why? Darwin, the system on which Apple's Mac OS X is built, is a derivative of 4.4BSD-Lite2 and FreeBSD.
    [Show full text]
  • Text Editing in UNIX: an Introduction to Vi and Editing
    Text Editing in UNIX A short introduction to vi, pico, and gedit Copyright 20062009 Stewart Weiss About UNIX editors There are two types of text editors in UNIX: those that run in terminal windows, called text mode editors, and those that are graphical, with menus and mouse pointers. The latter require a windowing system, usually X Windows, to run. If you are remotely logged into UNIX, say through SSH, then you should use a text mode editor. It is possible to use a graphical editor, but it will be much slower to use. I will explain more about that later. 2 CSci 132 Practical UNIX with Perl Text mode editors The three text mode editors of choice in UNIX are vi, emacs, and pico (really nano, to be explained later.) vi is the original editor; it is very fast, easy to use, and available on virtually every UNIX system. The vi commands are the same as those of the sed filter as well as several other common UNIX tools. emacs is a very powerful editor, but it takes more effort to learn how to use it. pico is the easiest editor to learn, and the least powerful. pico was part of the Pine email client; nano is a clone of pico. 3 CSci 132 Practical UNIX with Perl What these slides contain These slides concentrate on vi because it is very fast and always available. Although the set of commands is very cryptic, by learning a small subset of the commands, you can edit text very quickly. What follows is an outline of the basic concepts that define vi.
    [Show full text]