https://bit.ly/3bUvR1s presentation slides Introduction to Linux Dr. Yongjun Choi 03/16/2021 Geting help • Wiki: https://wiki.hpcc.msu.edu/x/4ACE • Submit a ticket: https://contact.icer.msu.edu/contact External resources to learn Linux http://ss64.com/bash/ "An A-Z Index of the Bash command line for Linux." with links to 'man' manual page for each. It includes many commands that you will never use, or are specific to certain versions of linux https://fosswire.com/post/2007/08/unixlinux-command-cheat-sheet/ useful cheatsheet organized by topic https://cvw.cac.cornell.edu/linux/ Full intro to Linux http://guide.bash.academy step by step intro with lots of explanatory text MSU HPC runs Linux • We will use the MSU High Performance Computing Cluster for this course. • consistent experience (Windows, Mac laptop diversity) • It’s a system we know • Highly availability • Course is pre-requisite for other HPCC courses Get started • 1. Install/identify laptop software to connect • 2. Connect to HPC with secure shell • 3. Does your account work? Troubleshoot if necessary • 4. Test basic commands • Step 1: Install Software (if you haven’t) • Windows: install ‘MobaXterm’ free Unix emulation app • When installed, start program and then start New Session • Mac OS X: included a unix terminal, but install Xquartz for GUI • Open the application /Applications/Utilities/Terminal.app Get connected • From the terminal/MobaXterm • Type: ssh [email protected] • You can also use: ssh -Y [email protected] • -Y (or -X) is an option for GUI • Enter password - it is hidden and doesn’t look like you are typing, but you are. Linux commands • What is command? • $ <command> <arguments(s)> <optional file/dir name> • $ ls -l -a • ls: command (list files), -l, -a: arguments (l: long format, a: all files including hidden files) • $ wc - l myfile.txt • wc: command (word count), -l: argument (line count), myfile.txt: filename Basic commands • cd: change directory • ls: list • pwd: current location • mkdir: make a directory • cp: copy files/dirs • rm: remove/rename files/dirs ls • ls: list files and directories • some options for ls command • -a list all files and directories including hidden contents • -h: print sizes in human readable format (e.g. 1k, 2.5M, 3.1G) • -l: list with a long listing format • -t: sort my modification time Exploring files and folders with ls • Step one, explore your home directory with a command ‘ls’ • Note when giving command examples, $ at the begging is the prompt (you SHOULD NOT type it). Try these commands and check the difference. • $ ls show files • $ ls -l shows files in long view • $ ls -al shows all files • $ ls /mnt/home lists folders of users • $ ls /mnt/home/your_net_id • Now try someone else’s folder - what happens? • $ ls /mnt/home/choiyj cd • cd directory_name: change to named directory • cd: change to home dir • cd ~: change to home dir • cd ..: change to one level upper dir • cd -: change to the previous dir cp, mkdir mv, rm, rmdir • cp <from> <to>: copy files • cp -r <from> <to>: copy recursively: files and directories • mkdir: make a named directory • mv <from> <to>: move/rename files/folders • rm filename: remove a named file • rm -r dirname: remove a nanmed dir (incluidng all sub directories and files) • EX: Can you create a folder ‘test folder’ with mkdir? There IS a space between ‘test’ and ‘folder’. • Question? Use man. eg: man cp manipulation exercise • Create linux_workshop dir on your home. • Copy a folder and contents for this class from • /mnt/research/common-data/workshops/intro2Linux_2021_03_16 • to linux_workshop dir on your home • Go to linux_workshop • Find a hidden directory and rename it to not_hidden • Check the contents of not_hidden • Create a new directory called new_dir • Copy the file youfoundit.txt into new_dir • remove garbage dir Files and folders • Linux has a single directory ‘tree’, separated by slash, th top is the ‘root’. • All additional disk are connected on /mnt ‘mounted’. • No concept of driver letters • From your home, try $ tree -L 1, and $ tree -L 2 Exploring files and folders • On our system • /mnt/home/ are all user home directories folders • /mnt/scratch/ are working files • /bin and /usr/bin and /opt/software software • User pwd to determin your current point in the tree • use ls -l to see all files in your folder; note it starts with two entries single dot (.) short cut representing current folder, and double dot (..) short cut representing parent folder • cd: change directory. cd .. —> go up one level • ~ is shortcut for your home directory: Try $ cd ~ • Files are specified by the ‘path’ through directory tree • Path can be full (/mnt/home/choiyj/.bashrc) starting with / • or relative, from the current location ~/.bashrc File permissions • $ ls /mnt/home/choiyj • $ ls /mnt/scratch/choiyj • Linux has a method to keep files private and save. • permission for user, user groups, and all others File permissions • User, Group, other have read, write, or execute permissions. • Permissions are set with ‘chmod’ command, and ownwerhsip set with ‘chown’ • You can only change these for files you are the owner of. • Dont grant other/owrld permission on yur home folder. Keep it private! chmod • excutable: 1, write: 2, read: 4, • chmod 777 file1.txt: open ‘file1.txt’ to the whole world. Never do that! • you can use • chmod g+w file1.txt: opn to your group for writing access. • https://en.wikipedia.org/wiki/Chmod redirection • >: write to, • >> : append to • 1: stdout • 2: stderr; • >&: redirection operation • eg: prog > outfile 2>&1 : send stdout and stderr to ‘outfile’ Wildcards • *: anything or nothing • ?: single character • [ ]: any character or range of characters • [! ]: inverse the match of [ ] • $ ls *.txt # list all txt files • $ ls *-?.txt # list all files with ‘-‘ and with one cha in front of ‘.txt’ • $ ls [0-9]*.txt # list all files start with a number. • $ ls [A-Z]*.txt # list all files start with a capital letter? • Try ls [[:lower:]].txt; ls [[:upper:]].txt; ls [[:lower:][:upper:]].txt • $ ls [!a-Z]*.txt # list txt files that don’t begin with any letter. • https://wiki.hpcc.msu.edu/display/ITH/Regular+Expressions Learning commands with manual pages • Linux includes a built in manual for nearly all commands. Type ‘man’ followed by the commands. e.g. • $man man • To navigate the man pages use the arrow keys to scroll up and down or use the enter key to advance al ine, space bar to advance a page, letter u to go back a page. Use the q key to quit out of the display. • The manual pages often include these sections: • Name: a one line desctiption of what it does • Synopsis: basic syntax for the command line. • Description: describes the program’s functionalities. • Options: lists commnand line options available for this program. • Example: examples of some of the options available. • See Also: list of related commands. • Note that options can be with single dash ‘-‘ or double dash ‘—’ Learning commands with manual pages • review tha man pages of common file/directory commnds • ls, mkdir, cp, pwd, cat • How can you list all files in a folder, including hidden files that start with a dot (.)? • Which command would create a folder ‘class’ in your home directory? Can you do that? what are the options for this command? • What does pwd do? Shell environment variables • Shell maintains and you can set ‘variables’ that the shell uses for configuration and in your script. • Variables start with $, and can be seen with echo $VARNAME • Explore common variables with the echo command and list what they are • $HOME, $USER, $SHELL, $PATH • Combine variables in an echo command to make a greeting. • Set a variable of your own, then use in the sentence. • GREETING=Hello; echo $GREETING, $USER - welcom to $HOME • https://wiki.hpcc.msu.edu/display/ITH/1.+Variables+-+Part+I • https://wiki.hpcc.msu.edu/display/ITH/1.+Variables+-+Part+II $PATH • BASH looks for programs in each folder listed in $PATH • When you type a program name, how does the shell find it? Let’s try the wich command, it shows the location of programs. • which date • which less • which python • each of these folders in the path variable. Command input/output • Most commands take text as input and output results as text • $ ls -l output text to the screen • $ ls -l > filelist.txt redirect text into a new file ‘filelist.txt’ • $ ls -l >> filelist.txt redirect text into ‘filelist.txt’ (added at the end of the file) • $ ls -l | grep txt pipe text into another command • $ wc -l < filelist.txt redirect file as input into command • $ cat filelist.txt |wc -l outputfile, pipe into command • Some commands for data • wc: count words, lines or characters • who | wc -l # number of users logged in • grep: find patterns in files or data, returns just matching lines • who |grep $USER • sort: given a list of items, sort in various ways • who | sort • head: list only top n lines of file • who > who.txt; head who.txt • tail: list only last n lines of file Some commands for data • zip: creat zip of multiple files • unzip: unzip • tar: create (tar -c) or extract (tar -x) ‘tape archive’ file. • Exercise : • Using the man pages, find out what the default number of lines that head and tail will display, and how to limit those to just one line • Can you tar all files and folders in workshop folder? Question? Use man. Creating/editing text files • You can create/edit files on your local machin and transfer them using ftp app. • Downside: very slow process. • need to run command ‘dos2unix’ if the file is created under Windows system. • Much faster in the long term to learn how to edit files with Linux editors.