Introduction to High Performance Computing at Case Western Reserve University Navigating Linux KSL Data Center
Total Page:16
File Type:pdf, Size:1020Kb
Introduction to High Performance Computing at Case Western Reserve University Navigating Linux [email protected] KSL Data Center Working in the Linux Shell Following curriculum from Software Carpentry • Navigating Files and Directories – http://swcarpentry.github.io/shell-novice/02-filedir/index.html Learning Outcomes -- Navigating • Explain the similarities and differences between a file and a directory. • Translate an absolute path into a relative path and vice versa. • Construct absolute and relative paths that identify specific files and directories. • Demonstrate the use of tab completion, and explain its advantages. Working in the Linux Shell Following curriculum from Software Carpentry • Working with Files and Directories – http://swcarpentry.github.io/shell-novice/03-create/index.html Learning Outcomes -- Working • Create a directory hierarchy that matches a given diagram. • Create files in that hierarchy using an editor or by copying and renaming existing files. • Delete, copy and move specified files and/or directories. General Linux File System / “root” Directory purpose Example/Key contents /home holds personal directories .bashrc .ssh /bin system binary files which ls cd bash /dev devices of the server /dev/sda /dev/hda /lib system and software libraries /mnt mounting external (network) file systems /mnt/cdrom /mnt/rstor /root “home” for ‘root’ user account /sys system files /sys/fs /sys/kernel /usr user accessible software /usr/bin /usr/lib /usr/src /var operating system will write data /var/log /var/www HPC Environment File Structure / [root] /home /scratch /mnt /usr /usr/bin /home/<caseid> /scratch/pbsjobs /mnt/pan /usr/lib xxx accounts for scheduled jobs additional high- system performance storage executables and libraries /scratch/users /mnt/projects general purpose research storage /usr/local temp storage installed software Setup for this working session Following curriculum from Software Carpentry • Open a terminal on your computer, or, on Rider (or elsewhere) • Obtain a sample data set: • http://swcarpentry.github.io/shell-novice/data/data-shell.zip • On Rider: cp –r /home/mrd20/hpcdemo/workinglinux/data-shell . • elsewhere: scp <caseid>@rider.case.edu: \ /home/mrd20/hpcdemo/workinglinux/data-shell.tgz . • Un”tar” the data set: tar –xvzf data-shell.tgz • Verify contents: ls data-shell Exploring the data-shell hierarchy • Use the ‘list’ and ‘change directory’ commands • Think about use of absolute and relative paths • Sample exercises: • Use ‘pwd’ to determine the current working directory • Are you in ‘data-shell’? If not, how would you navigate into that directory? • Use the ‘–F’ option with ‘ls’. How does that change the output? • Widen and narrow the shell window. How does this affect the results of ‘ls’? • Examine ‘man ls’ – ‘man’ is the manual page viewer • move downward using spacebar [page down], or the down arrow [line down] • move upward using ‘b’ [page up], or the up arrow [line up] Navigation Review • File system is a hierarchy • Our personal files (not always data) in the /____/<user> location • “System” files in numerous other branches of the hierarchy • Keeping track of the shell “location” • The command prompt on HPC provides structural help: [mrd20@hpc3 nav-linux] • The ‘pwd’ command provides the ________ path for current working directory • Moving within the hierarchy with ‘cd’ • Special character to move upward one directory: “..” • Be specific with an absolute path: “cd data/elements” • Listing directories has many built-in capabilities for output format Files & Directories How to see or find files and directories • The list command — ls • ls [OPTION]... [FILE]… • let’s ask for help in an example, using ‘- - help’ [demo] • options • - l : long-list, provides details (shows directory contents) • - d : only list a directory, don’t show contents • - Sh : size + ‘human’-readable output (typically for storage utilization) • - t : time-order • - r : reverse the ordering Combining flags • Exploring ‘ls’ flags • We know that ‘-l’ provides a “____” listing • Compare ‘ls –l’ with ‘ls –lh’: • What do you find? • Combine ‘-S’, as well as ‘-G’ • Use the ‘man ls’ if needed to further understand the results • Is there a way to ‘reverse’ the output listing? Relative Path Resolution If pwd displays /Users/thing, what will ls -F ../backup display? 1. ../backup: No such file or directory 2. 2012-12-01 2013-01-08 2013-01-27 3. 2012-12-01/ 2013-01-08/ 2013-01-27/ 4. original/ pnas_final/ pnas_sub/ Relative Path Resolution If pwd displays /Users/thing, what will ls -F ../backup display? 1. ../backup: No such file or directory 2. 2012-12-01 2013-01-08 2013-01-27 3. 2012-12-01/ 2013-01-08/ 2013-01-27/ 4. original/ pnas_final/ pnas_sub/ Relative Path Resolution If pwd displays /Users/thing, what will ls -F ../backup display? .. -- parent directory, Users 1. ../backup: No such file or directory 2. 2012-12-01 2013-01-08 2013-01-27 3. 2012-12-01/ 2013-01-08/ 2013-01-27/ 4. original/ pnas_final/ pnas_sub/ Relative Path Resolution If pwd displays /Users/backup, what command will display the following? pnas_sub/ pnas_final/ original/ 1. ls pwd 2. ls –r –F 3. ls –r –F /Users/backup Working with files and directories • http://swcarpentry.github.io/shell-novice/03-create/index.html • Creating directories: mkdir • Creating & editing files: text editors • nano – very simple and accessible • vi & emacs – more capable, longer to learn and feel comfortable • Creating files: touch <filename> • Removing files: rm [-r] • Removing directories: rmdir <dir-name> Files & Directories Making a new directory • The ‘make directory’ command — mkdir • mkdir [OPTION]... DIRECTORY... • let’s ask for help in an example, using ‘- - help’ [demo] • options • - m: manipulate the ‘mode’ (permissions) • - p: make parent directories is they do not exist (e.g. mkdir –p tutorial/downloads) • Exercise: • make a directory in data-shell named ‘thesis’ • change directory into ‘thesis’ File editing • Files in linux are ‘text’ files – even if the text is not human readable • Creating and editing then involves ‘text editors’ • Getting familiar with Nano • start nano in ‘data-shell/thesis’ • enter some text (the exercise provides a suggestion) • Great, now – how do we: • Save the text? • Close the file? Linux foundations • Control, Ctrl, or ^ Key • There are many prepresentations for ‘control’. It invokes special functions when used together with other keys. It’s important to identify the control key for the keyboard of the system you are using: • Control-X • Control+X • Ctrl-X • Ctrl+X • ^X • C-x Another way to create files • Let’s try the following command: • $ touch my_file.txt • What did the touch command do? When you look at your current directory using the GUI file explorer, does the file show up? • Use ls -l to inspect the files. How large is my_file.txt? • When might you want to create a file this way? Moving files & directories • The ‘move’ (rename) command — mv • mv [OPTION]... SOURCE DEST • Rename SOURCE to DEST, or move SOURCE(s) to DIRECTORY. • Exercise: using ./thesis/draft.txt as the source • rename ‘draft.txt’ to ‘quotes.txt’ • Exercise: using ./thesis/quotes.txt as the source • move ‘quotes.txt’ to the data-shell directory Moving to the current folder – ‘.’ • After running the following commands, Jamie realizes that she put the files sucrose.dat and maltose.dat into the wrong folder: • $ ls -F analyzed/ raw/ • $ ls -F analyzed fructose.dat glucose.dat maltose.dat sucrose.dat • $ cd raw/ Fill in the blanks to move these files to the current folder (i.e., the one she is currently in): • $ mv ___/sucrose.dat ___/maltose.dat ___ Moving files & directories • The ‘copy’ command — cp – copy files and directories • cp [OPTION]... SOURCE DEST • Copy SOURCE to DEST, or multiple SOURCE(s) to DIRECTORY. • options • -p: preserve mode, ownership, timestamps (--preserve is more flexible) • -r: copy directories recursively • -u: update, copy only when source is ‘newer’ than destination • -s: make symbolic links instead of copying Copy and verify • Like many Unix commands, ls can be given multiple paths at once: • $ cp quotes.txt thesis/quotations.txt • $ ls quotes.txt thesis/quotations.txt quotes.txt thesis/quotations.txt • We can also copy a directory and all its contents by using the recursive option -r, e.g. to back up a directory: $ cp -r thesis thesis_backup • We can check the result by listing the contents of both the thesis and thesis_backup directory: $ ls thesis thesis_backup • $ ls thesis thesis_backup Renaming files • If a file is named ‘statstics.txt’, rather than ‘statistics.txt’,which of the following commands could you use to do so? • cp statstics.txt statistics.txt • mv statstics.txt statistics.txt • mv statstics.txt . • cp statstics.txt . Additional consideration: How similar are cp and mv? Removing files • The ‘remove’ command -- rm – remove files or directories • rm [option] FILE • rm removes each specified file. By default, it does not remove directories. • Options • -i: prompt before every removal • -r: remove contents and directories that are empty • -d: remove empty directories Removing files • From ‘data-shell’, let’s clean up by removing ‘quotes.txt’ • $ rm quotes.txt • And confirm, using: $ls quotes.txt • Output: ls: cannot access quotes.txt: No such file or directory • How can we ensure that we *really* want to delete, expecially from a group of files? • $rm –i quotes.txt rm: remove regular empty file ‘quotes.txt’? y [enter] Recovering files • Linux assumes you mean to do each command. Removing files is generally not reversible, however, • HPC does protect the /home/<caseid> areas with .snapshot • $ cd • [mrd20@hpc3 ~]$ ls .snapshot • 2019.09.09.23.00.01.home 2019.09.11.23.00.01.home 2019.09.13.23.00.01.home 2019.09.15.23.00.01.home 2019.09.10.23.00.01.home 2019.09.12.23.00.01.home 2019.09.14.23.00.01.home • Snapshots are file structures, just like we’ve seen. • Recovery is a matter of using ‘cp’ and/or ‘mv’ (is there a risk with ‘mv’?) Search using Wildcards - I • * is a wildcard, which matches zero or more characters.