Linux for scientists or: What can I do at the black screen? Lennart C. Karssen PolyΩmica, The Netherlands [email protected] June 2016* * Git information: Hash: 5ddb1d6 Date: 2016-06-10 0 01 1 010 010 0 0 01 10 10 10 1 0 010 10 1 1 0 1 1 1 0 0 1 0 01 0 1 0 0 1 0 10 0 1 0 1 1 1 0 1 1 1 01 0 0 1 0 0 0 0 1 1 0 1 1 1 1 1 1 1 1 0 0 0 1 1 0 0 1 0 0 1 1 0 0 0 1 1 0 1 1 1 1 1 0 0 0 1 1 0 0 1 1 0 0 1 1 0 1 1 1 0 1 1 1 0 1 0 0 0 0 1 1 1 0 1 1 0 0 1 0 0 0 1 1 0 1 0 0 1 1 0 1 1 0 1 1 0 1 1 1 0 1 1 0 0 1 0 1 1 0 0 1 1 0 0 0 0 0 1 1 0 1 1 1 1 1 0 0 1 1 1 1 1 0 1 1 1 1 0 1 0 0 0 1 0 0 1 1 1 1 0 0 1 0 0 0 1 0 1 1 0 1 0 0 0 1 0 1 0 0 0 1 1 0 0 1 0 0 1 0 1 1 1 0 1 0 0 0 1 0 1 0 0 1 0 0 1 1 0 0 1 1 1 1 1 0 1 1 1 0 1 1 0 1 1 1 1 1 1 0 0 0 0 0 0 1 0 1 0 0 1 1 0 1 0 0 0 1 0 0 0 0 1 0 1 1 1 0 1 0 0 1 0 0 1 1 0 0 0 0 1 0 1 1 0 1 1 0 1 0 1 0 1 1 1 0 0 0 1 0 1 1 0 0 0 1 0 0 0 0 0 1 1 0 0 0 1 0 0 1 0 0 1 1 1 1 1 0 0 1 0 0 1 1 0 1 0 1 1 0 0 1 1 0 0 0 0 1 1 1 1 0 1 1 1 0 1 1 1 1 0 1 1 0 1 0 1 1 0 0 0 1 0 0 0 0 1 1 0 0 1 1 0 0 0 0 1 1 1 1 1 0 1 0 1 1 1 0 1 0 0 1 0 0 0 0 1 1 0 1 1 1 1 1 0 1 1 1 1 0 1 1 1 1 0 0 0 0 1 0 0 1 1 1 1 1 1 1 0 0 1 0 0 0 1 0 0 1 1 1 0 1 1 1 1 1 1 0 0 0 0 0 0 0 0 1 0 0 1 1 1 1 1 1 0 1 1 0 1 0 1 0 0 0 0 0 1 0 0 0 1 1 1 1 1 0 1 1 1 0 0 0 1 0 0 0 0 1 1 1 1 0 0 1 0 1 0 0 0 1 1 0 0 0 0 1 1 1 0 0 0 0 0 0 0 1 0 1 0 1 1 0 0 0 1 0 1 1 1 1 0 0 1 0 1 0 0 0 1 1 1 1 0 1 1 1 1 1 1 1 0 1 0 0 1 1 1 0 1 1 0 0 1 1 0 1 1 0 1 0 1 1 0 1 1 1 1 0 1 0 1 1 0 0 1 1 1 1 1 1 1 1 0 1 0 1 1 0 1 1 1 1 0 0 0 0 0 0 0 1 1 0 0 1 1 0 1 0 0 1 0 0 1 1 0 1 0 0 0 1 1 1 1 1 1 0 1 1 0 1 1 0 1 1 1 0 0 11 0 1 1 1 1 1 0 0 0 1 1 0 1 1 1 1 1 0 11 0 1 0 0 0 1 1 0 0 0 0 01 0 1 0 0 01 1 1 01 0 1 1 0 0 1 1 1 0 1 1 1 11 0 1 0 0 0 1 1 1 1 0 0 1 0 1 1 0 1 0 0 1 0 0101111 10100001001011 1 1011 1 111011 0 0 0 0101 1 0 0 10 1 00 1 101 1 1 101 0 1 10 1 01 1 1 00 0 00 1 0 0 1 1 1 11 1 0 1 1 010110101010 1 Contents Contents List of Tables x 1 Preface 3 1.1 About this book ........................... 4 1.2 Acknowledgements ......................... 5 2 What is Linux? 7 3 The basics 11 3.1 Logging in and out ......................... 12 3.2 Editors ................................. 13 3.3 The structure of Linux commands ................ 14 3.3.1 Exercises ........................... 17 3.4 Managing your account ...................... 19 3.5 Getting help ............................. 20 3.6 Working with files and directories ................ 21 3.6.1 Directories .......................... 24 3.6.2 Copying, moving, removing ................ 25 3.6.3 Wildcards .......................... 27 3.6.4 Exercises ........................... 28 3.7 Transferring files from one Linux machine to another .... 31 3.8 Pagers, or how to look at the contents of a file ........ 32 3.8.1 Exercises ........................... 33 3.9 Using compressed archives like .zip and tar.gz files ... 34 3.9.1 zip .............................. 35 3.9.2 gzip .............................. 35 3.9.3 tar .............................. 36 3.9.4 Exercises ........................... 36 3.10 File ownership and permissions ................. 37 3.10.1 Ownership .......................... 37 3.10.2 Permissions ......................... 38 3.11 Process management ........................ 39 3.11.1 Exercises ........................... 40 3.12 Miscellaneous commands ..................... 43 3.12.1 wget: downloading files to the server ......... 43 3.12.2 sort .............................. 44 3.12.3 uniq .............................. 46 iv Contents 3.12.4 wc: counting words and lines .............. 47 3.12.5 date .............................. 47 3.12.6 du: disk space usage .................... 48 3.12.7 Differences between files ................. 49 3.13 Input and output redirection ................... 51 3.13.1 Redirecting to and from files ............... 52 3.13.2 Redirecting output of one command to another ... 53 3.14 Aliases and creating your own commands ........... 55 4 Working with text files 61 4.1 Converting between Windows and Linux format ....... 62 4.1.1 Exercises ........................... 63 4.2 grep: finding text .......................... 64 4.2.1 Exercises ........................... 67 4.3 sed, the Stream EDitor ....................... 68 4.3.1 Exercises ........................... 69 4.4 cut: selecting columns ....................... 70 4.5 GAWK: more fun with columns .................. 71 4.5.1 Exercises ........................... 74 4.6 Putting it all together ........................ 76 4.6.1 Exercises ........................... 76 5 Writing Bash scripts 81 5.1 A simple script ............................ 82 5.2 Using variables ............................ 85 5.3 Using shell variables in GAWK ................... 89 5.4 Loops, for and while ....................... 90 5.5 if-clauses and tests ......................... 95 5.6 Dealing with errors in your script ................ 97 6 Working with the SGE queue system 103 6.1 Submitting jobs to the SGE queues ................ 104 6.1.1 Quick and dirty ....................... 105 6.1.2 Using a submission script ................ 105 6.1.3 Refinements to the submission script ......... 106 6.2 Monitoring progress ........................ 107 6.3 Deleting jobs from a queue .................... 108 v Contents 6.4 Getting info on a finished job ................... 109 6.5 Interactive jobs ........................... 111 6.6 Exercises ............................... 111 7 Good scripting practices, structured programming and data man- agement 117 7.1 Code layout .............................. 119 7.1.1 Indentation ......................... 119 7.1.2 Line length .......................... 120 7.1.3 Spaces ............................ 121 7.2 Comments .............................. 123 7.3 Variable names ........................... 124 7.4 File and directory names ..................... 124 7.5 Summary ............................... 125 8 Where to go from here? 129 8.1 More advanced topics ....................... 130 8.2 Further reading ........................... 131 A Answers to the exercises 135 B Reference Card of Basic Linux Commands 157 C List of acronyms 163 Bibliography 168 Index 172 vi 0 01 1 010 010 0 0 01 10 10 10 1 0 010 10 1 1 0 1 1 1 0 0 1 0 01 0 1 0 0 1 0 10 0 1 0 1 1 1 0 1 1 1 01 0 0 1 0 0 0 0 1 1 0 1 1 1 1 1 1 1 1 0 0 0 1 1 0 0 1 0 0 1 1 0 0 0 1 1 0 1 1 1 1 1 0 0 0 1 1 0 0 1 1 0 0 1 1 0 1 1 1 0 1 1 1 0 1 0 0 0 0 1 1 1 0 1 1 0 0 1 0 0 0 1 1 0 1 0 0 1 1 0 1 1 0 1 1 0 1 1 1 0 1 1 0 0 1 0 1 1 0 0 1 1 0 0 0 0 0 1 1 0 1 1 1 1 1 0 0 1 1 1 1 1 0 1 1 1 1 0 1 0 0 0 1 0 0 1 1 1 1 0 0 1 0 0 0 1 0 1 1 0 1 0 0 0 1 0 1 0 0 0 1 1 0 0 1 0 0 1 0 1 1 1 0 1 0 0 0 1 0 1 0 0 1 0 0 1 1 0 0 1 1 1 1 1 0 1 1 1 0 1 1 0 1 1 1 1 1 1 0 0 0 0 0 0 1 0 1 0 0 1 1 0 1 0 0 0 1 0 0 0 0 1 0 1 1 1 0 1 0 0 1 0 0 1 1 0 0 0 0 1 0 1 1 0 1 1 0 1 0 1 0 1 1 1 0 0 0 1 0 1 1 0 0 0 1 0 0 0 0 0 1 1 0 0 0 1 0 0 1 0 0 1 1 1 1 1 0 0 1 0 0 1 1 0 1 0 1 1 0 0 1 1 0 0 0 0 1 1 1 1 0 1 1 1 0 1 1 1 1 0 1 1 0 1 0 1 1 0 0 0 1 0 0 0 0 1 1 0 0 1 1 0 0 0 0 1 1 1 1 1 0 1 0 1 1 1 0 1 0 0 1 0 0 0 0 1 1 0 1 1 1 1 1 0 1 1 1 1 0 1 1 1 1 0 0 0 0 1 0 0 1 1 1 1 1 1 1 0 0 1 0 0 0 1 0 0 1 1 1 0 1 1 1 1 1 1 0 0 0 0 0 0 0 0 1 0 0 1 1 1 1 1 1 0 1 1 0 1 0 1 0 0 0 0 0 1 0 0 0 1 1 1 1 1 0 1 1 1 0 0 0 1 0 0 0 0 1 1 1 1 0 0 1 0 1 0 0 0 1 1 0 0 0 0 1 1 1 0 0 0 0 0 0 0 1 0 1 0 1 1 0 0 0 1 0 1 1 1 1 0 0 1 0 1 0 0 0 1 1 1 1 0 1 1 1 1 1 1 1 0 1 0 0 1 1 1 0 1 1 0 0 1 1 0 1 1 0 1 0 1 1 0 1 1 1 1 0 1 0 1 1 0 0 1 1 1 1 1 1 1 1 0 1 0 1 1 0 1 1 1 1 0 0 0 0 0 0 0 1 1 0 0 1 1 0 1 0 0 1 0 0 1 1 0 1 0 0 0 1 1 1 1 1 1 0 1 1 0 1 1 0 1 1 1 0 0 11 0 1 1 1 1 1 0 0 0 1 1 0 1 1 1 1 1 0 11 0 1 0 0 0 1 1 0 0 0 0 01 0 1 0 0 01 1 1 01 0 1 1 0 0 1 1 1 0 1 1 1 11 0 1 0 0 0 1 1 1 1 0 0 1 0 1 1 0 1 0 0 1 0 0101111 10100001001011 1 1011 1 111011 0 0 0 0101 1 0 0 10 1 00 1 101 1 1 101 0 1 10 1 01 1 1 00 0 00 1 0 0 1 1 1 11 1 0 1 1 010110101010 1 List of Tables List of Tables 3.1 Basic Emacs keyboard shortcuts.