The Research Computing User's Guide

Total Page：16

File Type：pdf, Size：1020Kb

The Research Computing User’s Guide i The Research Computing User’s Guide September 13, 2021 The Research Computing User’s Guide ii COLLABORATORS TITLE : The Research Computing User’s Guide ACTION NAME DATE SIGNATURE WRITTEN BY Jason W. Bacon September 13, 2021 Initial HTcondor Lars Olson September 13, 2021 content The Research Computing User’s Guide iii Contents 1 Goals 2 2 Self-Study Instructions 3 2.1 Unix Self-Study Instructions . .3 2.2 Parallel Computing Self-Study Instructions . .3 I Research Computing4 3 Computational Science 5 3.1 So, What is Computational Science? . .5 3.2 Not Just for Nerds Anymore . .5 3.3 Achieving Our Goals . .5 3.3.1 The Computation Time Line . .5 3.3.2 Development Time . .6 3.3.3 Deployment Time . .6 3.3.4 Learning Time . .6 3.3.5 Run Time . .6 3.4 Common Methods Used in Computational Science . .6 3.4.1 Numerical Analysis . .6 3.4.2 Computational Modeling . .7 3.4.3 Data Mining . .8 3.4.4 Parameter Sweeps . .8 3.4.5 Data Sifting . .9 3.4.6 Monte Carlo . .9 3.4.7 Everything Else . 10 3.5 Self-test . 10 The Research Computing User’s Guide iv 4 Where do I get the Software? 11 4.1 But I Hate Programming... 11 4.2 BuyIt......................................................... 12 4.3 Download It . 12 4.3.1 How to Shoot Yourself in the Foot with Open Source Software . 13 4.3.2 How Not to Shoot Yourself in the Foot with Open Source Software . 14 4.4 The Realities of Software Support . 14 4.5 A Comprehensive List of Research Software . 15 4.6 Write It . 15 4.7 Self-test . 16 5 Running Your Software 18 5.1 Where do I Run It? . 18 5.1.1 Public Resources . 18 College Computer Labs . 18 College Clusters and Grids . 18 XSEDE, Open Science Grid . 18 Commercial Services . 18 5.1.2 Using Your Own Computers . 18 Mac and other Unix . 19 Windows ................................................... 19 Virtual Machines . 19 5.2 Self-test . 19 6 Using Unix 21 6.1 Keep It Simple, Stupid . 21 6.2 What is Unix? . 22 6.2.1 Aw, man... I Have to Learn Another System? . 22 6.2.2 Operating System or Religion? . 24 6.2.3 The Unix Standard API . 27 6.2.4 Shake Out the Bugs . 28 6.2.5 The Unix Standard UI . 29 6.2.6 Freedom of Choice . 29 6.2.7 Fast, Stable and Secure . 29 6.2.8 Sharing Resources . 30 6.3 Self-test . 30 6.4 Unix User Interfaces . 30 6.4.1 Graphical User Interfaces (GUIs) . 30 6.4.2 X11 on Mac OS X . 33 The Research Computing User’s Guide v 6.4.3 Command Line Interfaces (CLIs): Unix Shells . 33 6.4.4 Terminals . 35 6.4.5 Basic Shell Use . 36 6.4.6 Self-test . 37 6.5 Still Need Windows? Don’t Panic! . 38 6.5.1 Cygwin: Try This First . 38 6.5.2 Windows Subsystem for Linux: Another Compatibility Layer . 51 6.6 Logging In Remotely . 52 6.6.1 Unix to Unix . 52 6.6.2 Windows to Unix . 53 Cygwin . 53 PuTTY .................................................... 53 6.6.3 Terminal Types . 56 6.6.4 Self-test . 56 6.7 Unix Command Basics . 57 6.7.1 Self-test . 58 6.8 Basic Shell Tools . 58 6.8.1 Common Unix Shells . 58 6.8.2 Command History . 59 6.8.3 Auto-completion . ..

513

Copy Link

Recommended publications

Linux on the Road
Linux on the Road Linux with Laptops, Notebooks, PDAs, Mobile Phones and Other Portable Devices Werner Heuser <wehe[AT]tuxmobil.org> Linux Mobile Edition Edition Version 3.22 TuxMobil Berlin Copyright © 2000-2011 Werner Heuser 2011-12-12 Revision History Revision 3.22 2011-12-12 Revised by: wh The address of the opensuse-mobile mailing list has been added, a section power management for graphics cards has been added, a short description of Intel's LinuxPowerTop project has been added, all references to Suspend2 have been changed to TuxOnIce, links to OpenSync and Funambol syncronization packages have been added, some notes about SSDs have been added, many URLs have been checked and some minor improvements have been made. Revision 3.21 2005-11-14 Revised by: wh Some more typos have been fixed. Revision 3.20 2005-11-14 Revised by: wh Some typos have been fixed. Revision 3.19 2005-11-14 Revised by: wh A link to keytouch has been added, minor changes have been made. Revision 3.18 2005-10-10 Revised by: wh Some URLs have been updated, spelling has been corrected, minor changes have been made. Revision 3.17.1 2005-09-28 Revised by: sh A technical and a language review have been performed by Sebastian Henschel. Numerous bugs have been fixed and many URLs have been updated. Revision 3.17 2005-08-28 Revised by: wh Some more tools added to external monitor/projector section, link to Zaurus Development with Damn Small Linux added to cross-compile section, some additions about acoustic management for hard disks added, references to X.org added to X11 sections, link to laptop-mode-tools added, some URLs updated, spelling cleaned, minor changes.
[Show full text]
Beginning Portable Shell Scripting from Novice to Professional
Beginning Portable Shell Scripting From Novice to Professional Peter Seebach 10436fmfinal 1 10/23/08 10:40:24 PM Beginning Portable Shell Scripting: From Novice to Professional Copyright © 2008 by Peter Seebach All rights reserved. No part of this work may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopying, recording, or by any information storage or retrieval system, without the prior written permission of the copyright owner and the publisher. ISBN-13 (pbk): 978-1-4302-1043-6 ISBN-10 (pbk): 1-4302-1043-5 ISBN-13 (electronic): 978-1-4302-1044-3 ISBN-10 (electronic): 1-4302-1044-3 Printed and bound in the United States of America 9 8 7 6 5 4 3 2 1 Trademarked names may appear in this book. Rather than use a trademark symbol with every occurrence of a trademarked name, we use the names only in an editorial fashion and to the benefit of the trademark owner, with no intention of infringement of the trademark. Lead Editor: Frank Pohlmann Technical Reviewer: Gary V. Vaughan Editorial Board: Clay Andres, Steve Anglin, Ewan Buckingham, Tony Campbell, Gary Cornell, Jonathan Gennick, Michelle Lowman, Matthew Moodie, Jeffrey Pepper, Frank Pohlmann, Ben Renow-Clarke, Dominic Shakeshaft, Matt Wade, Tom Welsh Project Manager: Richard Dal Porto Copy Editor: Kim Benbow Associate Production Director: Kari Brooks-Copony Production Editor: Katie Stence Compositor: Linda Weidemann, Wolf Creek Press Proofreader: Dan Shaw Indexer: Broccoli Information Management Cover Designer: Kurt Krames Manufacturing Director: Tom Debolski Distributed to the book trade worldwide by Springer-Verlag New York, Inc., 233 Spring Street, 6th Floor, New York, NY 10013.
[Show full text]
La Sécurité Informatique Edition Livres Pour Tous (
La sécurité informatique Edition Livres pour tous (www.livrespourtous.com) PDF générés en utilisant l’atelier en source ouvert « mwlib ». Voir http://code.pediapress.com/ pour plus d’informations. PDF generated at: Sat, 13 Jul 2013 18:26:11 UTC Contenus Articles 1-Principes généraux 1 Sécurité de l'information 1 Sécurité des systèmes d'information 2 Insécurité du système d'information 12 Politique de sécurité du système d'information 17 Vulnérabilité (informatique) 21 Identité numérique (Internet) 24 2-Attaque, fraude, analyse et cryptanalyse 31 2.1-Application 32 Exploit (informatique) 32 Dépassement de tampon 34 Rétroingénierie 40 Shellcode 44 2.2-Réseau 47 Attaque de l'homme du milieu 47 Attaque de Mitnick 50 Attaque par rebond 54 Balayage de port 55 Attaque par déni de service 57 Empoisonnement du cache DNS 66 Pharming 69 Prise d'empreinte de la pile TCP/IP 70 Usurpation d'adresse IP 71 Wardriving 73 2.3-Système 74 Écran bleu de la mort 74 Fork bomb 82 2.4-Mot de passe 85 Attaque par dictionnaire 85 Attaque par force brute 87 2.5-Site web 90 Cross-site scripting 90 Défacement 93 2.6-Spam/Fishing 95 Bombardement Google 95 Fraude 4-1-9 99 Hameçonnage 102 2.7-Cloud Computing 106 Sécurité du cloud 106 3-Logiciel malveillant 114 Logiciel malveillant 114 Virus informatique 120 Ver informatique 125 Cheval de Troie (informatique) 129 Hacktool 131 Logiciel espion 132 Rootkit 134 Porte dérobée 145 Composeur (logiciel) 149 Charge utile 150 Fichier de test Eicar 151 Virus de boot 152 4-Concepts et mécanismes de sécurité 153 Authentification forte
[Show full text]
Cygwin User's Guide
Cygwin User’s Guide Cygwin User’s Guide ii Copyright © Cygwin authors Permission is granted to make and distribute verbatim copies of this documentation provided the copyright notice and this per- mission notice are preserved on all copies. Permission is granted to copy and distribute modiﬁed versions of this documentation under the conditions for verbatim copying, provided that the entire resulting derived work is distributed under the terms of a permission notice identical to this one. Permission is granted to copy and distribute translations of this documentation into another language, under the above conditions for modiﬁed versions, except that this permission notice may be stated in a translation approved by the Free Software Foundation. Cygwin User’s Guide iii Contents 1 Cygwin Overview 1 1.1 What is it? . .1 1.2 Quick Start Guide for those more experienced with Windows . .1 1.3 Quick Start Guide for those more experienced with UNIX . .1 1.4 Are the Cygwin tools free software? . .2 1.5 A brief history of the Cygwin project . .2 1.6 Highlights of Cygwin Functionality . .3 1.6.1 Introduction . .3 1.6.2 Permissions and Security . .3 1.6.3 File Access . .3 1.6.4 Text Mode vs. Binary Mode . .4 1.6.5 ANSI C Library . .4 1.6.6 Process Creation . .5 1.6.6.1 Problems with process creation . .5 1.6.7 Signals . .6 1.6.8 Sockets . .6 1.6.9 Select . .7 1.7 What’s new and what changed in Cygwin . .7 1.7.1 What’s new and what changed in 3.2 .
[Show full text]
A First Course to Openfoam
Basic Shell Scripting Slides from Wei Feinstein HPC User Services LSU HPC & LON [email protected] September 2018 Outline • Introduction to Linux Shell • Shell Scripting Basics • Variables/Special Characters • Arithmetic Operations • Arrays • Beyond Basic Shell Scripting – Flow Control – Functions • Advanced Text Processing Commands (grep, sed, awk) Basic Shell Scripting 2 Linux System Architecture Basic Shell Scripting 3 Linux Shell What is a Shell ▪ An application running on top of the kernel and provides a command line interface to the system ▪ Process user’s commands, gather input from user and execute programs ▪ Types of shell with varied features o sh o csh o ksh o bash o tcsh Basic Shell Scripting 4 Shell Comparison Software sh csh ksh bash tcsh Programming language y y y y y Shell variables y y y y y Command alias n y y y y Command history n y y y y Filename autocompletion n y* y* y y Command line editing n n y* y y Job control n y y y y *: not by default http://www.cis.rit.edu/class/simg211/unixintro/Shell.html Basic Shell Scripting 5 What can you do with a shell? ▪ Check the current shell ▪ echo $SHELL ▪ List available shells on the system ▪ cat /etc/shells ▪ Change to another shell ▪ csh ▪ Date ▪ date ▪ wget: get online files ▪ wget https://ftp.gnu.org/gnu/gcc/gcc-7.1.0/gcc-7.1.0.tar.gz ▪ Compile and run applications ▪ gcc hello.c –o hello ▪ ./hello ▪ What we need to learn today? o Automation of an entire script of commands! o Use the shell script to run jobs – Write job scripts Basic Shell Scripting 6 Shell Scripting ▪ Script: a program written for a software environment to automate execution of tasks ▪ A series of shell commands put together in a file ▪ When the script is executed, those commands will be executed one line at a time automatically ▪ Shell script is interpreted, not compiled.
[Show full text]
Fira Code: Monospaced Font with Programming Ligatures
Personal Open source Business Explore Pricing Blog Support This repository Sign in Sign up tonsky / FiraCode Watch 282 Star 9,014 Fork 255 Code Issues 74 Pull requests 1 Projects 0 Wiki Pulse Graphs Monospaced font with programming ligatures 145 commits 1 branch 15 releases 32 contributors OFL-1.1 master New pull request Find file Clone or download lf- committed with tonsky Add mintty to the ligatures-unsupported list (#284) Latest commit d7dbc2d 16 days ago distr Version 1.203 (added `__`, closes #120) a month ago showcases Version 1.203 (added `__`, closes #120) a month ago .gitignore - Removed `!!!` `???` `;;;` `&&&` `|||` `=~` (closes #167) `~~~` `%%%` 3 months ago FiraCode.glyphs Version 1.203 (added `__`, closes #120) a month ago LICENSE version 0.6 a year ago README.md Add mintty to the ligatures-unsupported list (#284) 16 days ago gen_calt.clj Removed `/**` `**/` and disabled ligatures for `/*/` `*/*` sequences … 2 months ago release.sh removed Retina weight from webfonts 3 months ago README.md Fira Code: monospaced font with programming ligatures Problem Programmers use a lot of symbols, often encoded with several characters. For the human brain, sequences like -> , <= or := are single logical tokens, even if they take two or three characters on the screen. Your eye spends a non-zero amount of energy to scan, parse and join multiple characters into a single logical one. Ideally, all programming languages should be designed with full-fledged Unicode symbols for operators, but that’s not the case yet. Solution Download v1.203 · How to install · News & updates Fira Code is an extension of the Fira Mono font containing a set of ligatures for common programming multi-character combinations.
[Show full text]
Github: a Case Study of Linux/BSD Perceptions from Microsoft's
1 FLOSS != GitHub: A Case Study of Linux/BSD Perceptions from Microsoft’s Acquisition of GitHub Raula Gaikovina Kula∗, Hideki Hata∗, Kenichi Matsumoto∗ ∗Nara Institute of Science and Technology, Japan {raula-k, hata, matumoto}@is.naist.jp Abstract—In 2018, the software industry giants Microsoft made has had its share of disagreements with Microsoft [6], [7], a move into the Open Source world by completing the acquisition [8], [9], the only reported negative opinion of free software of mega Open Source platform, GitHub. This acquisition was not community has different attitudes towards GitHub is the idea without controversy, as it is well-known that the free software communities includes not only the ability to use software freely, of ‘forking’ so far, as it it is considered as a danger to FLOSS but also the libre nature in Open Source Software. In this study, development [10]. our aim is to explore these perceptions in FLOSS developers. We In this paper, we report on how external events such as conducted a survey that covered traditional FLOSS source Linux, acquisition of the open source platform by a closed source and BSD communities and received 246 developer responses. organization triggers a FLOSS developers such the Linux/ The results of the survey conﬁrm that the free community did trigger some communities to move away from GitHub and raised BSD Free Software communities. discussions into free and open software on the GitHub platform. The study reminds us that although GitHub is inﬂuential and II. TARGET SUBJECTS AND SURVEY DESIGN trendy, it does not representative all FLOSS communities.
[Show full text]
Examining the Hamilton C Shell
EXAMINING ROO M Examining the Hamilton C Shell Unix power for OS/2 Scott Richman tarting OS/2 for the first time was, file system), long filenames, and threads. in the background and the server will for me, like unlocking a Ferarri, Additionally, the Shell supports large be kept busy. sitting behind its wheel and find command lines and pipes (up to 64K) ing a Yugo's dash. What a disap and includes faster and more powerful Supporting Procedures pointment. Sure, the engine and utilities than those supplied with OS/2. Script programmers can create C Shell suspensionS were first rate, but the con This is more than Unix - this is a pow procedures, which are more like func trols were minimal, the clutch was stiff, erful requirement for development un tions: They accept a parameter list and and the pedals were nonresponsive! der OS/2. The ability to execute C shells return a value. These procedures are OS/2 comes with great stuff, but CMD simultaneously in different Presenta compiled into C Shell memory and are .EXE, the default command-line pro tion Manager (PM) text windows con then executed as new shell commands. cessor, is poor compared to the pow verts your PC into a flexible workstation. Procedures can greatly extend the erfuloperating system beneath. CMD.EXE The Hamilton C Shell comes with power and flexibility of your environ appears to be a port of the MS-DOS many programs and shell scripts. To ment. COMMAND.COM and lacks the major install the Shell, you simply copy the As an example, consider ZCW.CSH features of a serious front end.
[Show full text]
Linux Pocket Guide.Pdf
3rd Edition Linux Pocket Guide ESSENTIAL COMMANDS Daniel J. Barrett 3RD EDITION Linux Pocket Guide Daniel J. Barrett Linux Pocket Guide by Daniel J. Barrett Copyright © 2016 Daniel Barrett. All rights reserved. Printed in the United States of America. Published by O’Reilly Media, Inc., 1005 Gravenstein Highway North, Sebasto‐ pol, CA 95472. O’Reilly books may be purchased for educational, business, or sales promo‐ tional use. Online editions are also available for most titles (http://safaribook‐ sonline.com). For more information, contact our corporate/institutional sales department: 800-998-9938 or [email protected]. Editor: Nan Barber Production Editor: Nicholas Adams Copyeditor: Jasmine Kwityn Proofreader: Susan Moritz Indexer: Daniel Barrett Interior Designer: David Futato Cover Designer: Karen Montgomery Illustrator: Rebecca Demarest June 2016: Third Edition Revision History for the Third Edition 2016-05-27: First Release See http://oreilly.com/catalog/errata.csp?isbn=9781491927571 for release details. The O’Reilly logo is a registered trademark of O’Reilly Media, Inc. Linux Pocket Guide, the cover image, and related trade dress are trademarks of O’Reilly Media, Inc. While the publisher and the author have used good faith efforts to ensure that the information and instructions contained in this work are accurate, the publisher and the author disclaim all responsibility for errors or omissions, including without limitation responsibility for damages resulting from the use of or reliance on this work. Use of the information and instructions contained in this work is at your own risk. If any code samples or other technology this work contains or describes is subject to open source licenses or the intellec‐ tual property rights of others, it is your responsibility to ensure that your use thereof complies with such licenses and/or rights.
[Show full text]
The Bioinformatics Lab Linux Proficiency Terminal-Based Text Editors Version Control Systems
The Bioinformatics Lab Linux proficiency terminal-based text editors version control systems Jonas Reeb 30.04.2013 “What makes you proficient on the command line?” - General ideas I Use CLIs in the first place I Use each tool for what it does best I Chain tools for more complex tasks I Use power of shell for small scripting jobs I Automate repeating tasks I Knowledge of regular expression 1 / 22 Standard tools I man I ls/cd/mkdir/rm/touch/cp/mv/chmod/cat... I grep, sort, uniq I find I wget/curl I scp/ssh I top(/htop/iftop/iotop) I bg/fg 2 / 22 Input-Output RedirectionI By default three streams (“files”) open Name Descriptor stdin 0 stdout 1 stderr 2 Any program can check for its file descriptors’ redirection! (isatty) 3 / 22 Input-Output RedirectionII Output I M>f Redirect file descriptor M to file f, e.g. 1>f I Use >> for appending I &>f Redirect stdout and stderr to f I M>&N Redirect fd M to fd N Input I 0<f Read from file f 4 / 22 Pipes I Forward output of one program to input of another I Essential for Unix philosophy of specialized tools I grep -P -v "^>" *.fa | sort -u > seqs I Input and arguments are different things. Use xargs for arguments: ls *.fa | xargs rm 5 / 22 Scripting I Quick way to get basic programs running I Basic layout: #!/bin/bash if test"$1" then count=$1 else count=0 fi for i in {1..10} do echo $((i+count)) let"count +=1" done 6 / 22 Motivation - “What makes a good text editor” I Fast execution, little system load I Little bandwidth needed I Available for all (your) major platforms –> Familiar environment I Fully controllable via keyboard I Extensible and customizable I Auto-indent, Auto-complete, Syntax highlighting, Folding, ..
[Show full text]
Unix and Linux System Administration and Shell Programming
Unix and Linux System Administration and Shell Programming Unix and Linux System Administration and Shell Programming version 56 of August 12, 2014 Copyright © 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2009, 2010, 2011, 2012, 2013, 2014 Milo This book includes material from the http://www.osdata.com/ website and the text book on computer programming. Distributed on the honor system. Print and read free for personal, non-profit, and/or educational purposes. If you like the book, you are encouraged to send a donation (U.S dollars) to Milo, PO Box 5237, Balboa Island, California, USA 92662. This is a work in progress. For the most up to date version, visit the website http://www.osdata.com/ and http://www.osdata.com/programming/shell/unixbook.pdf — Please add links from your website or Facebook page. Professors and Teachers: Feel free to take a copy of this PDF and make it available to your class (possibly through your academic website). This way everyone in your class will have the same copy (with the same page numbers) despite my continual updates. Please try to avoid posting it to the public internet (to avoid old copies confusing things) and take it down when the class ends. You can post the same or a newer version for each succeeding class. Please remove old copies after the class ends to prevent confusing the search engines. You can contact me with a specific version number and class end date and I will put it on my website. version 56 page 1 Unix and Linux System Administration and Shell Programming Unix and Linux Administration and Shell Programming chapter 0 This book looks at Unix (and Linux) shell programming and system administration.
[Show full text]
CLI Introduction
2021/07/26 13:46 (UTC) 1/3 CLI Introduction Por traducir - Victor CLI Introduction A command line interface (CLI) is a way of interacting with an operating system or programs using text commands. As opposed to graphical user interfaces (GUI), CLI, due to its nature, is heavily keyboard oriented. Despite a steeper learning curve, this way of interaction has a wide range of benefits and is therefore preferred by many users. Since most Slackware configuration and administration is performed using the command line interface, it is highly recommended to understand and feel comfortable using them. An additional advantage is that 99% of the commands are universal for most Linux distributions. Things you learn here will most probably be applicable elsewhere. CLI is Good - Do Not Fear It The following are some of the advantages to using CLI: 1. CLI commands are more precise and powerful than GUI interaction. 2. GUI tools are often just frontends for command line tools. This brings another layer of complexity and is more difficult to troubleshoot. 3. Most commands are distro agnostic: they work on other Linux systems as well. 4. CLI commands are building blocks that can be glued together to create complex commands or custom scripts that are tailored to your needs. 5. CLI commands can be used to automate repetitive tasks 6. CLI commands do not change often. 7. Commands make a universal language that is easy to communicate and follow by other internet users. 8. CLI scripts can include comments that can act as notes and internal documentation explaining the functionality and simplifying future modifications.
[Show full text]