DOCSLIB.ORG

UWM Research Computing User's Guide

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
UWM Research Computing User's Guide UWM Research Computing User’s Guide i UWM Research Computing User’s Guide October 30, 2020 UWM Research Computing User’s Guide ii COLLABORATORS TITLE : UWM Research Computing User’s Guide ACTION NAME DATE SIGNATURE WRITTEN BY Jason W. Bacon October 30, 2020 Initial HTcondor Lars Olson October 30, 2020 content UWM Research Computing User’s Guide iii Contents 1 Goals 2 2 Self-Study Instructions 3 2.1 Unix Self-Study Instructions . .3 2.2 Parallel Computing Self-Study Instructions . .3 I Research Computing4 3 Computational Science 5 3.1 So, What is Computational Science? . .5 3.2 Not Just for Nerds Anymore . .5 3.3 Achieving Our Goals . .5 3.3.1 The Computation Time Line . .5 3.3.2 Development Time . .6 3.3.3 Deployment Time . .6 3.3.4 Learning Time . .6 3.3.5 Run Time . .6 3.4 Common Methods Used in Computational Science . .6 3.4.1 Numerical Analysis . .6 3.4.2 Computational Modeling . .7 3.4.3 Data Mining . .8 3.4.4 Parameter Sweeps . .8 3.4.5 Data Sifting . .9 3.4.6 Monte Carlo . .9 3.4.7 Everything Else . 10 3.5 Self-test . 10 UWM Research Computing User’s Guide iv 4 Where do I get the Software? 11 4.1 But I Hate Programming... 11 4.2 BuyIt......................................................... 12 4.3 Download It . 12 4.3.1 How to Shoot Yourself in the Foot with Open Source Software . 13 4.3.2 How Not to Shoot Yourself in the Foot with Open Source Software . 14 4.4 The Realities of Software Support . 14 4.5 A Comprehensive List of Research Software . 15 4.6 Write It . 15 4.7 Self-test . 16 5 Running Your Software 18 5.1 Where do I Run It? . 18 5.1.1 Public Resources . 18 College Computer Labs . 18 College Clusters and Grids . 18 XSEDE, Open Science Grid . 18 Commercial Services . 18 5.1.2 Using Your Own Computers . 18 Mac and other Unix . 19 Windows ................................................... 19 Virtual Machines . 19 5.2 Self-test . 19 6 Using UWM Resources 20 6.1 Obtaining Access . 20 6.2 Good Neighbor Guidelines for Shared Computers . 20 6.2.1 Security . 20 6.2.2 Resource Use . 21 6.3 The Ganglia Resource Monitor . 22 6.4 Connecting to UWM Resources . 22 6.4.1 Logging Into the Shell Environment . 22 6.4.2 File Transfers . 22 6.4.3 Direct File Server Connections . 23 6.5 Cluster Data Storage . 24 6.6 UWM Policies and Guidelines . 26 6.7 Using Available Software . 26 6.8 Code Development and Testing . 27 6.9 Off-campus Access . 28 6.10 UWM Data Management Services . 31 6.11 Customizing Your Login Shell . 31 UWM Research Computing User’s Guide v 7 Using Unix 32 7.1 Keep It Simple, Stupid . 32 7.2 What is Unix? . 33 7.2.1 Aw, man... I Have to Learn Another System? . 33 7.2.2 Operating System or Religion? . 35 7.2.3 The Unix Standard API . 38 7.2.4 Shake Out the Bugs . 39 7.2.5 The Unix Standard UI . 40 7.2.6 Freedom of Choice . 40 7.2.7 Fast, Stable and Secure . 40 7.2.8 Sharing Resources . 41 7.3 Self-test . 41 7.4 Unix User Interfaces . 41 7.4.1 Graphical User Interfaces (GUIs) . 41 7.4.2 X11 on Mac OS X . 44 7.4.3 Command Line Interfaces (CLIs): Unix Shells . 44 7.4.4 Terminals . 46 7.4.5 Basic Shell Use . ..
Load more

Recommended publications
  • Freenas® 11.0 User Guide
    FreeNAS® 11.0 User Guide June 2017 Edition FreeNAS® IS © 2011-2017 iXsystems FreeNAS® AND THE FreeNAS® LOGO ARE REGISTERED TRADEMARKS OF iXsystems FreeBSD® IS A REGISTERED TRADEMARK OF THE FreeBSD Foundation WRITTEN BY USERS OF THE FreeNAS® network-attached STORAGE OPERATING system. VERSION 11.0 CopYRIGHT © 2011-2017 iXsystems (https://www.ixsystems.com/) CONTENTS WELCOME....................................................1 TYPOGRAPHIC Conventions...........................................2 1 INTRODUCTION 3 1.1 NeW FeaturES IN 11.0..........................................3 1.2 HarDWARE Recommendations.....................................4 1.2.1 RAM...............................................5 1.2.2 The OperATING System DeVICE.................................5 1.2.3 StorAGE Disks AND ContrOLLERS.................................6 1.2.4 Network INTERFACES.......................................7 1.3 Getting Started WITH ZFS........................................8 2 INSTALLING AND UpgrADING 9 2.1 Getting FreeNAS® ............................................9 2.2 PrEPARING THE Media.......................................... 10 2.2.1 On FreeBSD OR Linux...................................... 10 2.2.2 On WindoWS.......................................... 11 2.2.3 On OS X............................................. 11 2.3 Performing THE INSTALLATION....................................... 12 2.4 INSTALLATION TROUBLESHOOTING...................................... 18 2.5 UpgrADING................................................ 19 2.5.1 Caveats:............................................
    [Show full text]
  • Linux on the Road
    Linux on the Road Linux with Laptops, Notebooks, PDAs, Mobile Phones and Other Portable Devices Werner Heuser <wehe[AT]tuxmobil.org> Linux Mobile Edition Edition Version 3.22 TuxMobil Berlin Copyright © 2000-2011 Werner Heuser 2011-12-12 Revision History Revision 3.22 2011-12-12 Revised by: wh The address of the opensuse-mobile mailing list has been added, a section power management for graphics cards has been added, a short description of Intel's LinuxPowerTop project has been added, all references to Suspend2 have been changed to TuxOnIce, links to OpenSync and Funambol syncronization packages have been added, some notes about SSDs have been added, many URLs have been checked and some minor improvements have been made. Revision 3.21 2005-11-14 Revised by: wh Some more typos have been fixed. Revision 3.20 2005-11-14 Revised by: wh Some typos have been fixed. Revision 3.19 2005-11-14 Revised by: wh A link to keytouch has been added, minor changes have been made. Revision 3.18 2005-10-10 Revised by: wh Some URLs have been updated, spelling has been corrected, minor changes have been made. Revision 3.17.1 2005-09-28 Revised by: sh A technical and a language review have been performed by Sebastian Henschel. Numerous bugs have been fixed and many URLs have been updated. Revision 3.17 2005-08-28 Revised by: wh Some more tools added to external monitor/projector section, link to Zaurus Development with Damn Small Linux added to cross-compile section, some additions about acoustic management for hard disks added, references to X.org added to X11 sections, link to laptop-mode-tools added, some URLs updated, spelling cleaned, minor changes.
    [Show full text]
  • Beginning Portable Shell Scripting from Novice to Professional
    Beginning Portable Shell Scripting From Novice to Professional Peter Seebach 10436fmfinal 1 10/23/08 10:40:24 PM Beginning Portable Shell Scripting: From Novice to Professional Copyright © 2008 by Peter Seebach All rights reserved. No part of this work may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopying, recording, or by any information storage or retrieval system, without the prior written permission of the copyright owner and the publisher. ISBN-13 (pbk): 978-1-4302-1043-6 ISBN-10 (pbk): 1-4302-1043-5 ISBN-13 (electronic): 978-1-4302-1044-3 ISBN-10 (electronic): 1-4302-1044-3 Printed and bound in the United States of America 9 8 7 6 5 4 3 2 1 Trademarked names may appear in this book. Rather than use a trademark symbol with every occurrence of a trademarked name, we use the names only in an editorial fashion and to the benefit of the trademark owner, with no intention of infringement of the trademark. Lead Editor: Frank Pohlmann Technical Reviewer: Gary V. Vaughan Editorial Board: Clay Andres, Steve Anglin, Ewan Buckingham, Tony Campbell, Gary Cornell, Jonathan Gennick, Michelle Lowman, Matthew Moodie, Jeffrey Pepper, Frank Pohlmann, Ben Renow-Clarke, Dominic Shakeshaft, Matt Wade, Tom Welsh Project Manager: Richard Dal Porto Copy Editor: Kim Benbow Associate Production Director: Kari Brooks-Copony Production Editor: Katie Stence Compositor: Linda Weidemann, Wolf Creek Press Proofreader: Dan Shaw Indexer: Broccoli Information Management Cover Designer: Kurt Krames Manufacturing Director: Tom Debolski Distributed to the book trade worldwide by Springer-Verlag New York, Inc., 233 Spring Street, 6th Floor, New York, NY 10013.
    [Show full text]
  • Building a SAN-Less Private Cloud with IBM Powervm and IBM Powervc
    Front cover Building a SAN-less Private Cloud with IBM PowerVM and IBM PowerVC Javier Bazan Lazcano Stephen Lutz Redpaper International Technical Support Organization Building a SAN-less Private Cloud with IBM PowerVM and IBM PowerVC July 2018 REDP-5455-02 Note: Before using this information and the product it supports, read the information in “Notices” on page v. Third Edition (July 2018) This edition applies to Version 1, Release 4, Modification 1 of IBM Cloud PowerVC Manager for Software-Defined Infrastructure V1.1 (product number 5765-VCD). © Copyright International Business Machines Corporation 2017, 2018. All rights reserved. Note to U.S. Government Users Restricted Rights -- Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp. Contents Notices . .v Trademarks . vi Preface . vii Authors. vii Now you can become a published author, too! . viii Comments welcome. viii Stay connected to IBM Redbooks . viii Chapter 1. Architecture of a software-defined infrastructure solution with IBM Power Systems servers and IBM PowerVC . 1 1.1 Software-defined infrastructure . 2 1.2 The motivation behind software-defined infrastructure . 2 1.3 Use cases . 4 1.3.1 Use case 1: Building a storage area network-less cloud . 4 1.3.2 Use case 2: Building an iSCSI-backed cloud . 5 1.3.3 More use cases. 5 1.4 Rack topologies and components . 6 1.4.1 Starter cloud . 7 1.4.2 Mini cloud . 8 1.4.3 Rack scale . 9 1.4.4 Storage/management switches. 10 1.4.5 Data switches . 10 1.4.6 Network nodes .
    [Show full text]
  • Cygwin User's Guide
    Cygwin User’s Guide Cygwin User’s Guide ii Copyright © Cygwin authors Permission is granted to make and distribute verbatim copies of this documentation provided the copyright notice and this per- mission notice are preserved on all copies. Permission is granted to copy and distribute modified versions of this documentation under the conditions for verbatim copying, provided that the entire resulting derived work is distributed under the terms of a permission notice identical to this one. Permission is granted to copy and distribute translations of this documentation into another language, under the above conditions for modified versions, except that this permission notice may be stated in a translation approved by the Free Software Foundation. Cygwin User’s Guide iii Contents 1 Cygwin Overview 1 1.1 What is it? . .1 1.2 Quick Start Guide for those more experienced with Windows . .1 1.3 Quick Start Guide for those more experienced with UNIX . .1 1.4 Are the Cygwin tools free software? . .2 1.5 A brief history of the Cygwin project . .2 1.6 Highlights of Cygwin Functionality . .3 1.6.1 Introduction . .3 1.6.2 Permissions and Security . .3 1.6.3 File Access . .3 1.6.4 Text Mode vs. Binary Mode . .4 1.6.5 ANSI C Library . .4 1.6.6 Process Creation . .5 1.6.6.1 Problems with process creation . .5 1.6.7 Signals . .6 1.6.8 Sockets . .6 1.6.9 Select . .7 1.7 What’s new and what changed in Cygwin . .7 1.7.1 What’s new and what changed in 3.2 .
    [Show full text]
  • The Elinks Manual the Elinks Manual Table of Contents Preface
    The ELinks Manual The ELinks Manual Table of Contents Preface.......................................................................................................................................................ix 1. Getting ELinks up and running...........................................................................................................1 1.1. Building and Installing ELinks...................................................................................................1 1.2. Requirements..............................................................................................................................1 1.3. Recommended Libraries and Programs......................................................................................1 1.4. Further reading............................................................................................................................2 1.5. Tips to obtain a very small static elinks binary...........................................................................2 1.6. ECMAScript support?!...............................................................................................................4 1.6.1. Ok, so how to get the ECMAScript support working?...................................................4 1.6.2. The ECMAScript support is buggy! Shall I blame Mozilla people?..............................6 1.6.3. Now, I would still like NJS or a new JS engine from scratch. .....................................6 1.7. Feature configuration file (features.conf).............................................................................7
    [Show full text]
  • A First Course to Openfoam
    Basic Shell Scripting Slides from Wei Feinstein HPC User Services LSU HPC & LON [email protected] September 2018 Outline • Introduction to Linux Shell • Shell Scripting Basics • Variables/Special Characters • Arithmetic Operations • Arrays • Beyond Basic Shell Scripting – Flow Control – Functions • Advanced Text Processing Commands (grep, sed, awk) Basic Shell Scripting 2 Linux System Architecture Basic Shell Scripting 3 Linux Shell What is a Shell ▪ An application running on top of the kernel and provides a command line interface to the system ▪ Process user’s commands, gather input from user and execute programs ▪ Types of shell with varied features o sh o csh o ksh o bash o tcsh Basic Shell Scripting 4 Shell Comparison Software sh csh ksh bash tcsh Programming language y y y y y Shell variables y y y y y Command alias n y y y y Command history n y y y y Filename autocompletion n y* y* y y Command line editing n n y* y y Job control n y y y y *: not by default http://www.cis.rit.edu/class/simg211/unixintro/Shell.html Basic Shell Scripting 5 What can you do with a shell? ▪ Check the current shell ▪ echo $SHELL ▪ List available shells on the system ▪ cat /etc/shells ▪ Change to another shell ▪ csh ▪ Date ▪ date ▪ wget: get online files ▪ wget https://ftp.gnu.org/gnu/gcc/gcc-7.1.0/gcc-7.1.0.tar.gz ▪ Compile and run applications ▪ gcc hello.c –o hello ▪ ./hello ▪ What we need to learn today? o Automation of an entire script of commands! o Use the shell script to run jobs – Write job scripts Basic Shell Scripting 6 Shell Scripting ▪ Script: a program written for a software environment to automate execution of tasks ▪ A series of shell commands put together in a file ▪ When the script is executed, those commands will be executed one line at a time automatically ▪ Shell script is interpreted, not compiled.
    [Show full text]
  • Powerview Command Reference
    PowerView Command Reference TRACE32 Online Help TRACE32 Directory TRACE32 Index TRACE32 Documents ...................................................................................................................... PowerView User Interface ............................................................................................................ PowerView Command Reference .............................................................................................1 History ...................................................................................................................................... 12 ABORT ...................................................................................................................................... 13 ABORT Abort driver program 13 AREA ........................................................................................................................................ 14 AREA Message windows 14 AREA.CLEAR Clear area 15 AREA.CLOSE Close output file 15 AREA.Create Create or modify message area 16 AREA.Delete Delete message area 17 AREA.List Display a detailed list off all message areas 18 AREA.OPEN Open output file 20 AREA.PIPE Redirect area to stdout 21 AREA.RESet Reset areas 21 AREA.SAVE Save AREA window contents to file 21 AREA.Select Select area 22 AREA.STDERR Redirect area to stderr 23 AREA.STDOUT Redirect area to stdout 23 AREA.view Display message area in AREA window 24 AutoSTOre ..............................................................................................................................
    [Show full text]
  • Push-Based Job Submission Using Reverse SSH Connections
    rvGAHP – Push-Based Job Submission using Reverse SSH Connections Scott Callaghan Gideon Juve Karan Vahi University of Southern California USC Information Sciences Institute USC Information Sciences Institute Los Angeles, California Marina Del Rey, California Marina Del Rey, California [email protected] [email protected] [email protected] Philip J. Maechling Thomas H. Jordan Ewa Deelman University of Southern California University of Southern California USC Information Sciences Institute Los Angeles, California Los Angeles, California Marina Del Rey, California [email protected] [email protected] [email protected] ABSTRACT using Reverse SSH Connections. In WORKS’17: WORKS’17: 12th Workshop Computational science researchers running large-scale scientific on Workflows in Support of Large-Scale Science, November 12–17, 2017, Denver, CO, USA. ACM, New York, NY, USA, 8 pages. https://doi.org/10.1145/3150994. workflow applications often want to run their workflows onthe 3151003 largest available compute systems to improve time to solution. Workflow tools used in distributed, heterogeneous, high perfor- mance computing environments typically rely on either a push- 1 INTRODUCTION based or a pull-based approach for resource provisioning from these Modern scientific applications typically require the execution of compute systems. However, many large clusters have moved to multiple codes, may include computational and data dependencies, two-factor authentication for job submission, making traditional au- and contain varied computational models ranging from a bag of tomated push-based job submission impossible. On the other hand, tasks to a monolithic parallel code. These applications are often pull-based approaches such as pilot jobs may lead to increased com- complex software suites with substantial computational and data plexity and a reduction in node-hour efficiency.
    [Show full text]
  • Examining the Hamilton C Shell
    EXAMINING ROO M Examining the Hamilton C Shell Unix power for OS/2 Scott Richman tarting OS/2 for the first time was, file system), long filenames, and threads. in the background and the server will for me, like unlocking a Ferarri, Additionally, the Shell supports large be kept busy. sitting behind its wheel and find­ command lines and pipes (up to 64K) ing a Yugo's dash. What a disap­ and includes faster and more powerful Supporting Procedures pointment. Sure, the engine and utilities than those supplied with OS/2. Script programmers can create C Shell suspensionS were first rate, but the con­ This is more than Unix - this is a pow­ procedures, which are more like func­ trols were minimal, the clutch was stiff, erful requirement for development un­ tions: They accept a parameter list and and the pedals were nonresponsive! der OS/2. The ability to execute C shells return a value. These procedures are OS/2 comes with great stuff, but CMD simultaneously in different Presenta­ compiled into C Shell memory and are .EXE, the default command-line pro­ tion Manager (PM) text windows con­ then executed as new shell commands. cessor, is poor compared to the pow­ verts your PC into a flexible workstation. Procedures can greatly extend the erfuloperating system beneath. CMD.EXE The Hamilton C Shell comes with power and flexibility of your environ­ appears to be a port of the MS-DOS many programs and shell scripts. To ment. COMMAND.COM and lacks the major install the Shell, you simply copy the As an example, consider ZCW.CSH features of a serious front end.
    [Show full text]
  • Lab Intro to Console Commands
    New Lab Intro to KDE Terminal Konsole After completing this lab activity the student will be able to; Access the KDE Terminal Konsole and enter basic commands. Enter commands using a typical command line interface (CLI). Explain the use of the following commands, ls, ls –al, dir, mkdir, whoami, Explain the directory structure of a typical user. This lab activity will introduce you to one of the many command line interfaces available in Linux/UNIX operating systems and a few of the most basic commands. The command line interface you will be using for this lab activity is the console called the Konsole and is also referred to as Terminal. Note: As you notice, in the KDE system many features are written with the capital letter “K” in place of the first letter or the utility to reflect the fact it was modified for the KDE system. The original UNIX system did not use a graphical user interface GUI but rather was a command line interface (CLI) similar to the command prompt in Windows operating systems. The command line interface is referred to as a shell. Even today the command line interface (the shell) is used to issue commands on a Linux server to minimize system resources. For example, there is no need to start the GUI on the server to add a new user to an existing system. Starting the GUI will reduce the system performance because it requires RAM to run the GUI. A GUI will affect the overall performance of the server when it is supporting many users (clients).
    [Show full text]
  • Introduction to Python University of Oxford Department of Particle Physics
    Particle Physics Cluster Infrastructure Introduction University of Oxford Department of Particle Physics October 2019 Vipul Davda Particle Physics Linux Systems Administrator Room 661 Telephone: x73389 [email protected] Particle Physics Computing Overview 1 Particle Physics Linux Infrastructure Distributed File System gluster NFS /data Worker Nodes /data/atlas /data/lhcb NFS HTCondor /home Batch Server Interactive Servers physics_s/eduroam Network Printer Managed Laptops Managed Desktops Particle Physics Computing Overview 2 Introduction to the Unix Operating System Unix is a Multi-User/Multi-Tasking operating system. Developed in 1969 at AT&T’s Bell Labs by Ken Thompson (Unix) Dennis Ritchie (C) Unix is written in C programming language. Unix was originally a command-line OS, but now has a graphical user interface. It is available in many different forms: Linux , Solaris, AIX, HP-UX, freeBSD It is a well-suited environment for program development: C, C++, Java, Fortran, Python… Unix is mainly used on large servers for scientific applications. Particle Physics Computing Overview 3 Linux Distributions Source: https://www.muylinux.com/2009/04/24/logos-de-distribuciones-gnulinux/ Particle Physics Computing Overview 4 Particle Physics Linux Infrastructure Particle Physics uses CentOS Linux on the cluster. It is a free version of RedHat Enterprise Linux. Particle Physics Computing Overview 5 Basic Linux Commands Particle Physics Computing Overview 6 Basic Linux Commands o ls - list directory contents ls –l - long listing
    [Show full text]