DOCSLIB.ORG

Introduction to Linux/Unix

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Introduction to Linux/Unix Introduction to Linux/Unix Xiaoge Wang, ICER [email protected] Feb. 4, 2016 How does this class work • We are going to cover some basics with hands on examples. • Exercises are denoted by the following icon: • Bold means commands which I expect you type on your terminal in most cases. Green and red sticky Use the sJcKy notes provided to help me help you. – No s%cky = I am worKing – Green = I am done and ready to move on (yea!) – Red = I am stucK and need more Jme and/or some help Agenda • IntroducJon • Linux – Commands • Navigaon • Get or create files • Organizing files • Closer looK into files • Search files • Distribute files • File permission • Learn new commands – Scripts • Pipeline • Make you own command • Environment of a shell • Summary Agenda • Linux – Commands • Navigaon • Get or create files • Organizing files • Closer looK into files • Search files • Distribute files • File permission • Learn new commands – Scripts • Pipeline • Make you own command • Environment of a shell • Summary Introduction • Get ready for adventure? – TicKet(account)? • Big map – Linux/Unix – Shell • Overview of the trail – Commands – Simple Shell script – Get ready for HPC. Exercise 0: Get ready • Connect to HPCC (gateway) – $ ssh [email protected] • Windows users MobaXterm • Mac users Terminal • Linux users? • Read important message – $ mod • Go to a development node – $ ssh dev-nodename Message of the day (mod) • Mac Show screen Big picture Shell Shell Big picture • Shell – CLI ✔ – GUI OS shell example Overview of the trail • Commands • Simple Shell script • Get ready for HPC Linux shell tour Ready? GO! Agenda • IntroducJon – Scripts • Pipeline • Make you own command • Environment of a shell • Summary Example 1: Navigation • TasK: Wander around on a node. Change from one directory to another directory and see the list of files in a directory. Must see directories: home, /, /bin. • Commands – echo $HOME – ls, ls –l, ls –a, – cd, cd .., cd ~, cd – – pwd • Concepts learned: – Short cuts: ~ , .. , . – Directories: current, parent, home, hidden – Command format ( opJons, parameters ) Exercise 1: • TasK: Wander around on scratch space. 1. Go to /mnt/ls15/scratch/users/yourNetID 2. Go to each directory and take looK. 3. Go to $SCRATCH directory and take looK. We Learned: “what is scratch space?” Example 2.1: Create a files • TasK: make a directory “ex1” that contains the file “Done”. Write the following in the file “Done”: This is what I have done. …… List of command you have done …... • Commands – mkdir – editor: nano – history – output using redirect > • We Learned: – Standard output redirect – editor Example 2.2: Get files • TasK: Make a directory “worKshop” that contains: 1. Cheat sheet of linux shell commands from hip://cli.learncodethehardway.org/ bash_cheat_sheet.pdf 2. “gunzip”. Copy from directory /bin • Commands – wget – cp, cp –r • We Learned: – Copy files and directory. Exercise 2: • TasK: Make a directory “WorKshop” that contains: 1. Cheat sheet of linux shell commands from hip://cli.learncodethehardway.org/ bash_cheat_sheet.pdf 2. Copy from /mnt/home/class0/Intro2Linux 3. A file “Done” same as example2.1 Hint: follow steps in example2.1 and 2.2 Example 3: Organizing files • TasK: Reorganize files. – Create new directory – Move a file from one directory to another directory (both up and down). – Rename a file • Commands – cp – mv – rm – mkdir • We Learned: – absolute/relave file name (path) – /bin, /lib, /doc, /src Exercise 3: • TasK: Organize files in “Intro2Linux” by copying data files into a directory “~/DATA” and moving pdf files into the directory “Intro2linux/DOC”. 1. Make directories “DATA” and “DOC” 2. copy data files to “DATA” 3. move pdf files to “DOC” Note: 1. DATA and DOC are at different levels. DOC is inside Intro2linux but DATA is outside. 2. Intro2linux contains same files as before. Example 4: Closer look • TasK: LooKing at the contents of the file “pet_store.csv”. Find out how many records of “dog” in it. • Commands: – cat, head, head –n , tail, tail –n, more, less – grep, grep –r, grep -i – wc, • We Learned: – Get more informaon from a file content. Exercise 4: Closer look • TasK: View the contents of the data file “polls.csv”. Find out how many polls were recorded from Michigan. – Go to the DATA directory and view the file – Use “MI” to find the data recorded from Michigan. Example 5: Searching a file • TasK: Searching for a file with a name. – Find data files “*.csv” in home directory – Find a file name “poll.csv” • Command: – $ find ~ –name “*.csv” • We Learned: – Search file in a place – Wild card * Exercise 5: Search a file • Find all CSV data files “*.csv” in your scratch space. • Find a file name “ex5”. Note: try not to use “cd” and “ls”. Instead, use “find” Example 6: Distribute files • TasK: PacK files in Part1 into an archive file, compress it and ready for distribuJon • Commands: – tar, – gzip, gunzip, • Learn: – Tar/untar – Compress/uncompress Exercise 6: • TasK: Make directory “Intro2Linux” into an archive file, compress it, copy it to your scratch space, open it. – PacK a tar file – Compress – Copy to scratch space – Open it Example 7: File permission • TasK: Protect a file from the damage of write/copy/mv. • Commands: – chmod • Learn: – Ownership of a file – Change permission Exercise 7: • TasK: Make sure the data files’ permission is right to allow people in the group to read and write. If it is not, then change it. Example 8 :Self learning • TasK: find out what is the command to shuffle lines of a data file “pet_store.csv”, then create a file “pet_shuffled.csv”. • Commands – man, -help, man -K – Google search “how to shuffle lines in a file linux shell”, “random permutaon in linux shell” • Learn: – How to find command – How to get details of a command – Command “shuf” Exercise 8: Seft learning • TasK: find out how to sort lines of a data file, then sort the file “polls.csv” to “polls_sort.csv”. 1. find out if there is a command could sort the file. 2. Try to sort data file “polls.csv”. Note: Do not sort the first line! Agenda • IntroducJon • Linux – Commands • Navigaon • Get or create files • Organizing files • Closer looK into files • Search files • Distribute files • File permission • Learn new commands • Summary From Command to Script Shell script Combine more commands together for more sophisJcate/complicated tasK. • Grammar • Environment and scope • Parameters, expressions, expansion, • Control flow (branch, loops) • ExecuJon (source vs. direct run, ) Examples : Pipeline • TasK: count number of files under /bin. – Soluon 1: ls /bin > bin_list wc bin_list – Soluon 2: ls /bin |wc • Learned: – If a standard output could be used as standard output of next command, a pipeline could be build – Commonly used: grep, wc, sort, less, more, uniq, head, tail,… Examples of pipeline • Filtering output showq |grep BatchHold • SorJng the output ls –l |sort • Counng ls |wc • Beer viewing ps |more Exercise 9: Pipeline • TasK: Find out how many jobs in job queue. Among them, how many are running. 1. Count total jobs in job queue. Use “showq” to get list of jobs in queue. 2. Count only the running jobs. Use Keyword “Running” to filter the jobs Example 10 : Make a command • TasK: Get ready for running jobs. Need to – Copy files to worKspace – Update a worK log file – Make sure it is ready: data files – Make the whole tasK as a command • Commands – $ cp –r BuildModel $SCRATCH – $ date >> $SCRATCH/BuildModel/log – $ ls -l $SCRATCH/BuildModel/ • Learn: – If a tasK could be implemented by more than single line command/pipeline and is needed repeatedly – Environment variable $SCRATCH – >> vs. > – File permission need to set to executable – Add it to $PATH Exercise 10: Make a command • TasK: follow example 10 to make a command “data_on_scratch” Example 11 : environment • TasK: In addiJon to example 10, we would liKe to – Load module – Go to scratch space – Make sure it is ready: module and scratch space • Commands – $ module load PETSc – $ echo $PATH – $ cd $SCRATCH/BuildModel – $ pwd • Learn: – Shell variable scope – Environment variable $PATH, $PWD – Source run vs. direct run Exercise 11: • TasK: 1. Follow example 11 to make your command “run_on_scratch”. Example 12 :expression • TasK: Same as Example 10 except that copy data file only and only when data file is newer. Need to – Copy data files to scratch if newer – Update a worK log file – Make sure it is ready: data files – Make the whole tasK as a command • Commands – $ cp ./Data.csv $SCRATCH/BuildModel/Data.csv – $ date >> $SCRATCH/BuildModel/log – $ ls -l $SCRATCH/BuildModel/Data.csv • Learn: – Expression Exercise 12: expression • TasK: follow example 12 to make a command “data_update_on_scratch” Example 13 : variables • TasK: There are 3 models under develop in BuildModel. Similar as Example 11, but we would liKe to go to parJcular directory when worKing on a model. Use variable to specify the parJcular module to load and directory to go to. – Take two variables • Commands – Similar as example 11 • Learn: – PosiJon variable and its usage. Exercise 13: • TasK: 1. Follow example 13 to make your command “run_model_on_scratch” with variable. Summary • Commands – Cheat sheet for quicK reference • Very simple shell script – Get examples from Internet & HPCC • Class material: Intro2Linux Q & A Tank You!.
Load more

Recommended publications
  • Asp Net Core Request Pipeline
    Asp Net Core Request Pipeline Is Royce always cowled and multilobate when achromatize some wall very instanter and lawlessly? Hobbyless and flustered brazensVladamir regressively? cocainizing her intangibles tiptoed or cluster advertently. Is Patric fuzzed or paintable when cogs some theocracy Or gray may choose to berry the request. Json files etc and community of response body is not inject all low maintenance, we will be same coin. Another through components that asp and cto of. How extensible it crashes, just by one of startup class controller. Do the extended with really very complex scenarios when he enjoys sharing it ever was to delete models first user makes for node. Even add to implement captcha in startup class to the same concept is the reason you want to. Let us a pipeline to any incoming request processing, firefox still create an output? For an app to build a cup of. Razor pages uses handler methods to deal of incoming HTTP request. Ask how the above mentioned in last middleware, the come to tell who are ready simply obsolete at asp and options. Have asp and asp net core request pipeline but will not be mapped to pipeline io threads to work both. The internet and when creating sawdust in snippets or improvements that by one description be a request pipeline branching. Help editing this article, ordinary code inside of hosting infrastructure asp and actions before, we issue was not. The body to deal with minimal footprint to entity framework of needed loans for each middleware will take a docker, that receive criticism.
    [Show full text]
  • Version 7.8-Systemd
    Linux From Scratch Version 7.8-systemd Created by Gerard Beekmans Edited by Douglas R. Reno Linux From Scratch: Version 7.8-systemd by Created by Gerard Beekmans and Edited by Douglas R. Reno Copyright © 1999-2015 Gerard Beekmans Copyright © 1999-2015, Gerard Beekmans All rights reserved. This book is licensed under a Creative Commons License. Computer instructions may be extracted from the book under the MIT License. Linux® is a registered trademark of Linus Torvalds. Linux From Scratch - Version 7.8-systemd Table of Contents Preface .......................................................................................................................................................................... vii i. Foreword ............................................................................................................................................................. vii ii. Audience ............................................................................................................................................................ vii iii. LFS Target Architectures ................................................................................................................................ viii iv. LFS and Standards ............................................................................................................................................ ix v. Rationale for Packages in the Book .................................................................................................................... x vi. Prerequisites
    [Show full text]
  • Stclang: State Thread Composition As a Foundation for Monadic Dataflow Parallelism Sebastian Ertel∗ Justus Adam Norman A
    STCLang: State Thread Composition as a Foundation for Monadic Dataflow Parallelism Sebastian Ertel∗ Justus Adam Norman A. Rink Dresden Research Lab Chair for Compiler Construction Chair for Compiler Construction Huawei Technologies Technische Universität Dresden Technische Universität Dresden Dresden, Germany Dresden, Germany Dresden, Germany [email protected] [email protected] [email protected] Andrés Goens Jeronimo Castrillon Chair for Compiler Construction Chair for Compiler Construction Technische Universität Dresden Technische Universität Dresden Dresden, Germany Dresden, Germany [email protected] [email protected] Abstract using monad-par and LVars to expose parallelism explicitly Dataflow execution models are used to build highly scalable and reach the same level of performance, showing that our parallel systems. A programming model that targets parallel programming model successfully extracts parallelism that dataflow execution must answer the following question: How is present in an algorithm. Further evaluation shows that can parallelism between two dependent nodes in a dataflow smap is expressive enough to implement parallel reductions graph be exploited? This is difficult when the dataflow lan- and our programming model resolves short-comings of the guage or programming model is implemented by a monad, stream-based programming model for current state-of-the- as is common in the functional community, since express- art big data processing systems. ing dependence between nodes by a monadic bind suggests CCS Concepts • Software and its engineering → Func- sequential execution. Even in monadic constructs that explic- tional languages. itly separate state from computation, problems arise due to the need to reason about opaquely defined state.
    [Show full text]
  • A Highly Configurable High-Level Synthesis Functional Pattern Library
    electronics Article A Highly Configurable High-Level Synthesis Functional Pattern Library Lan Huang 1,2,‡, Teng Gao 1,‡, Dalin Li 1,†, Zihao Wang 1 and Kangping Wang 1,2,* 1 College of Computer Science and Technology, Jilin University, Changchun 130012, China; [email protected] (L.H.); [email protected] (T.G.); [email protected] (D.L.); [email protected] (Z.W.) 2 Key Laboratory of Symbolic Computation and Knowledge Engineering, Jilin University, Changchun 130012, China * Correspondence: [email protected] † Current address: Zhuhai Laboratory of Key Laboratory of Symbol Computation and Knowledge Engineering of Ministry of Education, Department of Computer Science and Technology, Zhuhai College of Jilin University, Zhuhai 519041, China. ‡ These authors contributed equally to this work. Abstract: FPGA has recently played an increasingly important role in heterogeneous computing, but Register Transfer Level design flows are not only inefficient in design, but also require designers to be familiar with the circuit architecture. High-level synthesis (HLS) allows developers to design FPGA circuits more efficiently with a more familiar programming language, a higher level of abstraction, and automatic adaptation of timing constraints. When using HLS tools, such as Xilinx Vivado HLS, specific design patterns and techniques are required in order to create high-performance circuits. Moreover, designing efficient concurrency and data flow structures requires a deep understanding of the hardware, imposing more learning costs on programmers. In this paper, we propose a set of functional patterns libraries based on the MapReduce model, implemented by C++ templates, Citation: Huang, L.; Gao,T.; Li, D.; which can quickly implement high-performance parallel pipelined computing models on FPGA with Wang, Z.; Wang, K.
    [Show full text]
  • Other Useful Commands
    Bioinformatics 101 – Lecture 2 Introduction to command line Alberto Riva ([email protected]), J. Lucas Boatwright ([email protected]) ICBR Bioinformatics Core Computing environments ▪ Standalone application – for local, interactive use; ▪ Command-line – local or remote, interactive use; ▪ Cluster oriented: remote, not interactive, highly parallelizable. Command-line basics ▪ Commands are typed at a prompt. The program that reads your commands and executes them is the shell. ▪ Interaction style originated in the 70s, with the first visual terminals (connections were slow…). ▪ A command consists of a program name followed by options and/or arguments. ▪ Syntax may be obscure and inconsistent (but efficient!). Command-line basics ▪ Example: to view the names of files in the current directory, use the “ls” command (short for “list”) ls plain list ls –l long format (size, permissions, etc) ls –l –t sort newest to oldest ls –l –t –r reverse sort (oldest to newest) ls –lrt options can be combined (in this case) ▪ Command names and options are case sensitive! File System ▪ Unix systems are centered on the file system. Huge tree of directories and subdirectories containing all files. ▪ Everything is a file. Unix provides a lot of commands to operate on files. ▪ File extensions are not necessary, and are not recognized by the system (but may still be useful). ▪ Please do not put spaces in filenames! Permissions ▪ Different privileges and permissions apply to different areas of the filesystem. ▪ Every file has an owner and a group. A user may belong to more than one group. ▪ Permissions specify read, write, and execute privileges for the owner, the group, everyone else.
    [Show full text]
  • Let's Get Functional
    5 LET’S GET FUNCTIONAL I’ve mentioned several times that F# is a functional language, but as you’ve learned from previous chapters you can build rich applications in F# without using any functional techniques. Does that mean that F# isn’t really a functional language? No. F# is a general-purpose, multi paradigm language that allows you to program in the style most suited to your task. It is considered a functional-first lan- guage, meaning that its constructs encourage a functional style. In other words, when developing in F# you should favor functional approaches whenever possible and switch to other styles as appropriate. In this chapter, we’ll see what functional programming really is and how functions in F# differ from those in other languages. Once we’ve estab- lished that foundation, we’ll explore several data types commonly used with functional programming and take a brief side trip into lazy evaluation. The Book of F# © 2014 by Dave Fancher What Is Functional Programming? Functional programming takes a fundamentally different approach toward developing software than object-oriented programming. While object-oriented programming is primarily concerned with managing an ever-changing system state, functional programming emphasizes immutability and the application of deterministic functions. This difference drastically changes the way you build software, because in object-oriented programming you’re mostly concerned with defining classes (or structs), whereas in functional programming your focus is on defining functions with particular emphasis on their input and output. F# is an impure functional language where data is immutable by default, though you can still define mutable data or cause other side effects in your functions.
    [Show full text]
  • Bash Guide for Beginners
    Bash Guide for Beginners Machtelt Garrels Garrels BVBA <tille wants no spam _at_ garrels dot be> Version 1.11 Last updated 20081227 Edition Bash Guide for Beginners Table of Contents Introduction.........................................................................................................................................................1 1. Why this guide?...................................................................................................................................1 2. Who should read this book?.................................................................................................................1 3. New versions, translations and availability.........................................................................................2 4. Revision History..................................................................................................................................2 5. Contributions.......................................................................................................................................3 6. Feedback..............................................................................................................................................3 7. Copyright information.........................................................................................................................3 8. What do you need?...............................................................................................................................4 9. Conventions used in this
    [Show full text]
  • BIOVIA Pipeline Pilot System Requirements
    SYSTEM REQUIREMENTS PIPELINE PILOT 2020 Copyright Notice ©2019 Dassault Systèmes. All rights reserved. 3DEXPERIENCE, the Compass icon and the 3DS logo, CATIA, SOLIDWORKS, ENOVIA, DELMIA, SIMULIA, GEOVIA, EXALEAD, 3DVIA, 3DSWYM, BIOVIA, NETVIBES, IFWE and 3DEXCITE, are commercial trademarks or registered trademarks of Dassault Systèmes, a French "société européenne" (Versailles Commercial Register # B 322 306 440), or its subsidiaries in the U.S. and/or other countries. All other trademarks are owned by their respective owners. Use of any Dassault Systèmes or its subsidiaries trademarks is subject to their express written approval. Acknowledgments and References To print photographs or files of computational results (figures and/or data) obtained by using Dassault Systèmes software, acknowledge the source in an appropriate format. For example: "Computational results were obtained by using Dassault Systèmes BIOVIA software programs. Pipeline Pilot Server was used to perform the calculations and to generate the graphical results." Dassault Systèmes may grant permission to republish or reprint its copyrighted materials. Requests should be submitted to Dassault Systèmes Customer Support, either by visiting https://www.3ds.com/support/ and clicking Call us or Submit a request, or by writing to: Dassault Systèmes Customer Support 10, Rue Marcel Dassault 78140 Vélizy-Villacoublay FRANCE Contents About This Document 1 Definitions 1 Additional Information 1 Dassault Systèmes Support Resources 1 Pipeline Pilot Server Requirements 2 Minimum Hardware
    [Show full text]
  • PJM Command Line Interface
    PJM Command Line Interface PJM Interconnection LLC Version 1.5.1 11-18-2020 PJM Command Line Interface Table of Contents Purpose ..................................................................................................................................................................................... 4 System Requirements ............................................................................................................................................................... 4 Release History ......................................................................................................................................................................... 4 Usage ........................................................................................................................................................................................ 5 Standalone Application ......................................................................................................................................................... 5 Example Standalone Execution ....................................................................................................................................... 5 Parameter Details ............................................................................................................................................................. 7 Password Encryption .......................................................................................................................................................
    [Show full text]
  • TASSEL 3.0 / 4.0 Pipeline Command Line Interface: Guide to Using Tassel Pipeline
    TASSEL 3.0 / 4.0 Pipeline Command Line Interface: Guide to using Tassel Pipeline Terry Casstevens ([email protected]) Institute for Genomic Diversity, Cornell University, Ithaca, NY 14853-2703 March 28, 2014 Prerequisites ............................................................................................................................................................ 1 Source Code ............................................................................................................................................................ 1 Install ....................................................................................................................................................................... 1 Execute .................................................................................................................................................................... 1 Increasing Heap Size ............................................................................................................................................... 2 Examples ................................................................................................................................................................. 2 Examples (XML Configuration Files) .................................................................................................................... 2 Usage ......................................................................................................................................................................
    [Show full text]
  • Fast Linux I/O in the Unix Tradition
    — preprint only: final version will appear in OSR, July 2008 — PipesFS: Fast Linux I/O in the Unix Tradition Willem de Bruijn Herbert Bos Vrije Universiteit Amsterdam Vrije Universiteit Amsterdam and NICTA [email protected] [email protected] ABSTRACT ory wall” [26]). To improve throughput, it is now essential This paper presents PipesFS, an I/O architecture for Linux to avoid all unnecessary memory operations. Inefficient I/O 2.6 that increases I/O throughput and adds support for het- primitives exacerbate the effects of the memory wall by in- erogeneous parallel processors by (1) collapsing many I/O curring unnecessary copying and context switching, and as interfaces onto one: the Unix pipeline, (2) increasing pipe a result of these cache misses. efficiency and (3) exploiting pipeline modularity to spread computation across all available processors. Contribution. We revisit the Unix pipeline as a generic model for streaming I/O, but modify it to reduce overhead, PipesFS extends the pipeline model to kernel I/O and com- extend it to integrate kernel processing and complement it municates with applications through a Linux virtual filesys- with support for anycore execution. We link kernel and tem (VFS), where directory nodes represent operations and userspace processing through a virtual filesystem, PipesFS, pipe nodes export live kernel data. Users can thus interact that presents kernel operations as directories and live data as with kernel I/O through existing calls like mkdir, tools like pipes. This solution avoids new interfaces and so unlocks all grep, most languages and even shell scripts.
    [Show full text]
  • Making a Faster Curry with Extensional Types
    Making a Faster Curry with Extensional Types Paul Downen Simon Peyton Jones Zachary Sullivan Microsoft Research Zena M. Ariola Cambridge, UK University of Oregon [email protected] Eugene, Oregon, USA [email protected] [email protected] [email protected] Abstract 1 Introduction Curried functions apparently take one argument at a time, Consider these two function definitions: which is slow. So optimizing compilers for higher-order lan- guages invariably have some mechanism for working around f1 = λx: let z = h x x in λy:e y z currying by passing several arguments at once, as many as f = λx:λy: let z = h x x in e y z the function can handle, which is known as its arity. But 2 such mechanisms are often ad-hoc, and do not work at all in higher-order functions. We show how extensional, call- It is highly desirable for an optimizing compiler to η ex- by-name functions have the correct behavior for directly pand f1 into f2. The function f1 takes only a single argu- expressing the arity of curried functions. And these exten- ment before returning a heap-allocated function closure; sional functions can stand side-by-side with functions native then that closure must subsequently be called by passing the to practical programming languages, which do not use call- second argument. In contrast, f2 can take both arguments by-name evaluation. Integrating call-by-name with other at once, without constructing an intermediate closure, and evaluation strategies in the same intermediate language ex- this can make a huge difference to run-time performance in presses the arity of a function in its type and gives a princi- practice [Marlow and Peyton Jones 2004].
    [Show full text]