DOCSLIB.ORG

File Layout and Directories

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
File Layout and Directories 11/8/17 Where Are We? • Covered: COS 318: Operating Systems "! Management of CPU & concurrency "! Management of main memory & virtual memory "! Management of I/O devices File Structure •! Currently --- File Systems "! This lecture: File Structure Jaswinder Pal Singh Computer Science Department •! Then: Princeton University •! Naming and directories (http://www.cs.princeton.edu/courses/cos318/) •! Efficiency and performance •! Reliability and protection The File System Abstraction File System •! Open, close, read, write … named files, arranged in folders or directories ◆! Naming "! File name and directory File naming Physical Reality File System Abstraction ◆! File access File access "! Read, write, other operations block oriented byte oriented (char stream) Buffer cache ◆! Buffer cache "! Reduce client/server disk I/Os Management physical sector #’s named files Disk allocation ◆! Disk allocation Disk Drivers no protection users protected from ◆! Layout, mapping files to blocks each other ◆! Security, protection, reliability, durability data might be corrupted robust to machine failures ◆! Management tools if machine crashes 4 1 11/8/17 Topics Typical File Attributes ◆! File system structure •! Name ◆! Disk allocation and i-nodes •! Type – needed for systems that support different types ◆! Directory and link implementations •! Location – pointer to file location on device. ◆! Physical layout for performance •! Size – current file size. •! Protection – controls who can read, write, execute •! Time, date, and user identification – data for protection, security, and usage monitoring •! Information about files are kept in the directory structure, which is maintained on the disk 2 Typical Layout of a Disk Partition File Types – Name, Extension ◆! Boot block Boot block "! Code to load and boot OS File Type Usual extension Function Executable exe, com, bin or ready-to-run machine- ◆! Super-block defines a file system Superblock none language program "! File system info: type, no of blocks, ... Object obj, o complied, machine language, not linked "! File metadata File metadata area Source code c, p, pas, 177, source code in various (i-nodes in Unix) "! Information about / ptr to free blocks asm, a languages Batch bat, sh commands to the "! Location of descriptor of root directory Directory data command interpreter ◆! File metadata Text txt, doc textual data doc uments "! Each descriptor describes a file Word processor wp, tex, rrf, etc. various word-processor formats ◆! Directories File data Library lib, a libraries of routines "! Directory data (directory and file names) Print or view ps, dvi, gif ASCII or binary file ◆! File data Archive arc, zip, tar related files grouped "! Data blocks into one file, sometimes compressed. 3 2 11/8/17 Typical File Operations Open A File: Open(fd, name, access) ◆! Various checking (directory and file name lookup, authenticate) •! Create ◆! Copy the file descriptors into the in-memory data structure •! Write ◆! Create an entry in the open file table (system wide) •! Read ◆! Create an entry in PCB File system •! Reposition within file – file seek ◆! Return user a pointer to “file descriptor” info Process File •! Delete control Open-file table descriptors File block (system-wide) (Metadata) •! Truncate metadata •! Open(Fi) – search the directory structure on disk for Directories entry Fi, and move the content of entry to memory. •! Close (Fi) – move the content of entry Fi in memory to Open directory structure on disk. file File data pointer . array . 5 Translating from user to system view File system design constraints •! User wants to read 10 bytes from file starting at byte 2? •! For small files: "! Seek byte 2, fetch the block, read 10 bytes "! Small blocks for storage efficiency "! Files used together should be stored together •! User wants to write 10 bytes to file starting at byte 2? "! Seek byte 2, fetch the block, write 10 bytes, write out block •! For large files: "! Contiguous allocation for sequential access •! Everything inside file system is in whole size blocks "! Efficient lookup for random access "! Even getc and putc buffers 4096 bytes •! May not know at file creation whether file will become •! From now on, file is collection of blocks. small or large 3 11/8/17 File usage patterns File system design •! How do users access files? •! Data structures "! Sequential: bytes read in order "! Directories: file name -> file metadata "! Random: read/write element out of middle of arrays •! Store directories as files "! Content-based access: find me next byte starting with “COS318” "! File metadata: used to find file data blocks •! How are files used? "! Free map: list of free disk blocks "! Most files are small "! Large files use up most of the disk space •! How do we organize these data structures? "! Most transfers are small "! Large files account for most of the bytes transferred •! Bad news "! Need everything to be efficient Data Structures for Storage Allocation Data structures for disk management •! A file header for each file (part of the file meta-data) ◆! A File "! Disk sectors associated with each file "! Metadata "! A list of data blocks •! A data structure to track free space on disk ◆! Free space data structure "! Bit map "! Bit map indicating the status of 11111111111111111000000000000000 •! 1 bit per block (sector) disk blocks 00000111111110000000000111111111 … •! blocks numbered in cylinder-major order, why? "! Linked list that chains free "! blocks together 11000001111000111100000000000000 Linked list "! Others? "! Buddy system Free "! … link •! What about allocation for the blocks associated with a file? •! Let’s look at some ways of keeping addr link … track of file data size addr size 6 4 11/8/17 Contiguous Allocation Linked Files ◆! Allocate contiguous blocks on storage ◆! File structure (Alto) File header "! Bitmap: find N contiguous 0’s 3 "! File metadata points to 1st block on storage "! Linked list: find a region (size >= N) "! A block points to the next ◆! File metadata "! Last block has a NULL pointer "! First block in file ◆! Pros "! Number of blocks . "! Can grow files dynamically ◆! Pros "! File data tracked similarly to free "! Fast sequential access list of blocks "! Easy random access File A File B "! Doesn’t waste space ◆! Cons ◆! Cons null "! External fragmentation "! Random access: bad (what if file C needs 4 blocks) "! Unreliable: losing a block means "! Hard to grow files losing the rest 7 8 Linked files (cont’d) File Allocation Table (FAT) •! Idea is to keep the linked list metadata foo 217 (pointers) in memory, rather than on disk •! Allocation table at beginning of each volume 0 ◆! N entries for N blocks ◆! Want to keep it in memory •! File structure (MS-DOS) 217 619 "! A file is a linked list of blocks "! File metadata points to first block of file 399 "! The entry of first block points to next, … EOF •! Pros "! Simple 619 399 •! Cons "! Random access: still not good "! Wastes space - table for each file expensive to keep in memory FAT Allocation Table 8 5 11/8/17 DEMOS (Cray-1) Single-level Indexed File File metadata ◆! Idea •! User declares max size size0 "! Try contiguous allocation size1 •! File header holds array of pointers "! Allow non-contiguous Disk size to disk blocks 9 File header blocks ◆! File structure "! Small file metadata has 10 (base,size) pointers •! Pros: "! Big file has 10 indirect pointers "! Can grow up to a limit ◆! Pros & Cons size0 "! Random access is fast size "! Can grow (max 10GB) 1 "! No external fragmentation "! Fragmentation size9 •! Cons: size 0 "! Clumsy to grow beyond limit size1 size0 size1 "! Still lots of seeks size9 size9 11 Single-level indexed files (cont’d) Multi-level Indexed Files ! outer-index index table file 6 11/8/17 Hybrid Multi-level Indexed Files (Unix) Original Unix i-node ◆! 13 Pointers in a header ◆! Mode: file type, protection bits, setuid, setgid bits "! 10 direct pointers data ◆! Link count: no. of directory entries pointing to this file "! 11: 1-level indirect ◆! Uid: uid of the file owner "! 12: 2-level indirect data ◆! Gid: gid of the file owner "! 13: 3-level indirect 1 .2 . ◆! File size ◆! Pros & Cons 11 . 12 . data ◆! Times (access, modify, change) "! In favor of small files 13 "! Can grow . "! Limit is 16G . data ◆! 10 pointers to data blocks "! Can have lots of seeking ◆! Single indirect pointer . ◆! Double indirect pointer . data ◆! Triple indirect pointer 12 13 Extents Naming Files ◆! An extent is a variable number of Can name files via: blocks Block offset ◆! Main idea length ◆! Index (i-node number): Not easy for users to specify "! A file is a number of extents Starting block ◆! Text name: Need to map it to index "! XFS uses 8Kbyte blocks ◆! "! Max extent size is 2M blocks Icon: Need to map it to index or to text and then to index ◆! Index nodes need to have . "! Block offset ◆! Directories "! Length ◆! Table of file name, file index pairs "! Starting block ◆! Map name to file index (where to find the header) •! Microsoft NTFS, Linux EXT4, … ◆! Pros: little metadata, fast seq ◆! A directory is itself stored as a file access, can grow over time, less fragmentation ◆! Cons: external fragmentation still problem 14 15 7 11/8/17 Naming Tricks Directory Organization Examples •! Bootstrapping: Where do you start looking? ◆! Flat (CP/M) "! Root directory "! All files are in one directory "! inode #2 on the system ◆! Hierarchical (Unix) "! 0 and 1 used for other purposes "! /u/cos318/foo •! Special names: "! Directory is stored in a file containing (name, i-node) pairs "! Root directory:
Load more

Recommended publications
  • The Microsoft Compound Document File Format"
    OpenOffice.org's Documentation of the Microsoft Compound Document File Format Author Daniel Rentz ✉ mailto:[email protected] http://sc.openoffice.org License Public Documentation License Contributors Other sources Hyperlinks to Wikipedia ( http://www.wikipedia.org) for various extended information Mailing list ✉ mailto:[email protected] Subscription ✉ mailto:[email protected] Download PDF http://sc.openoffice.org/compdocfileformat.pdf XML http://sc.openoffice.org/compdocfileformat.odt Project started 2004-Aug-30 Last change 2007-Aug-07 Revision 1.5 Contents 1 Introduction ......................................................................................................... 3 1.1 License Notices 3 1.2 Abstract 3 1.3 Used Terms, Symbols, and Formatting 4 2 Storages and Streams ........................................................................................... 5 3 Sectors and Sector Chains ................................................................................... 6 3.1 Sectors and Sector Identifiers 6 3.2 Sector Chains and SecID Chains 7 4 Compound Document Header ............................................................................. 8 4.1 Compound Document Header Contents 8 4.2 Byte Order 9 4.3 Sector File Offsets 9 5 Sector Allocation ............................................................................................... 10 5.1 Master Sector Allocation Table 10 5.2 Sector Allocation Table 11 6 Short-Streams ...................................................................................................
    [Show full text]
  • Mac OS X Server Administrator's Guide
    034-9285.S4AdminPDF 6/27/02 2:07 PM Page 1 Mac OS X Server Administrator’s Guide K Apple Computer, Inc. © 2002 Apple Computer, Inc. All rights reserved. Under the copyright laws, this publication may not be copied, in whole or in part, without the written consent of Apple. The Apple logo is a trademark of Apple Computer, Inc., registered in the U.S. and other countries. Use of the “keyboard” Apple logo (Option-Shift-K) for commercial purposes without the prior written consent of Apple may constitute trademark infringement and unfair competition in violation of federal and state laws. Apple, the Apple logo, AppleScript, AppleShare, AppleTalk, ColorSync, FireWire, Keychain, Mac, Macintosh, Power Macintosh, QuickTime, Sherlock, and WebObjects are trademarks of Apple Computer, Inc., registered in the U.S. and other countries. AirPort, Extensions Manager, Finder, iMac, and Power Mac are trademarks of Apple Computer, Inc. Adobe and PostScript are trademarks of Adobe Systems Incorporated. Java and all Java-based trademarks and logos are trademarks or registered trademarks of Sun Microsystems, Inc. in the U.S. and other countries. Netscape Navigator is a trademark of Netscape Communications Corporation. RealAudio is a trademark of Progressive Networks, Inc. © 1995–2001 The Apache Group. All rights reserved. UNIX is a registered trademark in the United States and other countries, licensed exclusively through X/Open Company, Ltd. 062-9285/7-26-02 LL9285.Book Page 3 Tuesday, June 25, 2002 3:59 PM Contents Preface How to Use This Guide 39 What’s Included
    [Show full text]
  • Filesystem Hierarchy Standard
    Filesystem Hierarchy Standard LSB Workgroup, The Linux Foundation Filesystem Hierarchy Standard LSB Workgroup, The Linux Foundation Version 3.0 Publication date March 19, 2015 Copyright © 2015 The Linux Foundation Copyright © 1994-2004 Daniel Quinlan Copyright © 2001-2004 Paul 'Rusty' Russell Copyright © 2003-2004 Christopher Yeoh Abstract This standard consists of a set of requirements and guidelines for file and directory placement under UNIX-like operating systems. The guidelines are intended to support interoperability of applications, system administration tools, development tools, and scripts as well as greater uniformity of documentation for these systems. All trademarks and copyrights are owned by their owners, unless specifically noted otherwise. Use of a term in this document should not be regarded as affecting the validity of any trademark or service mark. Permission is granted to make and distribute verbatim copies of this standard provided the copyright and this permission notice are preserved on all copies. Permission is granted to copy and distribute modified versions of this standard under the conditions for verbatim copying, provided also that the title page is labeled as modified including a reference to the original standard, provided that information on retrieving the original standard is included, and provided that the entire resulting derived work is distributed under the terms of a permission notice identical to this one. Permission is granted to copy and distribute translations of this standard into another language, under the above conditions for modified versions, except that this permission notice may be stated in a translation approved by the copyright holder. Dedication This release is dedicated to the memory of Christopher Yeoh, a long-time friend and colleague, and one of the original editors of the FHS.
    [Show full text]
  • What Is UNIX? the Directory Structure Basic Commands Find
    What is UNIX? UNIX is an operating system like Windows on our computers. By operating system, we mean the suite of programs which make the computer work. It is a stable, multi-user, multi-tasking system for servers, desktops and laptops. The Directory Structure All the files are grouped together in the directory structure. The file-system is arranged in a hierarchical structure, like an inverted tree. The top of the hierarchy is traditionally called root (written as a slash / ) Basic commands When you first login, your current working directory is your home directory. In UNIX (.) means the current directory and (..) means the parent of the current directory. find command The find command is used to locate files on a Unix or Linux system. find will search any set of directories you specify for files that match the supplied search criteria. The syntax looks like this: find where-to-look criteria what-to-do All arguments to find are optional, and there are defaults for all parts. where-to-look defaults to . (that is, the current working directory), criteria defaults to none (that is, select all files), and what-to-do (known as the find action) defaults to ‑print (that is, display the names of found files to standard output). Examples: find . –name *.txt (finds all the files ending with txt in current directory and subdirectories) find . -mtime 1 (find all the files modified exact 1 day) find . -mtime -1 (find all the files modified less than 1 day) find . -mtime +1 (find all the files modified more than 1 day) find .
    [Show full text]
  • Lecture 17: Files and Directories
    11/1/16 CS 422/522 Design & Implementation of Operating Systems Lecture 17: Files and Directories Zhong Shao Dept. of Computer Science Yale University Acknowledgement: some slides are taken from previous versions of the CS422/522 lectures taught by Prof. Bryan Ford and Dr. David Wolinsky, and also from the official set of slides accompanying the OSPP textbook by Anderson and Dahlin. The big picture ◆ Lectures before the fall break: – Management of CPU & concurrency – Management of main memory & virtual memory ◆ Current topics --- “Management of I/O devices” – Last week: I/O devices & device drivers – Last week: storage devices – This week: file systems * File system structure * Naming and directories * Efficiency and performance * Reliability and protection 1 11/1/16 This lecture ◆ Implementing file system abstraction Physical Reality File System Abstraction block oriented byte oriented physical sector #’s named files no protection users protected from each other data might be corrupted robust to machine failures if machine crashes File system components ◆ Disk management User – Arrange collection of disk blocks into files File File ◆ Naming Naming access – User gives file name, not track or sector number, to locate data Disk management ◆ Security / protection – Keep information secure Disk drivers ◆ Reliability/durability – When system crashes, lose stuff in memory, but want files to be durable 2 11/1/16 User vs. system view of a file ◆ User’s view – Durable data structures ◆ System’s view (system call interface) – Collection of bytes (Unix) ◆ System’s view (inside OS): – Collection of blocks – A block is a logical transfer unit, while a sector is the physical transfer unit.
    [Show full text]
  • File Systems
    File Systems Profs. Bracy and Van Renesse based on slides by Prof. Sirer Storing Information • Applications could store information in the process address space • Why is this a bad idea? – Size is limited to size of virtual address space – The data is lost when the application terminates • Even when computer doesn’t crash! – Multiple process might want to access the same data File Systems • 3 criteria for long-term information storage: 1. Able to store very large amount of information 2. Information must survive the processes using it 3. Provide concurrent access to multiple processes • Solution: – Store information on disks in units called files – Files are persistent, only owner can delete it – Files are managed by the OS File Systems: How the OS manages files! File Naming • Motivation: Files abstract information stored on disk – You do not need to remember block, sector, … – We have human readable names • How does it work? – Process creates a file, and gives it a name • Other processes can access the file by that name – Naming conventions are OS dependent • Usually names as long as 255 characters is allowed • Windows names not case sensitive, UNIX family is File Extensions • Name divided into 2 parts: Name+Extension • On UNIX, extensions are not enforced by OS – Some applications might insist upon them • Think: .c, .h, .o, .s, etc. for C compiler • Windows attaches meaning to extensions – Tries to associate applications to file extensions File Access • Sequential access – read all bytes/records from the beginning – particularly convenient for magnetic tape • Random access – bytes/records read in any order – essential for database systems File Attributes • File-specific info maintained by the OS – File size, modification date, creation time, etc.
    [Show full text]
  • File Systems
    “runall” 2002/9/24 page 305 CHAPTER 10 File Systems 10.1 BASIC FUNCTIONS OF FILE MANAGEMENT 10.2 HIERARCHICAL MODEL OF A FILE SYSTEM 10.3 THE USER’S VIEW OF FILES 10.4 FILE DIRECTORIES 10.5 BASIC FILE SYSTEM 10.6 DEVICE ORGANIZATION METHODS 10.7 PRINCIPLES OF DISTRIBUTED FILE SYSTEMS 10.8 IMPLEMENTING DISTRIBUTED FILE SYSTEM Given that main memory is volatile, i.e., does not retain information when power is turned off, and is also limited in size, any computer system must be equipped with secondary memory on which the user and the system may keep information for indefinite periods of time. By far the most popular secondary memory devices are disks for random access purposes and magnetic tapes for sequential, archival storage. Since these devices are very complex to interact with, and, in multiuser systems are shared among different users, operating systems (OS) provide extensive services for managing data on secondary memory. These data are organized into files, which are collections of data elements grouped together for the purposes of access control, retrieval, and modification. A file system is the part of the operating system that is responsible for managing files and the resources on which these reside. Without a file system, efficient computing would essentially be impossible. This chapter discusses the organization of file systems and the tasks performed by the different components. The first part is concerned with general user and implementation aspects of file management emphasizing centralized systems; the last sections consider extensions and methods for distributed systems. 10.1 BASIC FUNCTIONS OF FILE MANAGEMENT The file system, in collaboration with the I/O system, has the following three basic functions: 1.
    [Show full text]
  • A Directory Structure for TEX Files TUG Working Group on a TEX Directory Structure (TWG-TDS) Version 1.1 June 23, 2004
    A Directory Structure for TEX Files TUG Working Group on a TEX Directory Structure (TWG-TDS) version 1.1 June 23, 2004 Copyright c 1994, 1995, 1996, 1997, 1998, 1999, 2003, 2004 TEX Users Group. Permission to use, copy, and distribute this document without modification for any purpose and without fee is hereby granted, provided that this notice appears in all copies. It is provided “as is” without expressed or implied warranty. Permission is granted to copy and distribute modified versions of this document under the condi- tions for verbatim copying, provided that the modifications are clearly marked and the document is not represented as the official one. This document is available on any CTAN host (see Appendix D). Please send questions or suggestions by email to [email protected]. We welcome all comments. This is version 1.1. Contents 1 Introduction 2 1.1 History . 2 1.2 The role of the TDS ................................... 2 1.3 Conventions . 3 2 General 3 2.1 Subdirectory searching . 3 2.2 Rooting the tree . 4 2.3 Local additions . 4 2.4 Duplicate filenames . 5 3 Top-level directories 5 3.1 Macros . 6 3.2 Fonts............................................ 8 3.3 Non-font METAFONT files................................ 10 3.4 METAPOST ........................................ 10 3.5 BIBTEX .......................................... 11 3.6 Scripts . 11 3.7 Documentation . 12 4 Summary 13 4.1 Documentation tree summary . 14 A Unspecified pieces 15 A.1 Portable filenames . 15 B Implementation issues 16 B.1 Adoption of the TDS ................................... 16 B.2 More on subdirectory searching . 17 B.3 Example implementation-specific trees .
    [Show full text]
  • Answers to Even-Numbered Exercises
    4 Answers to Even-numbered Exercises 1. 2. List the commands you can use to perform these operations: a. Make your home directory the working directory b. Identify the working directory a. cd; b. pwd 3. 4. The df utility displays all mounted filesystems along with information about each. Use the df utility with the –h (human-readable) option to answer the following questions. $ df -h Filesystem Size Used Avail Capacity Mounted on /dev/disk2s10 20G 2.6G 17G 13% / devfs 114K 114K 0B 100% /dev fdesc 1.0K 1.0K 0B 100% /dev <volfs> 512K 512K 0B 100% /.vol /dev/disk0s9 77G 37G 39G 49% /Volumes/Scratch /dev/disk1s9 25G 16G 9.5G 63% /Volumes/Sys /dev/disk2s12 94G 43M 94G 0% /Volumes/New /dev/disk1s10 86G 71G 15G 83% /Volumes/Home automount -nsl [223] 0B 0B 0B 100% /Network automount -fstab [232] 0B 0B 0B 100% /automount/Servers automount -static [232] 0B 0B 0B 100% /automount/static a. How many filesystems are mounted on your Mac OS X system? b. Which filesystem stores your home directory? c. Assuming that your answer to exercise 4a is two or more, attempt to create a hard link to a file on another filesystem. What error message do you get? What happens when you attempt to create a symbolic link to the file instead? 1 2 Answers to Even-numbered Exercises Following are sample answers to these questions. Your answers will be different because your filesystem is different. a. five; b. /dev/disk2s10; c. ln: xxx: Cross-device link. No problem creating a cross-device symbolic link.
    [Show full text]
  • Apple File System Reference
    Apple File System Reference Developer Contents About Apple File System 7 General-Purpose Types 9 paddr_t .................................................. 9 prange_t ................................................. 9 uuid_t ................................................... 9 Objects 10 obj_phys_t ................................................ 10 Supporting Data Types ........................................... 11 Object Identifier Constants ......................................... 12 Object Type Masks ............................................. 13 Object Types ................................................ 14 Object Type Flags .............................................. 20 EFI Jumpstart 22 Booting from an Apple File System Partition ................................. 22 nx_efi_jumpstart_t ........................................... 24 Partition UUIDs ............................................... 25 Container 26 Mounting an Apple File System Partition ................................... 26 nx_superblock_t ............................................. 27 Container Flags ............................................... 36 Optional Container Feature Flags ...................................... 37 Read-Only Compatible Container Feature Flags ............................... 38 Incompatible Container Feature Flags .................................... 38 Block and Container Sizes .......................................... 39 nx_counter_id_t ............................................. 39 checkpoint_mapping_t ........................................
    [Show full text]
  • Unix Command Line; Editors Tools for Reproducible Research
    Unix command line; editors Tools for Reproducible Research Karl Broman Biostatistics & Medical Informatics, UW–Madison kbroman.org github.com/kbroman @kwbroman Course web: kbroman.org/Tools4RR My goal in this lecture is to convince you that (a) command-line-based tools are the things to focus on, (b) you need to choose a powerful, universal text editor (you’ll use it a lot), (c) you want to be comfortable and skilled with each. For your work to be reproducible, it needs to be code-based; don’t touch that mouse! Windows vs. Mac OSX vs. Linux Remote vs. Not 2 The Windows operating system is not very programmer-friendly. Mac OSX isn’t either, but under the hood, it’s just unix. Don’t touch the mouse! Open a terminal window and start typing. I do most of my work directly on my desktop or laptop. You might prefer to work remotely on a server, instead. But I can’t stand having any lag in looking at graphics. If you use Windows... Consider Git Bash (or Cygwin) (and perhaps Mintty or ComEmu) 3 Cygwin is an effort to get Unix command-line tools in Windows. Git Bash combines git (for version control) and bash (the unix shell); it’s simpler to deal with than Cygwin. Mintty and ConEmu are terminal emulators. If you use a Mac... Consider Homebrew and iTerm2 Also the XCode command line tools 4 Homebrew is a packaging system; iTerm2 is a Terminal replacement. The XCode command line tools are a must for most unixy things on a Mac.
    [Show full text]
  • Introduction to High Performance Computing at Case Western Reserve University ! HPC Access & Linux [email protected]
    Introduction to High Performance Computing at Case Western Reserve University ! HPC Access & Linux [email protected] KSL Data Center Working on the Cluster • The command line: How does it work? • Connecting to the cluster • Files & directories • Navigating • Creating, Editing, Moving/Renaming • The file structure: What & How? • The session environment • Variables • Modules & Hierarchies Why do we learn to use the shell? • Repeating a workflow can be labour intensive • Allows to automate repetitive tasks • Capture small data manipulation steps to make research reproducible • Manual manipulation of data files: • is often not captured in documentation • is hard to reproduce; troubleshoot; review; or improve • The Shell allows… • Workflows can be automated through the use of shell scripts • Focus on easy string manipulation (e.g. sort, grep, etc.) • Every step can be captured, enhancing reproducibility and troubleshooting What does the “Shell” Do? • The shell is a process providing the CLI • The heart of a cli is a read-evaluate-print loop, or REPL: you type a command, press Enter, the shell: • Reads it • Executes (or “evaluates” it) • Prints the output • Then restores the prompt, waits for you to enter another command Linux Shell Concepts • path/absolute path: specify the location of a file/dir • current working directory: applies in a shell session • filename extension: a label, indicates data type • home directory: ‘privileged’ location, easy to reference • Shell script: interpreted program to automate tasks • Standard Input: The input
    [Show full text]