Unix / Linux File System Basics
Total Page:16
File Type:pdf, Size:1020Kb
Load more
Recommended publications
-
Copy on Write Based File Systems Performance Analysis and Implementation
Copy On Write Based File Systems Performance Analysis And Implementation Sakis Kasampalis Kongens Lyngby 2010 IMM-MSC-2010-63 Technical University of Denmark Department Of Informatics Building 321, DK-2800 Kongens Lyngby, Denmark Phone +45 45253351, Fax +45 45882673 [email protected] www.imm.dtu.dk Abstract In this work I am focusing on Copy On Write based file systems. Copy On Write is used on modern file systems for providing (1) metadata and data consistency using transactional semantics, (2) cheap and instant backups using snapshots and clones. This thesis is divided into two main parts. The first part focuses on the design and performance of Copy On Write based file systems. Recent efforts aiming at creating a Copy On Write based file system are ZFS, Btrfs, ext3cow, Hammer, and LLFS. My work focuses only on ZFS and Btrfs, since they support the most advanced features. The main goals of ZFS and Btrfs are to offer a scalable, fault tolerant, and easy to administrate file system. I evaluate the performance and scalability of ZFS and Btrfs. The evaluation includes studying their design and testing their performance and scalability against a set of recommended file system benchmarks. Most computers are already based on multi-core and multiple processor architec- tures. Because of that, the need for using concurrent programming models has increased. Transactions can be very helpful for supporting concurrent program- ming models, which ensure that system updates are consistent. Unfortunately, the majority of operating systems and file systems either do not support trans- actions at all, or they simply do not expose them to the users. -
CST8207 – Linux O/S I
Mounting a Filesystem Directory Structure Fstab Mount command CST8207 - Algonquin College 2 Chapter 12: page 467 - 496 CST8207 - Algonquin College 3 The mount utility connects filesystems to the Linux directory hierarchy. The mount point is a directory in the local filesystem where you can access mounted filesystem. This directory must exist before you can mount a filesystem. All filesystems visible on the system exist as a mounted filesystem someplace below the root (/) directory CST8207 - Algonquin College 4 can be mounted manually ◦ can be listed in /etc/fstab, but not necessary ◦ all mounting information supplied manually at command line by user or administrator can be mounted automatically on startup ◦ must be listed /etc/fstab, with all appropriate information and options required Every filesystem, drive, storage device is listed as a mounted filesystem associated to a directory someplace under the root (/) directory CST8207 - Algonquin College 5 CST8207 - Algonquin College 6 Benefits Scalable ◦ As new drives are added and new partitions are created, further filesystems can be mounted at various mount points as required. ◦ This means a Linux system does not need to worry about running out of disk space. Transparent ◦ No application would stop working if transferred to a different partition, because access to data is done via the mount point. ◦ Also transparent to user CST8207 - Algonquin College 7 All known filesystems volumes are typically listed in the /etc/fstab (static information about filesystem) file to help automate the mounting process If it is not listed in the /etc/fstab file, then all appropriate information about the filesystem needs to be listed manually at the command line. -
The Microsoft Compound Document File Format"
OpenOffice.org's Documentation of the Microsoft Compound Document File Format Author Daniel Rentz ✉ mailto:[email protected] http://sc.openoffice.org License Public Documentation License Contributors Other sources Hyperlinks to Wikipedia ( http://www.wikipedia.org) for various extended information Mailing list ✉ mailto:[email protected] Subscription ✉ mailto:[email protected] Download PDF http://sc.openoffice.org/compdocfileformat.pdf XML http://sc.openoffice.org/compdocfileformat.odt Project started 2004-Aug-30 Last change 2007-Aug-07 Revision 1.5 Contents 1 Introduction ......................................................................................................... 3 1.1 License Notices 3 1.2 Abstract 3 1.3 Used Terms, Symbols, and Formatting 4 2 Storages and Streams ........................................................................................... 5 3 Sectors and Sector Chains ................................................................................... 6 3.1 Sectors and Sector Identifiers 6 3.2 Sector Chains and SecID Chains 7 4 Compound Document Header ............................................................................. 8 4.1 Compound Document Header Contents 8 4.2 Byte Order 9 4.3 Sector File Offsets 9 5 Sector Allocation ............................................................................................... 10 5.1 Master Sector Allocation Table 10 5.2 Sector Allocation Table 11 6 Short-Streams ................................................................................................... -
Lecture 7 Network Management and Debugging
SYSTEM ADMINISTRATION MTAT.08.021 LECTURE 7 NETWORK MANAGEMENT AND DEBUGGING Prepared By: Amnir Hadachi and Artjom Lind University of Tartu, Institute of Computer Science [email protected] / [email protected] 1 LECTURE 7: NETWORK MGT AND DEBUGGING OUTLINE 1.Intro 2.Network Troubleshooting 3.Ping 4.SmokePing 5.Trace route 6.Network statistics 7.Inspection of live interface activity 8.Packet sniffers 9.Network management protocols 10.Network mapper 2 1. INTRO 3 LECTURE 7: NETWORK MGT AND DEBUGGING INTRO QUOTE: Networks has tendency to increase the number of interdependencies among machine; therefore, they tend to magnify problems. • Network management tasks: ✴ Fault detection for networks, gateways, and critical servers ✴ Schemes for notifying an administrator of problems ✴ General network monitoring, to balance load and plan expansion ✴ Documentation and visualization of the network ✴ Administration of network devices from a central site 4 LECTURE 7: NETWORK MGT AND DEBUGGING INTRO Network Size 160 120 80 40 Management Procedures 0 AUTOMATION ILLUSTRATION OF NETWORK GROWTH VS MGT PROCEDURES AUTOMATION 5 LECTURE 7: NETWORK MGT AND DEBUGGING INTRO • Network: • Subnets + Routers / switches Time to consider • Automating mgt tasks: • shell scripting source: http://www.eventhelix.com/RealtimeMantra/Networking/ip_routing.htm#.VvjkA2MQhIY • network mgt station 6 2. NETWORK TROUBLES HOOTING 7 LECTURE 7: NETWORK MGT AND DEBUGGING NETWORK TROUBLESHOOTING • Many tools are available for debugging • Debugging: • Low-level (e.g. TCP/IP layer) • high-level (e.g. DNS, NFS, and HTTP) • This section progress: ping trace route GENERAL ESSENTIAL TROUBLESHOOTING netstat TOOLS STRATEGY nmap tcpdump … 8 LECTURE 7: NETWORK MGT AND DEBUGGING NETWORK TROUBLESHOOTING • Before action, principle to consider: ✴ Make one change at a time ✴ Document the situation as it was before you got involved. -
Serverless Network File Systems
Serverless Network File Systems Thomas E. Anderson, Michael D. Dahlin, Jeanna M. Neefe, David A. Patterson, Drew S. Roselli, and Randolph Y. Wang Computer Science Division University of California at Berkeley Abstract In this paper, we propose a new paradigm for network file system design, serverless network file systems. While traditional network file systems rely on a central server machine, a serverless system utilizes workstations cooperating as peers to provide all file system services. Any machine in the system can store, cache, or control any block of data. Our approach uses this location independence, in combination with fast local area networks, to provide better performance and scalability than traditional file systems. Further, because any machine in the system can assume the responsibilities of a failed component, our serverless design also provides high availability via redundant data storage. To demonstrate our approach, we have implemented a prototype serverless network file system called xFS. Preliminary performance measurements suggest that our architecture achieves its goal of scalability. For instance, in a 32-node xFS system with 32 active clients, each client receives nearly as much read or write throughput as it would see if it were the only active client. 1. Introduction A serverless network file system distributes storage, cache, and control over cooperating workstations. This approach contrasts with traditional file systems such as Netware [Majo94], NFS [Sand85], Andrew [Howa88], and Sprite [Nels88] where a central server machine stores all data and satisfies all client cache misses. Such a central server is both a performance and reliability bottleneck. A serverless system, on the other hand, distributes control processing and data storage to achieve scalable high performance, migrates the responsibilities of failed components to the remaining machines to provide high availability, and scales gracefully to simplify system management. -
A Brief History of UNIX File Systems
A Brief History of UNIX File Systems Val Henson IBM, Inc. [email protected] Summary • Review of UNIX file system concepts • File system formats, 1974-2004 • File system comparisons and recommendations • Fun trivia • Questions and answers (corrections ONLY during talk) 1 VFS/vnode architecture • VFS: Virtual File System: common object-oriented interface to fs's • vnode: virtual node: abstract file object, includes vnode ops • All operations to fs's and files done through VFS/vnode in- terface • S.R. Kleiman, \Vnodes: An Architecture for Multiple File System Types in Sun UNIX," Summer USENIX 1986 2 Some Definitions superblock: fs summary, pointers to other information inode: on-disk structure containing information about a file indirect block: block containing pointers to other blocks metadata: everything that is not user data, including directory entries 3 Disk characteristics • Track - contiguous region, can be read at maximum speed • Seek time - time to move the head between different tracks • Rotational delay - time for part of track to move under head • Fixed per I/O overhead means bigger I/Os are better 4 In the beginning: System V FS (S5FS) (c. 1974) • First UNIX file system, referred to as \FS" • Disk layout: superblock, inodes, followed by everything else • 512-1024 byte block size, no fragments • Super simple - and super slow! 2-5% of raw disk bandwidth 5 Berkeley Fast File System (FFS or UFS) (c. 1984) • Metadata spread throughout the disk in \cylinder groups" • Block size 4KB minimum, frag size 1KB (to avoid 45% wasted space) • Physical -
Mac OS X Server Administrator's Guide
034-9285.S4AdminPDF 6/27/02 2:07 PM Page 1 Mac OS X Server Administrator’s Guide K Apple Computer, Inc. © 2002 Apple Computer, Inc. All rights reserved. Under the copyright laws, this publication may not be copied, in whole or in part, without the written consent of Apple. The Apple logo is a trademark of Apple Computer, Inc., registered in the U.S. and other countries. Use of the “keyboard” Apple logo (Option-Shift-K) for commercial purposes without the prior written consent of Apple may constitute trademark infringement and unfair competition in violation of federal and state laws. Apple, the Apple logo, AppleScript, AppleShare, AppleTalk, ColorSync, FireWire, Keychain, Mac, Macintosh, Power Macintosh, QuickTime, Sherlock, and WebObjects are trademarks of Apple Computer, Inc., registered in the U.S. and other countries. AirPort, Extensions Manager, Finder, iMac, and Power Mac are trademarks of Apple Computer, Inc. Adobe and PostScript are trademarks of Adobe Systems Incorporated. Java and all Java-based trademarks and logos are trademarks or registered trademarks of Sun Microsystems, Inc. in the U.S. and other countries. Netscape Navigator is a trademark of Netscape Communications Corporation. RealAudio is a trademark of Progressive Networks, Inc. © 1995–2001 The Apache Group. All rights reserved. UNIX is a registered trademark in the United States and other countries, licensed exclusively through X/Open Company, Ltd. 062-9285/7-26-02 LL9285.Book Page 3 Tuesday, June 25, 2002 3:59 PM Contents Preface How to Use This Guide 39 What’s Included -
Unix/Linux Command Reference
Unix/Linux Command Reference .com File Commands System Info ls – directory listing date – show the current date and time ls -al – formatted listing with hidden files cal – show this month's calendar cd dir - change directory to dir uptime – show current uptime cd – change to home w – display who is online pwd – show current directory whoami – who you are logged in as mkdir dir – create a directory dir finger user – display information about user rm file – delete file uname -a – show kernel information rm -r dir – delete directory dir cat /proc/cpuinfo – cpu information rm -f file – force remove file cat /proc/meminfo – memory information rm -rf dir – force remove directory dir * man command – show the manual for command cp file1 file2 – copy file1 to file2 df – show disk usage cp -r dir1 dir2 – copy dir1 to dir2; create dir2 if it du – show directory space usage doesn't exist free – show memory and swap usage mv file1 file2 – rename or move file1 to file2 whereis app – show possible locations of app if file2 is an existing directory, moves file1 into which app – show which app will be run by default directory file2 ln -s file link – create symbolic link link to file Compression touch file – create or update file tar cf file.tar files – create a tar named cat > file – places standard input into file file.tar containing files more file – output the contents of file tar xf file.tar – extract the files from file.tar head file – output the first 10 lines of file tar czf file.tar.gz files – create a tar with tail file – output the last 10 lines -
File System, Files, and *Tab /Etc/Fstab
File system, files, and *tab File system files directories volumes, file systems mounting points local versus networked file systems 1 /etc/fstab Specifies what is to be mounted where and how fs_spec: describes block special device for remote filesystem to be mounted fs_file: describes the mount point fs_vfstype: describes the type of file system fs_mntops: describes the mount options associated with the filesystem 2 /etc/fstab cont. fs_freq: used by the dump command fs_passno: used by fsck to determine the order in which checks are done at boot time. Root file systems should be specified as 1, others should be 2. Value 0 means that file system does not need to be checked 3 /etc/fstab 4 from blocks to mounting points metadata inodes directories superblocks 5 mounting file systems mounting e.g., mount -a unmounting manually or during shutdown umount 6 /etc/mtab see what is mounted 7 Network File System Access file system (FS) over a network looks like a local file system to user e.g. mount user FS rather than duplicating it (which would be a disaster) Developed by Sun Microsystems (mid 80s) history for NFS: NFS, NFSv2, NFSv3, NFSv4 RFC 3530 (from 2003) take a look to see what these RFCs are like!) 8 Network File System How does this actually work? server needs to export the system client needs to mount the system server: /etc/exports file client: /etc/fstab file 9 Network File System Security concerns UID GID What problems could arise? 10 Network File System example from our raid system (what is a RAID again?) Example of exports file from -
Unix/Linux Command Reference
Unix/Linux Command Reference .com File Commands System Info ls – directory listing date – show the current date and time ls -al – formatted listing with hidden files cal – show this month's calendar cd dir - change directory to dir uptime – show current uptime cd – change to home w – display who is online pwd – show current directory whoami – who you are logged in as mkdir dir – create a directory dir finger user – display information about user rm file – delete file uname -a – show kernel information rm -r dir – delete directory dir cat /proc/cpuinfo – cpu information rm -f file – force remove file cat /proc/meminfo – memory information rm -rf dir – force remove directory dir * man command – show the manual for command cp file1 file2 – copy file1 to file2 df – show disk usage cp -r dir1 dir2 – copy dir1 to dir2; create dir2 if it du – show directory space usage doesn't exist free – show memory and swap usage mv file1 file2 – rename or move file1 to file2 whereis app – show possible locations of app if file2 is an existing directory, moves file1 into which app – show which app will be run by default directory file2 ln -s file link – create symbolic link link to file Compression touch file – create or update file tar cf file.tar files – create a tar named cat > file – places standard input into file file.tar containing files more file – output the contents of file tar xf file.tar – extract the files from file.tar head file – output the first 10 lines of file tar czf file.tar.gz files – create a tar with tail file – output the last 10 lines -
Openvms: an Introduction
The Operating System Handbook or, Fake Your Way Through Minis and Mainframes by Bob DuCharme VMS Table of Contents Chapter 7 OpenVMS: An Introduction.............................................................................. 7.1 History..........................................................................................................................2 7.1.1 Today........................................................................................................................3 7.1.1.1 Popular VMS Software..........................................................................................4 7.1.2 VMS, DCL................................................................................................................4 Chapter 8 Getting Started with OpenVMS........................................................................ 8.1 Starting Up...................................................................................................................7 8.1.1 Finishing Your VMS Session...................................................................................7 8.1.1.1 Reconnecting..........................................................................................................7 8.1.2 Entering Commands..................................................................................................8 8.1.2.1 Retrieving Previous Commands............................................................................9 8.1.2.2 Aborting Screen Output.........................................................................................9 -
Filesystem Hierarchy Standard
Filesystem Hierarchy Standard LSB Workgroup, The Linux Foundation Filesystem Hierarchy Standard LSB Workgroup, The Linux Foundation Version 3.0 Publication date March 19, 2015 Copyright © 2015 The Linux Foundation Copyright © 1994-2004 Daniel Quinlan Copyright © 2001-2004 Paul 'Rusty' Russell Copyright © 2003-2004 Christopher Yeoh Abstract This standard consists of a set of requirements and guidelines for file and directory placement under UNIX-like operating systems. The guidelines are intended to support interoperability of applications, system administration tools, development tools, and scripts as well as greater uniformity of documentation for these systems. All trademarks and copyrights are owned by their owners, unless specifically noted otherwise. Use of a term in this document should not be regarded as affecting the validity of any trademark or service mark. Permission is granted to make and distribute verbatim copies of this standard provided the copyright and this permission notice are preserved on all copies. Permission is granted to copy and distribute modified versions of this standard under the conditions for verbatim copying, provided also that the title page is labeled as modified including a reference to the original standard, provided that information on retrieving the original standard is included, and provided that the entire resulting derived work is distributed under the terms of a permission notice identical to this one. Permission is granted to copy and distribute translations of this standard into another language, under the above conditions for modified versions, except that this permission notice may be stated in a translation approved by the copyright holder. Dedication This release is dedicated to the memory of Christopher Yeoh, a long-time friend and colleague, and one of the original editors of the FHS.