DOCSLIB.ORG

Transactions for Durability

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Transactions for Durability CS162: Operating Systems and Systems Programming Lecture 26: Transactions for Durability 4 August 2015 Charles $eiss #ttps://cs162.eecs.ber'eley&edu% Recall: Filesystem )nconsistency *+ample: system cras#es w#ile create %usr/lib/foo Maybe there's a directory entry for it … but t#e inode is still mar'ed as free /usr/lib: free inumber list: (bar, #1236) … (foo, #7823) #7820 ... #7823 #7824 #7825 2 Approach #1: Careful Ordering Se2uence operations in a specific order ― Careful design to allo, sequence to be interrupted safely Post4crash recovery ― $ead data structures to see if there ,ere any operations in progress ― Clean up%finis# as needed pproach taken in ( T, ((S 7fsc'8, and many app- le5el recovery schemes 7e&g&, 9ord) 0 FFS: Create a File Normal operation: llocate data block 9rite data block llocate inode 9rite inode block Update directory ,it# 3le name 4; file number Update modify time for directory 4 FFS: Create a File Normal operation: Recovery: (fsck) entire inode array llocate data block Scan If any unlin'ed files (not in any 9rite data block directory)6 delete ― Or relink (make up a name8 llocate inode Compare free block bitmap against 9rite inode block inode trees Update directory ,it# 3le name Scan directories for missing 4; file number update/access times Update modify time for directory Time proportional to size of disk " Application Level <ormal operation: Reco5ery: 9rite name of each open file to app folder On startup, see if any 3les were left open 9rite changes to backup file If so, loo' for backup file $ename backup file to be 3le 7atomic operation If so, as' user to provided by 3le system8 compare versions Delete list in app folder on clean shutdo,n 6 Approach #2: Copy on Write 71) To update 3le system6 ,rite a ne, version of the 3le system containing the update ― <ever update in place ― $euse e+isting unchanged disk blocks pproach taken in networ' file server appliances (WAFL, >FS8 = Approach #2: Copy on Write 72) Seems e+pensive! Aut ― :pdates can be batched ― lmost all disk writes can occur in parallel ― Most of cost of write is in seeks, not actual transfer ― pplications often rewrite whole file any,ays ? Approach #2: Copy4on49rite 708 To change a 3le, make a new copy of the file, then change the directory entry 9#y is this consistent? ― ssume directory entry has to point to the old version or the other 9#y is this durableC ― Don.t say the operation is done until we've replaced the directory entry B *mulating COW @ user level Open/Create a ne, 3le foo&5 ― ,here v is the 5ersion 1 Do all the updates based on t#e old foo ― $eading from foo and writing to foo&5 ― Including copying over any unchanged parts :pdate the link ― link("foo.v", "foo"); /* replace hardlink – over,rite file # E% 10 *mulating COW @ user level ― Open/Create a ne, file foo&5 ― Do all the updates based on the old foo ― :pdate the link (rename78/lin'788 Does it wor'C 9#at if multiple updaters at same timeC Gow to keep track of every version of file? ― 9ould we want to do thatC 11 Creating a New Version old version new version Write )f file represented as a tree of blocks6 just need to update the leading fringe 12 Creating a New Version old version new version Write If file represented as a tree of blocks, just need to update the leading fringe 13 ZFS Hariable siJed bloc's: "12 B – 12? KA Symmetric tree ― Know if it is large or small when we ma'e the copy Store version number ,ith pointers ― Can create new version by adding bloc's and new pointers Auffers a collection of ,rites before creating a ne, version ,ith them (ree space represented as tree of extents in eac# bloc' group ― Delay updates to freespace (in log8 and do them all when bloc' group is activated 14 COW with smaller-radixold version new version blocks Write If file represented as a tree of blocks, just need to update the leading fringe 15 -ore General Solutions Transactions for Atomic :pdates ― *nsure t#at multiple related updates are performed atomically ― i.e., if a crash occurs in t#e middle, t#e state of t#e systems reflects eit#er all or none of t#e updates ― Most modern file systems use transactions internally to update t#e many pieces ― Many applications implement t#eir own transactions Redundancy for media failures ― Redundant representation 7error correcting codes) ― Replication ― *.g., RA)D disks 16 Recall: Transaction n atomic sequence of actions 7reads%,rites) on a storage system 7or database) T#at takes it from one consistent state to another transaction consistent state 1 consistent state 2 Recall: Fast <D Right ??? T#e concepts related to transactions appear in many aspects of systems Reliability ― File Systems Performance ― Data Base systems ― Concurrent Programming *+ample of a powerful6 elegant concept simplifying implementation <D achieving etter performance& !ystem ehavior is viewed from a particular perspective" ― Properties are met from that perspective Redo Logging *+ample: A: Ban' of merica6 9: 9ells Fargo Prepare phase (con't$ commit case): recei5es guarantee of commit: ― A ,rites "transfer P1!! from 9Q Rall committed>" to log ― A applies transaction ― 4;9: ).ll commit, too 9 finaliJes cras#es transaction: before applying ― W ,ritestransaction? O<all committed>" to log ― W applies transaction Recall: Redo Logging *+ample: A: Ban' of merica6 9: 9ells Fargo Prepare phase (con't$ commit case): recei5es guarantee of commit: ― A ,rites "transfer P1!! from 9Q Rall committed>" to log ― A applies transaction ― 4;9: ).ll commit, cras#es too before applying 9 finaliJes transaction?transaction: ― W ,rites O<all$eads committed>" log w#en to log it is restartedQ ― W applies reappliestransaction transaction Called redo logging Typical Structure With Redo Logging Aegin a transaction – get transaction id Do a bunc# of updates TO THE LOG ― If any fail along the way, roll-back ― Or, if any conNicts ,ith other transactions, roll-back Commit the transaction TO THE LOG ctually do the the updates to the non-log 5ersion Typical Structure With Redo Logging Aegin a transaction – get transaction id Do a bunc# of updates TO THE LOG ― If any fail along the way, roll-back ― Or, if any conNicts ,ith other transactions, roll-back Cras# before committing to t#e Commitlog? the transaction TO THE LOG ctually do the the updates to the non-log 5ersion 9#en rereading log, ignore those updates& Typical Structure With Redo Logging Aegin a transaction – get transaction id Do a bunc# of updates TO THE LOG ― If any fail along the way, roll-back ― Or, if any conNicts ,ith other transactions, roll-back Commit the transaction TO THE LOG ctually doCras# the the afterupdates committing to the non-log to 5ersion t#e log? 9#en rereading log, redo those updates" Assumptions about the Log Log can be written in order ― If "Commit TransactionO is present, then pre5ious parts of the log – the updates – will be too Log writes actually complete before we do the real updates ― Commit record must always be present if any of the updates it would cause are ― Transaction didn.t commit S no changes 9hat do disks actually do? Partially written sectorC ― *rror4correcting codes identify S sector read ,ill fail Write log in orderC ― 9ell, …& Recall: Disk Scheduling Also something controller does (dis' controller #as its own queue) More criteria to optimize for, li'e process scheduling ― Fairness versus Throughput 5ersus $esponse Time 26 9hat do disks actually do? Partially written sectorC ― *rror4correcting codes identify S sector read ,ill fail Write log in orderC ― 9ell, they might reorder ,rites, keep them 2ueued OS needs to tell controller to do writes in order ― 9ait for confirmation write is actually complete ― Disk controllers have caches + delayed writes! Transactional File Systems Aetter reliability t#roug# use of log ― )ll changes are treated as transactions ― A transaction is committed once it is written to t#e log ● Data forced to dis' for reliability ● Process can be accelerated ,ith NH$ - ― Alt#ough File system may not be updated immediately, data preserved in the log Journaling File System ― Applies updates to system metadata using transactions (using logs6 etc&8 ― *x: NTFS, Apple HFS+, Linux XFS, JFS, ext06 ext4 Redo Logging Prepare Reco5ery ― 9rite all changes 7in ― $ead log transaction8 to log ― $edo any operations for Commit committed transactions ― ― Single dis' write to ma'e Marbage collect log transaction durable Redo ― Copy changes to dis' Marbage collection ― $eclaim space in log Creating a 3le 7<o transaction8 (ind free data block(s8 Free Space (ind free inode entry map / (ind dirent insertion point Data blocks )node table 44444444444444444444444444 Directory 9rite map (i.e&6 mark used8 entries 9rite inode entry to point to block(s8 9rite dirent to point to inode Creating a 3le 7As transaction8 (ind free data bloc'7s8 (ind free inode entry Free Space (ind dirent insertion point map / 44444444444444444444444444 Data blocks Write map (used8 Inode table Write inode entry to point to bloc'(s8 Directory Write dirent to point to inode entries tail head t i t r pending m done a t m s o c Log in non45olatile storage (Flash or on Dis'8 ReDo log fter Commit Free ll access to file system 3rst Space map loo's in log … Data blocks Inode table *5entually copy changes to Directory disk entries tail tail tail tail tail head t i t r done m a m t o s c Log in non-volatile storage (Flash) pending Crash during logging - Recover :pon reco5ery scan t#e long Free Detect transaction start ,it# Space map no commit … Data blocks &iscard log entries Inode table Directory Disk remains unc#anged entries tail head t done pending r a t s Log in non-volatile storage (Flash or on Disk) Recovery fter Commit Scan log, find start Find matching commit Free Space map Redo it as usual … Data blocks ― Or just let it happen later Inode table Directory entries tail head t i t r done pending m a m t o s c Log in non-volatile storage (Flash or on Disk) 9hat if had already started writing back the transaction C Idempotent – the result does not change if t#e operation is repeat se5eral times& Just write them again during reco5ery& *+ample: ― Log might say "mark blocks #0!!45 as usedO.
Load more

Recommended publications
  • Andrew File System (AFS) Google File System February 5, 2004
    Advanced Topics in Computer Systems, CS262B Prof Eric A. Brewer Andrew File System (AFS) Google File System February 5, 2004 I. AFS Goal: large-scale campus wide file system (5000 nodes) o must be scalable, limit work of core servers o good performance o meet FS consistency requirements (?) o managable system admin (despite scale) 400 users in the “prototype” -- a great reality check (makes the conclusions meaningful) o most applications work w/o relinking or recompiling Clients: o user-level process, Venus, that handles local caching, + FS interposition to catch all requests o interaction with servers only on file open/close (implies whole-file caching) o always check cache copy on open() (in prototype) Vice (servers): o Server core is trusted; called “Vice” o servers have one process per active client o shared data among processes only via file system (!) o lock process serializes and manages all lock/unlock requests o read-only replication of namespace (centralized updates with slow propagation) o prototype supported about 20 active clients per server, goal was >50 Revised client cache: o keep data cache on disk, metadata cache in memory o still whole file caching, changes written back only on close o directory updates are write through, but cached locally for reads o instead of check on open(), assume valid unless you get an invalidation callback (server must invalidate all copies before committing an update) o allows name translation to be local (since you can now avoid round-trip for each step of the path) Revised servers: 1 o move
    [Show full text]
  • A Survey of Distributed File Systems
    A Survey of Distributed File Systems M. Satyanarayanan Department of Computer Science Carnegie Mellon University February 1989 Abstract Abstract This paper is a survey of the current state of the art in the design and implementation of distributed file systems. It consists of four major parts: an overview of background material, case studies of a number of contemporary file systems, identification of key design techniques, and an examination of current research issues. The systems surveyed are Sun NFS, Apollo Domain, Andrew, IBM AIX DS, AT&T RFS, and Sprite. The coverage of background material includes a taxonomy of file system issues, a brief history of distributed file systems, and a summary of empirical research on file properties. A comprehensive bibliography forms an important of the paper. Copyright (C) 1988,1989 M. Satyanarayanan The author was supported in the writing of this paper by the National Science Foundation (Contract No. CCR-8657907), Defense Advanced Research Projects Agency (Order No. 4976, Contract F33615-84-K-1520) and the IBM Corporation (Faculty Development Award). The views and conclusions in this document are those of the author and do not represent the official policies of the funding agencies or Carnegie Mellon University. 1 1. Introduction The sharing of data in distributed systems is already common and will become pervasive as these systems grow in scale and importance. Each user in a distributed system is potentially a creator as well as a consumer of data. A user may wish to make his actions contingent upon information from a remote site, or may wish to update remote information.
    [Show full text]
  • The Influence of Scale on Distributed File System Design
    IEEE TRANSAmIONS ON SOFIWARE ENGINEERING, VOL. 18, NO. I, JANUARY lY92 The Influence of Scale on Distributed File System Design Mahadev Satyanarayanan, Member, IEEE Abstract- Scale should be recognized as a primary factor into autonomous or semi-autonomous organizations for man- influencing the architecture and implementation of distributed agement purposes. Hence a distributed system that has many systems. This paper uses Andrew and Coda, distributed file users or nodes will also span many organizations. Regardless systems built at Carnegie Mellon University, to validate this proposition. Performance, operability, and security are dominant of the specific metric of scale, the designs of distributed considerations in the design of these systems. Availability is a systems that scale well are fundamentally different from less further consideration in the design of Coda. Client caching, scalable designs. bulk data transfer, token-based mutual authentication, and hi- In this paper we describe the lessons we have learned erarchical organization of the protection domain have emerged about scalability from the Andrew File System and the Codu as mechanisms that enhance scalability. The separation of con- cerns made possible by functional specialization has also proved File System. These systems are particularly appropriate for valuable in scaling. Heterogeneity is an important by-product our discussion, because they have been designed for growth of growth, but the mechanisms available to cope with it are to thousands of nodes and users. We focus on distributed rudimentary. Physical separation of clients and servers turns out file systems, because they are the most widely used kind of to be a critical requirement for scalability.
    [Show full text]
  • Distributed File Systems
    Please note: Please start working your research project (proposal will be due on Feb. 19 in class) Each group needs to turn in a printed version of their proposal and intermediate report. Also, before class each group needs to email me a DOC version. Instructor’s Guide for Coulouris, Dollimore and Kindberg Distributed Systems: Concepts and Design Edn. 4 © Pearson Education 2005 Remote Procedure Call (1): at-least-once or at-most-once semantics client: "stub" instead of "proxy" (same function, different names) behaves like a local procedure, marshal arguments, communicate the request server: dispatcher "stub": unmarshal arguments, communicate the results back Instructor’s Guide for Coulouris, Dollimore and Kindberg Distributed Systems: Concepts and Design Edn. 4 © Pearson Education 2005 Remote Procedure Call (2) client process server process Request Reply client stub server stub procedure procedure client service program Communication Communication procedure module module dispatcher Instructor’s Guide for Coulouris, Dollimore and Kindberg Distributed Systems: Concepts and Design Edn. 4 © Pearson Education 2005 Sun RPC (1): Designed for client-server communication in the SUN NFS (network file system) Supplied as a part of SUN and other UNIX operating systems Over either UDP or TCP Provides an interface definition language (IDL) initially XDR is for data representation, extended to be IDL less modern than CORBA IDL and Java program numbers (obtained from a central authority) instead of interface names procedure numbers (used as a procedure identifier) instead of procedure names only a single input parameter is allowed (then we have to use a ?) Offers an interface compiler (rpcgen) for C language, which generates the following: client stub server main procedure, dispatcher, and server stub XDR marshalling, unmarshaling Instructor’s Guide for Coulouris, Dollimore and Kindberg Distributed Systems: Concepts and Design Edn.
    [Show full text]
  • Design and Evolution of the Apache Hadoop File System(HDFS)
    Design and Evolution of the Apache Hadoop File System(HDFS) Dhruba Borthakur Engineer@Facebook Committer@Apache HDFS SDC, Sept 19 2011 2011 Storage Developer Conference. © Insert Your Company Name. All Rights Reserved. Outline Introduction Yet another file-system, why? Goals of Hadoop Distributed File System (HDFS) Architecture Overview Rational for Design Decisions 2011 Storage Developer Conference. © Insert Your Company Name. All Rights Reserved. Who Am I? Apache Hadoop FileSystem (HDFS) Committer and PMC Member Core contributor since Hadoop’s infancy Facebook (Hadoop, Hive, Scribe) Yahoo! (Hadoop in Yahoo Search) Veritas (San Point Direct, Veritas File System) IBM Transarc (Andrew File System) Univ of Wisconsin Computer Science Alumni (Condor Project) 2011 Storage Developer Conference. © Insert Your Company Name. All Rights Reserved. Hadoop, Why? Need to process Multi Petabyte Datasets Data may not have strict schema Expensive to build reliability in each application. Failure is expected, rather than exceptional. Elasticity, # of nodes in a cluster is never constant. Need common infrastructure Efficient, reliable, Open Source Apache License 2011 Storage Developer Conference. © Insert Your Company Name. All Rights Reserved. Goals of HDFS Very Large Distributed File System 10K nodes, 1 billion files, 100 PB Assumes Commodity Hardware Files are replicated to handle hardware failure Detect failures and recovers from them Optimized for Batch Processing Data locations exposed so that computations can move to where data resides Provides very high aggregate bandwidth User Space, runs on heterogeneous OS 2011 Storage Developer Conference. © Insert Your Company Name. All Rights Reserved. Commodity Hardware Typically in 2 level architecture – Nodes are commodity PCs – 20-40 nodes/rack – Uplink from rack is 4 gigabit – Rack-internal is 1 gigabit 2011 Storage Developer Conference.
    [Show full text]
  • Andrew File System (AFS)
    Andrew File System ♦ Andrew File System (AFS) 8 started as a joint effort of Carnegie Mellon University and IBM 8 today basis for DCE/DFS: the distributed file system included in the Open Software Foundations’s Distributed Computing Environment 8 some UNIX file system usage observations, as pertaining to caching – infrequently updated shared files and local user files will remain valid for long periods of time (the latter because they are being updated on owners workstations) – allocate large local disk cache, e.g., 100 MByte, that can provide a large enough working set for all files of one user such that the file is still in this cache when used next time – assumptions about typical file accesses (based on empirical evidence) iusually small files, less than 10 Kbytes ireads much more common than writes (appr. 6:1) iusually sequential access, random access not frequently found iuser-locality: most files are used by only one user iburstiness of file references: once file has been used, it will be used in the nearer future with high probability Distributed Systems - Fall 2001 V - 39 © Stefan Leue 2002 tele Andrew File System ♦ Andrew File System (AFS) 8 design decisions for AFS – whole-file serving: entire contents of directories and files transfered from server to client (AFS-3: in chunks of 64 Kbytes) – whole file caching: when file transfered to client it will be stored on that client’s local disk Distributed Systems - Fall 2001 V - 40 © Stefan Leue 2002 tele Andrew File System ♦ AFS architecture: Venus, network and Vice Workstations
    [Show full text]
  • Using the Andrew File System with BSD
    Using the Andrew File System with BSD H. Meiland May 4, 2006 Abstract Since the beginning of networks, one of the basic idea’s has been sharing of files; even though with the Internet as advanced as today, simple platform independent file sharing is not common. Why is the closest thing we use WebDAV, a ’neat trick over http’, instead of a real protocol? In this paper the Andrew File System will be described which has been (and is) the file sharing core of many universities and companies world- wide. Also the reason for it’s relative unawareness in the community will be answered, and it’s actual features and performance in comparison with alternative network filesystems. Finally some information will be given on how to use it with our favorite OS: BSD. 1 History • 1984 Carnegie Mellon University, first release • 1989 TransArc Corporation formed by part of original team members • 1994 TransArc purchased by IBM • 1997 Start of Arla development at stacken.kth.se • 2000 IBM releases AFS in opensource (IBM License) • 2000 http://www.OpenAFS.org • 2006 good support for lot’s of platforms, many new features etc. 1 2 Overview 2.1 User point of view 2.1.1 Global namespace While discussing global filesystem, it is easy to dive into a organization, and explain wonderfull features like having replicas of often accessed data in branch-offices, and moving home-directories to local fileservers when mov- ing employees between departments. An essential feature of AFS is often overlooked: a common root as accesspoint of all AFS stored data.
    [Show full text]
  • The Andrew File System (AFS)
    50 The Andrew File System (AFS) The Andrew File System was introduced at Carnegie-Mellon University (CMU) 1 in the 1980’s [H+88]. Led by the well-known Professor M. Satya- narayanan of Carnegie-Mellon University (“Satya” for short), the main goal of this project was simple: scale. Specifically, how can one design a distributed file system such that a server can support as many clients as possible? Interestingly, there are numerous aspects of design and implementa- tion that affect scalability. Most important is the design of the protocol be- tween clients and servers. In NFS, for example, the protocol forces clients to check with the server periodically to determine if cached contents have changed; because each check uses server resources (including CPU and network bandwidth), frequent checks like this will limit the number of clients a server can respond to and thus limit scalability. AFS also differs from NFS in that from the beginning, reasonable user- visible behavior was a first-class concern. In NFS, cache consistency is hard to describe because it depends directly on low-level implementa- tion details, including client-side cache timeout intervals. In AFS, cache consistency is simple and readily understood: when the file is opened, a client will generally receive the latest consistent copy from the server. 50.1 AFS Version 1 We will discuss two versions of AFS [H+88, S+85]. The first version (which we will call AFSv1, but actually the original system was called the ITC distributed file system [S+85]) had some of the basic design in place, but didn’t scale as desired, which led to a re-design and the final protocol (which we will call AFSv2, or just AFS) [H+88].
    [Show full text]
  • The Andrew File System (From CMU) Case Study ­ Andrew File System
    DFS Case Studies, Part 2 The Andrew File System (from CMU) Case Study - Andrew File System ● Designed to support information sharing on a large scale by minimizing client-server communications ● Makes heavy use of caching technologies ● Adopted as the basis for the DCE/DFS file system in the Open Software Foundation©s Distributed Computing Environment (DCE) AFS Characteristics ● Provides transparent access to remote shared files for UNIX programs (using the normal UNIX file primitives) ● Programs access to remote files without modification or recompilation ● AFS is compatible with NFS, in that files may be remotely accessed using NFS ● However, AFS differs markedly from NFS in its design and implementation AFS Design Goals ● Scalability is the most important design goal for the AFS designers ● Designed to perform well with larger numbers of active users (when compared to NFS) ● Key strategy in achieving scalability is the caching of whole (complete) files in client nodes AFS Design Characteristics ● Whole-file serving - the entire contents of directories and files are transmitted to client computers by AFS servers ● Whole-file caching - once a copy of a file (or a file- chunk) has been transferred to a client computer, it is stored in a cache on the local disk; the cache is permanent, surviving reboots of the client computer and it is used to satisfy clients© open requests in preference to remote copies whenever possible AFS Observations ● Shared files that are infrequently updated (such as UNIX commands and libraries) and files accessed
    [Show full text]
  • Distributed File Systems
    CS 425 / ECE 428 Distributed Systems Fall 2020 Indranil Gupta (Indy) Lecture 25: Distributed File Systems All slides © IG Jokes for this Topic • Why did NFS like boxing, but AFS did not show up for the fight? Because NFS can do blocks, but AFS is an optimistic person who only sends callback promises. • Did you know why written exams not use too big a page size (larger than letter/A4 size)? Because it can result in false sharing. 2 File System • Contains files and directories (folders) • Higher level of abstraction – Prevents users and processes from dealing with disk blocks and memory blocks 3 File Contents • Typical File Header Block 0 Block 1 … Block N-1 File contents are in here • Timestamps: creation, read, write, header • File type, e.g., .c, .java • Ownership, e.g., edison • Access Control List: who can access this file and in what mode • Reference Count: Number of directories containing this file • May be > 1 (hard linking of files) 4 • When 0, can delete file What about Directories? • They’re just files! • With their “data” containing – The meta-information about files the directory contains – Pointers (on disk) to those files 5 Unix File System: Opening and Closing Files • Uses notion of file descriptors – Handle for a process to access a file • Each process: Needs to open a file before reading/writing file – OS creates an internal datastructure for a file descriptor, returns handle • filedes=open(name, mode) – mode = access mode, e.g., r, w, x • filedes=creat(name, mode) – Create the file, return the file descriptor • close(filedes)
    [Show full text]
  • A Distributed File System for Distributed Conferencing System
    A DISTRIBUTED FILE SYSTEM FOR DISTRIBUTED CONFERENCING SYSTEM By PHILIP S. YEAGER A THESIS PRESENTED TO THE GRADUATE SCHOOL OF THE UNIVERSITY OF FLORIDA IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF MASTER OF SCIENCE UNIVERSITY OF FLORIDA 2003 Copyright 2003 by Philip S. Yeager ACKNOWLEDGMENTS I would like to thank Dr. Richard Newman for his help and guidance with this project. I would like to express my gratitude to Dr. Jonathan C.L. Liu and Dr. Beverly Sanders for serving on my committee. I would like to thank Dr. Joseph Wilson for serving as a substitute at my defense. I would also like to thank Vijay Manian and the other DCS group members for their advice and contributions. I thank my parents and friends for their encouragement and support. Finally, I would like to thank Candice Williams for everything she has done. Without these people this work would not have been possible. iii TABLE OF CONTENTS Page ACKNOWLEDGMENTS ................................................................................................. iii LIST OF FIGURES ......................................................................................................... viii ABSTRACT....................................................................................................................... ix CHAPTER 1 INTRODUCTION ........................................................................................................1 1.1 Introduction........................................................................................................1
    [Show full text]
  • Of File Systems and Storage Models
    Chapter 4 Of File Systems and Storage Models Disks are always full. It is futile to try to get more disk space. Data expands to fill any void. –Parkinson’sLawasappliedto disks 4.1 Introduction This chapter deals primarily with how we store data. Virtually all computer systems require some way to store data permanently; even so-called “diskless” systems do require access to certain files in order to boot, run and be useful. Albeit stored remotely (or in memory), these bits reside on some sort of storage system. Most frequently, data is stored on local hard disks, but over the last few years more and more of our files have moved “into the cloud”, where di↵erent providers o↵er easy access to large amounts of storage over the network. We have more and more computers depending on access to remote systems, shifting our traditional view of what constitutes a storage device. 74 CHAPTER 4. OF FILE SYSTEMS AND STORAGE MODELS 75 As system administrators, we are responsible for all kinds of devices: we build systems running entirely without local storage just as we maintain the massive enterprise storage arrays that enable decentralized data replication and archival. We manage large numbers of computers with their own hard drives, using a variety of technologies to maximize throughput before the data even gets onto a network. In order to be able to optimize our systems on this level, it is important for us to understand the principal concepts of how data is stored, the di↵erent storage models and disk interfaces.Itisimportanttobeawareofcertain physical properties of our storage media, and the impact they, as well as certain historic limitations, have on how we utilize disks.
    [Show full text]