DOCSLIB.ORG

Souborové Systémy V Linuxu

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Souborové Systémy V Linuxu Souborov´esyst´emyv Linuxu Red Hat Luk´aˇsCzerner May 16, 2016 Copyright c 2016 Luk´aˇsCzerner, Red Hat. Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.3 or any later version published by the Free Software Foundation; with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license is included in the COPYING file. Agenda 1 Co je to souborov´ysyst´em 2 Z´akladn´ıpojmy 3 Rozhran´ısouborov´ychsyst´em˚u 4 Intern´ıstruktury 5 Konzistence pˇriv´ypadku 6 Pokroˇcil´efunkce 7 Nov´etypy zaˇr´ızen´ı 8 Jak se zapojit 9 Ot´azky Part I Co je to souborov´ysyst´em? Co je to souborov´ysyst´em? Zp˚usoborganizace dat na nosn´emm´ediuve formˇesoubor˚ua adres´aˇr˚u Snadn´ypˇr´ıstup Uˇzivatelsk´adata snadno pˇr´ıstupn´apojmenovan´ychsouborech Soubory seskupen´ev pojmenovan´ychadres´aˇr´ıch Snadno pochopiteln´astromov´astruktura Virtualizace adresov´ehoprostoru m´edia Adresov´yprostor souboru vs. logick´yprostor m´edia Prostory jednotliv´ychsoubor˚ujsou na sobˇenez´avisl´e R´ızen´ıpˇr´ıstupuˇ Pr´avake ˇcten´ı,z´apisu Kv´oty pro omezen´ımnoˇzstv´ıdat Typy souborov´ychsyst´em˚u Souborov´esys´emyv uˇzivatelsk´emprostoru pouˇzit´ımjadern´ehomodulu FUSE GlusterFS, sshfs Souborov´esyst´emyv jadern´emprostoru Distribuovan´e,s´ıˇtov´e,lok´aln´ı Speci´aln´ısouborov´esyst´emy Pseudo souborov´esyst´emy Utility v uˇzivatelsk´emprostoru N´astrojepro vytvoˇren´ısouborov´eho syst´emu mkfs.ext4, mkfs.xfs, ... Vytvoˇren´ısouborov´ehosyst´emus dan´ymiparametry N´astrojepro kontrolu souborov´ehosyst´emu fsck.ext4, xfs repair, ... Kontrola, oprava, optimalizace N´astrojepro spr´avusouborov´ehosyst´emu btrfs, resize2fs, tune2fs, xfs growfs, debugfs Vˇseod zmˇenyvelikosti, pˇresexport metadat aˇzk detailn´ı ´upravˇeintern´ıchstruktur Part II Pojmy D˚uleˇzit´estruktury Inode - Index node Struktura reprezentuj´ıc´ıvˇsechnytypy soubor˚u- v pamˇeti i mode - typ souboru i ino - ˇc´ısloinode i nlink - poˇcetodkaz˚una inode i size - velikost inode dalˇs´ıviz. include/linux/fs.h:528 Dentry - Directory entry Struktura mapuj´ıc´ıjm´enosouboru na ˇc´ısloinode - v pamˇeti d parent - ukazatel rodiˇcovskou dentry d name - struktura obsahuj´ıc´ıjm´enoz´aznamu d inode - odkaz na inode - m˚uˇzeb´ytNULL dalˇs´ıviz. include/linux/dcache.h:108 D˚uleˇzit´estruktury - pokraˇcov´an´ı File Reprezentuje otevˇren´ysoubor f path - struktura reprezentuje cestu k souboru f inode - odkaz na pˇr´ısluˇsnouinode f mode - m´odotevˇren´ehosouboru f pos - aktu´aln´ıpozice v souboru dalˇs´ıviz. include/linux/fs.h:776 Superblock Identifikuje dan´ysouborov´ysyst´emna m´ediu- v pamˇeti s dev - ˇc´ısloidentifikuj´ıc´ızaˇr´ızen´ı s blocksize - velikost bloku s type - struktura popisuj´ıc´ıtyp souborov´ehosyst´emu s magic - magick´e ˇc´ısloidentifikuj´ıc´ıtyp souborov´ehosyst´emu dalˇs´ıviz. include/linux/fs.h:1821 Dalˇs´ıpojmy Blok Nejmenˇs´ıalokovateln´ajednotka souborov´ehosyst´emu Str´anka Pˇresundat mezi pamˇet´ıa z´aznamov´ymm´ediem Part III Rozhran´ısouborov´ychsyst´em˚u The Linux I/O Stack Diagram version 1.0, 2012-06-20 outlines the Linux I/O stack as of Kernel version 3.3 mmap (anonymous pages) Applications (Processes) malloc ... stat(2) read(2) open(2) write(2) chmod(2) VFS block based FS Network FS pseudo FS special ext2 ext3 ext4 purpose FS direct I/O NFS coda proc sysfs Page xfs btrfs tmpfs (O_DIRECT) ifs smbfs ... pipefs futexfs ramfs Cache iso9660 gfs ocfs ... devtmpfs ... usbfs network stackable Block I/O Layer optional stackable devices on topLVM of “normal” block devices – work on bios mdraid device drbd ... mapper BIOs (Block I/O) I/O Scheduler maps bios to requests cfq deadline noop hooked in Device Drivers (hook in similar like stacked devices like request-based mdraid/device mapper do) device mapper targets /dev/fio* /dev/rssd* dm-multipath SCSI upper layer iomemory-vsl mtip32xx with module option /dev/vd* /dev/fio* /dev/sda /dev/sdb ... /dev/nvme#n# nvme sysfs (transport attributes) SCSI mid layer virtio_blk iomemory-vsl Transport Classes scsi_transport_fc scsi_transport_sas SCSI low layer scsi_transport_... libata megaraid sas aacraid qla2xxx lpfc iscsi_tcp ... ahci ata_piix ... network HDD SSD DVD LSI Adaptec Qlogic Emulex ... Fusion-io nvme Micron drive RAID RAID HBA HBA PCIe Card device PCIe Card Physical devices The Linux I/O Stack Diagram (version 1.0, 2012-06-20) http://www.thomas-krenn.com/en/oss/linux-io-stack-diagram.html Created by Werner Fischer and Georg Schönberger License: CC-BY-SA 3.0, see http://creativecommons.org/licenses/by-sa/3.0/ Rozhran´ısouborov´ychsyst´em˚u Syst´emov´avol´an´ı Standardn´ırozhran´ıpro komunikaci s j´adrem read, write, stat, open, close, unlink, fallocate, ... Input/output control - ioctl P˚uvodnˇeslouˇzilopro komunikaci s HW zaˇr´ızen´ım Dnes se zneuˇz´ıv´ajako "levn´e"rozhran´ıpro cokoliv FIFREEZE, FITRIM, EXT4 IOC RESIZE FS, XFS IOC ZERO RANGE, ... procfs, sysfs Speci´aln´ısoubory vˇetˇsinouslouˇz´ıpro exportov´an´ıinformac´ıdo uˇzivatelsk´ehoprostoru /sys/fs/ext4/features/lazy itable init Nˇekdyvˇsaki pro nastaven´ıparametr˚u /sys/fs/ext4/sda1/extent max zeroout kb The Linux I/O Stack Diagram version 1.0, 2012-06-20 outlines the Linux I/O stack as of Kernel version 3.3 mmap (anonymous pages) Applications (Processes) malloc ... stat(2) read(2) open(2) write(2) chmod(2) VFS block based FS Network FS pseudo FS special ext2 ext3 ext4 purpose FS direct I/O NFS coda proc sysfs Page xfs btrfs tmpfs (O_DIRECT) ifs smbfs ... pipefs futexfs ramfs Cache iso9660 gfs ocfs ... devtmpfs ... usbfs network stackable Block I/O Layer optional stackable devices on topLVM of “normal” block devices – work on bios mdraid device drbd ... mapper BIOs (Block I/O) I/O Scheduler maps bios to requests cfq deadline noop hooked in Device Drivers (hook in similar like stacked devices like request-based mdraid/device mapper do) device mapper targets /dev/fio* /dev/rssd* dm-multipath SCSI upper layer iomemory-vsl mtip32xx with module option /dev/vd* /dev/fio* /dev/sda /dev/sdb ... /dev/nvme#n# nvme sysfs (transport attributes) SCSI mid layer virtio_blk iomemory-vsl Transport Classes scsi_transport_fc scsi_transport_sas SCSI low layer scsi_transport_... libata megaraid sas aacraid qla2xxx lpfc iscsi_tcp ... ahci ata_piix ... network HDD SSD DVD LSI Adaptec Qlogic Emulex ... Fusion-io nvme Micron drive RAID RAID HBA HBA PCIe Card device PCIe Card Physical devices The Linux I/O Stack Diagram (version 1.0, 2012-06-20) http://www.thomas-krenn.com/en/oss/linux-io-stack-diagram.html Created by Werner Fischer and Georg Schönberger License: CC-BY-SA 3.0, see http://creativecommons.org/licenses/by-sa/3.0/ VFS - Virtual File System Switch Pohled uˇzivatelsk´ehoprostoru Abstraktn´ıvrstva poskytuj´ıc´ıjednotn´euˇzivatelsk´erozhran´ı mezi uˇzivatelsk´ymi aplikacemi a r˚uzn´ymisouborov´ymisyst´emy Aplikace nemus´ıvˇedˇetzda pˇristupuje k lok´aln´ımu,nebo s´ıˇtov´emusouborov´emusyst´emu Pohled j´adra Abstraktn´ıvrstva poskytuj´ıc´ıjednotn´erozhran´ımezi j´adrem a souborov´ymisyst´emy Poskytuje funkce spoleˇcn´epro vˇsechnysouborov´esyst´emy Usnadˇnujev´yvojnov´ehosouborov´ehosyst´emu Objektovˇeorientovan´ypˇr´ıstup file operations inode operations dentry operations super operations address space operations File operations struct file_operations { .... loff_t (*llseek) (struct file *, loff_t, int); ssize_t (*read) (struct file *, char __user *, size_t, loff_t *); ssize_t (*write) (struct file *, const char __user *, size_t, loff_t *); int (*mmap) (struct file *, struct vm_area_struct *); int (*open) (struct inode *, struct file *); int (*fsync) (struct file *, loff_t, loff_t, int datasync); int (*fasync) (int, struct file *, int); long (*fallocate)(struct file *file, int mode, loff_t offset, loff_t len); ... }; VFS - Virtual File System Switch Dentry cache Spravuje hashovac´ıtabulku adres´aˇrov´ychz´aznam˚u Pˇrivytv´aˇren´ıcesty k inode jsou uloˇzenyvˇsechny prvky cesty Pˇrivytvoˇren´ız´aznamuv dentry cache je z´aroveˇnvytvoˇren pˇr´ısluˇsn´yz´aznamv inode cache Inode cache Spravuje hashovac´ıtabulku inode Urychluje pˇr´ıstupk inode Novou, pr´azdnouinode naplˇnujes´amsouborov´ysyst´em The Linux I/O Stack Diagram version 1.0, 2012-06-20 outlines the Linux I/O stack as of Kernel version 3.3 mmap (anonymous pages) Applications (Processes) malloc ... stat(2) read(2) open(2) write(2) chmod(2) VFS block based FS Network FS pseudo FS special ext2 ext3 ext4 purpose FS direct I/O NFS coda proc sysfs Page xfs btrfs tmpfs (O_DIRECT) ifs smbfs ... pipefs futexfs ramfs Cache iso9660 gfs ocfs ... devtmpfs ... usbfs network stackable Block I/O Layer optional stackable devices on topLVM of “normal” block devices – work on bios mdraid device drbd ... mapper BIOs (Block I/O) I/O Scheduler maps bios to requests cfq deadline noop hooked in Device Drivers (hook in similar like stacked devices like request-based mdraid/device mapper do) device mapper targets /dev/fio* /dev/rssd* dm-multipath SCSI upper layer iomemory-vsl mtip32xx with module option /dev/vd* /dev/fio* /dev/sda /dev/sdb ... /dev/nvme#n# nvme sysfs (transport attributes) SCSI mid layer virtio_blk iomemory-vsl Transport Classes scsi_transport_fc scsi_transport_sas SCSI low layer scsi_transport_... libata megaraid sas aacraid qla2xxx lpfc iscsi_tcp ... ahci ata_piix ... network HDD SSD DVD LSI Adaptec Qlogic Emulex ... Fusion-io nvme Micron drive RAID RAID HBA HBA PCIe Card device PCIe Card Physical devices The Linux I/O Stack Diagram (version 1.0, 2012-06-20) http://www.thomas-krenn.com/en/oss/linux-io-stack-diagram.html Created by Werner Fischer and Georg Schönberger License: CC-BY-SA 3.0, see http://creativecommons.org/licenses/by-sa/3.0/ Page Cache Diskov´acache - urychluje opakovan´ypˇr´ıstupk dat˚umna m´ediu Pouˇz´ıv´ase pro vˇsechnypˇrenosov´eoperace (kromˇeoperace pˇr´ım´ehopˇr´ıstupudirect I/O) Se str´ankami se opˇetpracuje pomoc´ıalgoritmu LRU Inode address space Obsahuje mimo jin´eodkaz na strom str´anekpˇr´ısluˇs´ıc´ıch dan´emusouboru address space operations pro manipulaci se str´ankami readpage, writepage, invalidatepage Ulehˇcujepr´acisouborov´emusyst´emu- pr´acese str´ankami Pˇr´ıznakystr´anek PG dirty, PG uptodate, PG locked, PG active, ..
Load more

Recommended publications
  • DMFS - a Data Migration File System for Netbsd
    DMFS - A Data Migration File System for NetBSD William Studenmund Veridian MRJ Technology Solutions NASAAmes Research Center" Abstract It was designed to support the mass storage systems de- ployed here at NAS under the NAStore 2 system. That system supported a total of twenty StorageTek NearLine ! have recently developed DMFS, a Data Migration File tape silos at two locations, each with up to four tape System, for NetBSD[I]. This file system provides ker- drives each. Each silo contained upwards of 5000 tapes, nel support for the data migration system being devel- and had robotic pass-throughs to adjoining silos. oped by my research group at NASA/Ames. The file system utilizes an underlying file store to provide the file The volman system is designed using a client-server backing, and coordinates user and system access to the model, and consists of three main components: the vol- files. It stores its internal metadata in a flat file, which man master, possibly multiple volman servers, and vol- resides on a separate file system. This paper will first man clients. The volman servers connect to each tape describe our data migration system to provide a context silo, mount and unmount tapes at the direction of the for DMFS, then it will describe DMFS. It also will de- volman master, and provide tape services to clients. The scribe the changes to NetBSD needed to make DMFS volman master maintains a database of known tapes and work. Then it will give an overview of the file archival locations, and directs the tape servers to move and mount and restoration procedures, and describe how some typi- tapes to service client requests.
    [Show full text]
  • An Incremental Path Towards a Safer OS Kernel
    An Incremental Path Towards a Safer OS Kernel Jialin Li Samantha Miller Danyang Zhuo University of Washington University of Washington Duke University Ang Chen Jon Howell Thomas Anderson Rice University VMware Research University of Washington LoC Abstract Incremental Progress Tens of Linux Safe Linux Linux has become the de-facto operating system of our age, Millions FreeBSD but its vulnerabilities are a constant threat to service availabil- Hundreds of Singularity ity, user privacy, and data integrity. While one might scrap Thousands Biscuit Linux and start over, the cost of that would be prohibitive due Theseus Thousands RedLeaf seL4 to Linux’s ubiquitous deployment. In this paper, we propose Hyperkernel Safety an alternative, incremental route to a safer Linux through No Type Ownership Functional proper modularization and gradual replacement module by Guarantees Safety Safety Verification module. We lay out the research challenges and potential Figure 1: Our vision and the current state of systems. solutions for this route, and discuss the open questions ahead. security vulnerabilities are reported each year, and lifetime CCS Concepts analysis suggests that the new code added this year has intro- duced tens of thousands of more bugs. ! • Software and its engineering Software verification; While operating systems are far from the only source of se- ! • Computer systems organization Reliability. curity vulnerabilities, it is hard to envision building trustwor- Keywords thy computer systems without addressing operating system correctness. kernel safety, verified systems, reliable systems One attractive option is to scrap Linux and start over. Many ACM Reference Format: past works focus on building more secure and correct op- Jialin Li, Samantha Miller, Danyang Zhuo, Ang Chen, Jon Howell, erating systems from the ground up: ones based on strong and Thomas Anderson.
    [Show full text]
  • FS Design Around SMR Approved SNIA Tutorial © 2015 Storage Networking Industry Association
    PRESENTATIONFS Design TITLE Around GOES HERE SMR Seagate’s Journey with EXT4 Adrian Palmer - Seagate SNIA Legal Notice The material contained in this tutorial is copyrighted by the SNIA unless otherwise noted. Member companies and individual members may use this material in presentations and literature under the following conditions: Any slide or slides used must be reproduced in their entirety without modification The SNIA must be acknowledged as the source of any material used in the body of any document containing material from these presentations. This presentation is a project of the SNIA Education Committee. Neither the author nor the presenter is an attorney and nothing in this presentation is intended to be, or should be construed as legal advice or an opinion of counsel. If you need legal advice or a legal opinion please contact your attorney. The information presented herein represents the author's personal opinion and current understanding of the relevant issues involved. The author, the presenter, and the SNIA do not assume any responsibility or liability for damages arising out of any reliance on or use of this information. NO WARRANTIES, EXPRESS OR IMPLIED. USE AT YOUR OWN RISK. FS Design Around SMR Approved SNIA Tutorial © 2015 Storage Networking Industry Association. All Rights Reserved. 2 Abstract SMR Friendly File System SMR is a fundamental drive technology, embraced by drive vendors. SMR changes fundamental assumptions of file system management. This long-held notion of Random-Writes now resembles the write profile of sequential-access tape. Seagate is leading the way in providing a standards compliant IO stack for use with the new drives.
    [Show full text]
  • Advanced File Systems and ZFS
    Advanced File Systems and ZFS Jan Šenolt [email protected] Solaris Engineering May 6, 2021 1 Agenda • Crash Consistency Problem • fsck • Journalling • Log-structured File Systems • Soft-Updates • ZFS 2 Crash Consistency Problem 3 Traditional UNIX File System • appending a new block to the file involves at least 3 writes to different data structures: • block bitmap - allocate the block • inode - update e2di_blocks[], e2di_size • data block - actual payload • what will happen if we fail to make some of these changes persistent? • crash-consistency problem • File System Inconsistency • how to deal with? 4 File System Checker, fsck • a reactive approach • let the inconsistencies happen and try to find (and eventually fix) them later (on reboot) • metadata-only based checks • verify that each allocated block is referenced by exactly one inode • … but what if it is not?? • unable to detect corrupted (missing) user data • does not scale well • O(file system size) • improvements? • check only recently changed data? • … still useful! 5 Journaling, logging 1. start a new transaction 2. write all planned change to the journal 3. make sure that all writes to log completed properly • close the transaction 4. make the actual in-place updates • journal reply • after crash, on reboot • walk the journal, find all complete transactions and apply them 6 Journaling, logging (2) • journal can be a (preallocated) file within the file system or a dedicated device • small circular buffer • UFS: 1MB per 1GB, 64MB max • types of journals • physical - stores
    [Show full text]
  • Ted Ts'o on Linux File Systems
    Ted Ts’o on Linux File Systems An Interview RIK FARROW Rik Farrow is the Editor of ;login:. ran into Ted Ts’o during a tutorial luncheon at LISA ’12, and that later [email protected] sparked an email discussion. I started by asking Ted questions that had I puzzled me about the early history of ext2 having to do with the perfor- mance of ext2 compared to the BSD Fast File System (FFS). I had met Rob Kolstad, then president of BSDi, because of my interest in the AT&T lawsuit against the University of California and BSDi. BSDi was being sued for, among other things, Theodore Ts’o is the first having a phone number that could be spelled 800-ITS-UNIX. I thought that it was important North American Linux for the future of open source operating systems that AT&T lose that lawsuit. Kernel Developer, having That said, when I compared the performance of early versions of Linux to the current version started working with Linux of BSDi, I found that they were closely matched, with one glaring exception. Unpacking tar in September 1991. He also archives using Linux (likely .9) was blazingly fast compared to BSDi. I asked Rob, and he served as the tech lead for the MIT Kerberos explained that the issue had to do with synchronous writes, finally clearing up a mystery for me. V5 development team, and was the architect at IBM in charge of bringing real-time Linux Now I had a chance to ask Ted about the story from the Linux side, as well as other questions in support of real-time Java to the US Navy.
    [Show full text]
  • Freebsd Enterprise Storage Polish BSD User Group Welcome 2020/02/11 Freebsd Enterprise Storage
    FreeBSD Enterprise Storage Polish BSD User Group Welcome 2020/02/11 FreeBSD Enterprise Storage Sławomir Wojciech Wojtczak [email protected] vermaden.wordpress.com twitter.com/vermaden bsd.network/@vermaden https://is.gd/bsdstg FreeBSD Enterprise Storage Polish BSD User Group What is !nterprise" 2020/02/11 What is Enterprise Storage? The wikipedia.org/wiki/enterprise_storage page tells nothing about enterprise. Actually just redirects to wikipedia.org/wiki/data_storage page. The other wikipedia.org/wiki/computer_data_storage page also does the same. The wikipedia.org/wiki/enterprise is just meta page with lin s. FreeBSD Enterprise Storage Polish BSD User Group What is !nterprise" 2020/02/11 Common Charasteristics o Enterprise Storage ● Category that includes ser$ices/products designed &or !arge organizations. ● Can handle !arge "o!umes o data and !arge num%ers o sim#!tano#s users. ● 'n$olves centra!ized storage repositories such as SA( or NAS de$ices. ● )equires more time and experience%expertise to set up and operate. ● Generally costs more than consumer or small business storage de$ices. ● Generally o&&ers higher re!ia%i!it'%a"aila%i!it'%sca!a%i!it'. FreeBSD Enterprise Storage Polish BSD User Group What is !nterprise" 2020/02/11 EnterpriCe or EnterpriSe? DuckDuckGo does not pro$ide search results count +, Goog!e search &or enterprice word gi$es ~ 1 )00 000 results. Goog!e search &or enterprise word gi$es ~ 1 000 000 000 results ,1000 times more). ● /ost dictionaries &or enterprice word sends you to enterprise term. ● Given the *+,CE o& many enterprise solutions it could be enterPRICE 0 ● 0 or enterpri$e as well +.
    [Show full text]
  • Journaling File Systems
    JOURNALING FILE SYSTEMS CS124 – Operating Systems Spring 2021, Lecture 26 2 File System Robustness • The operating system keeps a cache of filesystem data • Secondary storage devices are much slower than main memory • Caching frequently-used disk blocks in memory yields significant performance improvements by avoiding disk-IO operations • ProBlem 1: Operating systems crash. Hardware fails. • ProBlem 2: Many filesystem operations involve multiple steps • Example: deleting a file minimally involves removing a directory entry, and updating the free map • May involve several other steps depending on filesystem design • If only some of these steps are successfully written to disk, filesystem corruption is highly likely 3 File System Robustness (2) • The OS should try to maintain the filesystem’s correctness • …at least, in some minimal way… • Example: ext2 filesystems maintain a “mount state” in the filesystem’s superBlock on disk • When filesystem is mounted, this value is set to indicate how the filesystem is mounted (e.g. read-only, etc.) • When the filesystem is cleanly unmounted, the mount-state is set to EXT2_VALID_FS to record that the filesystem is trustworthy • When OS starts up, if it sees an ext2 drive’s mount-state as not EXT2_VALID_FS, it knows something happened • The OS can take steps to verify the filesystem, and fix it if needed • Typically, this involves running the fsck system utility • “File System Consistency checK” • (Frequently, OSes also run scheduled filesystem checks too) 4 The fsck Utility • To verify the filesystem,
    [Show full text]
  • 62 ABI. See Application Binary Inter
    Index /,38–39, 348 implementation of, 509–510 .., 125, 307, 315, 327, 339–340, 345, 621 purpose of, 509 ., 307, 315, 339–340, 621 address, socket, 453–454 #!,62 address space. See virtual address space address structure A Internet, 454 local domain, 454 ABI. See application binary interface socket, 438, 645 absolute pathname, 39, 615, 635 address translation, 136, 615 accept system call, 440, 453, 455–457, 469, addresses, IPv6, 560–561 540, 564 adjtime system call, 66 definition, 439 administrative control, 123–129 access control, filesystem, 41 ad_start(), 280 access control list, 41, 125, 301, 615 adstrategy(), 280–281 access rights, 450, 460 advanced configuration and power interface, receiving, 463 266, 289 access system call, 254 advanced programmable interrupt controller, access vnode operator, 296 265, 268, 600 accounting, process resource, 58–59, 73–74, advisory locking, 232, 296, 615 111 advlock vnode operator, 296 accton, 608 AF_INET, 438, 564 ACL. See access control list AF_INET6, 564 ACPI. See advanced configuration and AH. See authentication header power interface ahc_action(), 279 active page list, 188 ahc_done(), 279 ad_done(), 281 aio_error system call, 230–231 address family, 450–451, 454, 473, 615 aio_read system call, 230, 260 address resolution protocol, 492, 509–510, aio_return system call, 231 567, 615–616 aio_suspend system call, 231 653 Front Page 653 654 Index aio_waitcomplete system call, 231 asymmetric cryptography, 581, 616 aio_write system call, 230, 260 asynchronous I/O, 229–231 algorithm asynchronous system trap, 105, 616 for disksort(), 224 AT A disk, 24, 265–267, 279–281, 286–289 elevator sorting, 224 I/O request, 280–281 mbuf storage-management, 447–448 layer, 279–281 for physical I/O, 221 AT A.
    [Show full text]
  • The Zettabyte File System
    The Zettabyte File System Jeff Bonwick, Matt Ahrens, Val Henson, Mark Maybee, Mark Shellenbaum Jeff[email protected] Abstract Today’s storage environment is radically different from that of the 1980s, yet many file systems In this paper we describe a new file system that continue to adhere to design decisions made when provides strong data integrity guarantees, simple 20 MB disks were considered big. Today, even a per- administration, and immense capacity. We show sonal computer can easily accommodate 2 terabytes that with a few changes to the traditional high-level of storage — that’s one ATA PCI card and eight file system architecture — including a redesign of 250 GB IDE disks, for a total cost of about $2500 the interface between the file system and volume (according to http://www.pricewatch.com) in late manager, pooled storage, a transactional copy-on- 2002 prices. Disk workloads have changed as a result write model, and self-validating checksums — we of aggressive caching of disk blocks in memory; since can eliminate many drawbacks of traditional file reads can be satisfied by the cache but writes must systems. We describe a general-purpose production- still go out to stable storage, write performance quality file system based on our new architecture, now dominates overall performance[8, 16]. These the Zettabyte File System (ZFS). Our new architec- are only two of the changes that have occurred ture reduces implementation complexity, allows new over the last few decades, but they alone warrant a performance improvements, and provides several reexamination of file system architecture from the useful new features almost as a side effect.
    [Show full text]
  • A BRIEF HISTORY of the BSD FAST FILE SYSTEM 9 June07login Press.Qxd:Login June 06 Volume 31 5/27/07 10:22 AM Page 10
    June07login_press.qxd:login June 06 Volume 31 5/27/07 10:22 AM Page 9 I FIRST STARTED WORKING ON THE UNIX file system with Bill Joy in the late MARSHALL KIRK MCKUSICK 1970s. I wrote the Fast File System, now called UFS, in the early 1980s. In this a brief history of article, I have written a survey of the work that I and others have done to the BSD Fast File improve the BSD file systems. Much of System this research has been incorporated into other file systems. Dr. Marshall Kirk McKusick writes books and articles, teaches classes on UNIX- and BSD-related subjects, and provides expert-witness testimony on software patent, trade secret, and copyright issues, particu - 1979: Early Filesystem Work larly those related to operating systems and file sys - tems. While at the University of California at Berke - The first work on the UNIX file system at Berke - ley, he implemented the 4.2BSD Fast File System and was the Research Computer Scientist at the Berkeley ley attempted to improve both the reliability and Computer Systems Research Group (CSRG) oversee - the throughput of the file system. The developers ing the development and release of 4.3BSD and improved reliability by staging modifications to 4.4BSD. critical filesystem information so that the modifi - [email protected] cations could be either completed or repaired cleanly by a program after a crash [14]. Doubling the block size of the file system improved the per - formance of the 4.0BSD file system by a factor of more than 2 when compared with the 3BSD file system.
    [Show full text]
  • Crash Consistency: FSCK and Journaling
    42 Crash Consistency: FSCK and Journaling As we’ve seen thus far, the file system manages a set of data structures to implement the expected abstractions: files, directories, and all of the other metadata needed to support the basic abstraction that we expect from a file system. Unlike most data structures (for example, those found in memory of a running program), file system data structures must persist, i.e., they must survive over the long haul, stored on devices that retain data despite power loss (such as hard disks or flash-based SSDs). One major challenge faced by a file system is how to update persis- tent data structures despite the presence of a power loss or system crash. Specifically, what happens if, right in the middle of updating on-disk structures, someone trips over the power cord and the machine loses power? Or the operating system encounters a bug and crashes? Because of power losses and crashes, updating a persistent data structure can be quite tricky, and leads to a new and interesting problem in file system implementation, known as the crash-consistency problem. This problem is quite simple to understand. Imagine you have to up- date two on-disk structures, A and B, in order to complete a particular operation. Because the disk only services a single request at a time, one of these requests will reach the disk first (either A or B). If the system crashes or loses power after one write completes, the on-disk structure will be left in an inconsistent state. And thus, we have a problem that all file systems need to solve: THE CRUX: HOW TO UPDATE THE DISK DESPITE CRASHES The system may crash or lose power between any two writes, and thus the on-disk state may only partially get updated.
    [Show full text]
  • OS Lecture 18
    Filesystem Reliability OS Lecture 18 UdS/TUKL WS 2015 MPI-SWS 1 What could go wrong? Expectation: stored data is persistent and correct. 1. Device failure: »disk crash (permanent failure) »bit flips on storage medium (What about host memory?) »transient or permanent sector read errors 1. OS crash or power failure during filesystem manipulation 2. Accidental data deletion/corruption by users. 3. Malicious tampering by attacker. MPI-SWS 2 Last Line of Defense: Backups! Good regular backups can help with all of these issues. »Once a day or more frequently to limit data loss. »Need a history of backups, not just latest snapshot (➞ bit errors, human error, attacks). »Backups should not be reachable from host, even if fully compromised (➞ attacks). »Downside: restoring from backup can be very slow. MPI-SWS 3 Dealing with Human Error Accidental data deletion or corruption due to confguration errors or software bugs. » Snapshotting filesystem: filesystem takes a (readonly) "snapshot" at regular intervals (e.g., every 24h). »copy-on-write makes this relatively cheap » Examples: ZFS, btrfs (Linux), HAMMER (DragonflyBSD) » Versioning filesystem: every file version is retained for some time (e.g., last 30 days) »Similar to Dropbox, but part of the low-level FS (➞ efciency) » Example: HAMMER retains a version every 30-60 seconds on sync MPI-SWS 4 Dealing with Device Failures (1/3) Partial failures: bit rot (= bit fips), bad sectors, and transient read errors. »Bit rot: aging effects and electro-magnetic interference (EMI) can corrupt data on disk
    [Show full text]