DOCSLIB.ORG

BTRFS: Investigation of Several Aspects Regarding Fragmentation

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
BTRFS: Investigation of Several Aspects Regarding Fragmentation BTRFS: AUTODEFRAG AND DEFRAGMENTATION Page 1 of 13 BTRFS: Investigation of several aspects regarding fragmentation Elijah D. Mirecki Athabasca University February 26, 2014 BTRFS: AUTODEFRAG AND DEFRAGMENTATION Page 2 of 13 Abstract BTRFS defragmentation is currently an active topic of discussion among operating system designers. Defragmentation is especially important for BTRFS because of its copy-on-write (COW) nature, which causes data extents to be scattered all over the system when writing occurs. This paper provides an investigation of the current BTRFS defragmentation algorithm as well as the autodefrag algorithm. Several illustrations have been created to represent how the defragmentation algorithm works at a high level. Discussion about possible solutions and ideas for defragmentation and autodefrag is also included. The main problem with the current defragmentation algorithm is that it sometimes merges more extents than are needed to provide relatively unfragmented files. Another problem is that when choosing which files to defragment, flash drives and disk drives are treated the same. This is an issue because the two drive types should have different goals. Disk drives aim for contiguous space, while flash drives need less writes to prolong their lifetime. The solutions and ideas discussed in this paper should theoretically allow defragmentation to merge extents more appropriately as well as meeting the goals for both flash and disk drives. This paper should be followed by simulations comparing several areas of defragmentation and file selection algorithms. Some implementation details are provided as well in case the simulations return positive. Several other filesystems are touched upon, including ext4, HFS+ and NTFS. Defragmentation and their fragmentation situation are both covered and compared briefly. Introduction and background Introduction to BTRFS BTRFS is a extent based copy-on-write (COW) file system. An extent is basically an array of blocks. Instead of the filesystem metadata leaf nodes pointing at each individual block, only a single block pointer and an extent length must be set in the inode. With COW, when a block is modified, a write does not occur on an existing extent, but is written to a new extent on another part of the drive. Assuming a written block has occurred in the middle of an existing extent, the extent will have to be split into multiple parts. The inode will point to the beginning of the old extent, to the new extent created by the write, and then a third pointer to the end of the old extent. The data in the middle of the old extent may then be deallocated. This allows a much more versatile file system, but results in the system becoming much more fragmented. Some of the main features of BTRFS include compression, clones, and snapshots. Snapshots allow the filesystem state to be saved for backups or archives. Clones are copies of files that initially share the same extents. As changes are made to the clone or the original file, the files will slowly share less with each other due to COW. Since BTRFS is a COW system, snapshots and clones are able to be created extremely quickly by simply creating new file pointers. Once there are no more pointers to an BTRFS: AUTODEFRAG AND DEFRAGMENTATION Page 3 of 13 extent on the drive, it will be deallocated. BTRFS is also transaction based. Write operations are not completed immediately, but are placed in a transaction located in memory. The system commits all of the operations at regular intervals, or when otherwise required such as if the system is shutting down. This allows compression and write placement optimization to take place while the transaction resides in memory, before the data is even written to the disk. This is also a form of the delayed-allocation method – a method designed with the intent of reducing fragmentation by combining several write operations before the actual write occurs. As mentioned above, BTRFS suffers greatly from fragmentation. For enterprise systems, the built-in BTRFS defragmenter may be run according to the system schedule. But for a general solution, BTRFS implements a mechanism called autodefrag. Autodefrag works by automatically choosing which files to defragment as they are modified. Files which are chosen will be defragmented along with each transaction being performed. This is a good solution for most systems because it keeps fragmentation low when small writes occur in files, which is the circumstance that generally causes the most file fragmentation. Motivation for Research As I was reading about ZFS on miscellaneous websites, I stumbled upon several other file systems such as BTRFS and XFS. At the time I started reading some more articles about BTRFS and the basic concepts; Copy-on-write, B-tree data structures, etc. Once it was required that I find a topic to research, my first thought was BTRFS, because I was quite interested with it in the past. I then headed over to the Athabasca online library, where I found a recent article which covered many topics of BTRFS from several levels (Resource 1). As I was reading the article, it seemed as though the defragmentation system has been one of the largest problems with BTRFS currently. From what I have gathered in several articles, the autodefrag algorithm is the solution for selecting which files should be defragmented by default. Another problem that I realized was that the current defragmentation algorithm tends to strip out sharing of extents with clones and snapshots. This is a big issue, since clones and snapshots are core elements of the system. I also discovered that the defragmentation algorithms treat flash drives the same as disk drives. This stood out as yet another major problem, because flash drives behave much different. On a flash drive, access time is basically a constant time, so contiguous data doesn't matter nearly as much. Problems Excess Defragmentation. The current defragmentation has a major flaw. When a file begins to be defragmented, the ENTIRE file will be scanned for extents that are worthy of being merged. There are several problems with this approach: 1. Performance will be affected. 2. A lot of sharing with be lost with clones and snapshots. 3. Drive lifetime will be reduced. The performance will be degraded for several reasons. One reason is that the algorithm will have to scan the entire file, instead of just the part of the file that needs attention (the section of the file which has been modified). Another reason is of course, because extra extents will have to be COWed. BTRFS: AUTODEFRAG AND DEFRAGMENTATION Page 4 of 13 In a large file such as a database, this could be a massive overhead. Much more sharing with snapshots and clones will also be lost unnecessarily. Defragmenting only the small section of a file which has been modified will most likely not require a lot of COW operations. If an entire file is defragmented, then extents all over the file will be merged, and therefore duplicated among clones and snapshots. Drive lifetime will be decreased when more writing occurs. Drive lifetime is especially critical to flash drives, which are starting to dominate the secondary storage market these days. Autodefrag Optimization for Flash Drives Currently, the autodefrag algorithm basically just checks for the size of a COW operation to determine whether or not to add a file to the defragmentation list, despite which kind of drive is in use. The problem here is that with flash drives, defragmentation is not as big of an issue as on disk drives, so we don't want to defragment the file as often. The main problem is that flash drives have a shorter life span than disk drives, but don't provide a means of factoring this into account. Because the defragmentation algorithm will behave the same on a flash drive, it will try to merge extents to form contiguous data, which will wear out the flash drives if it is overdone. Another more minor problem is that performance may be impacted if more files are defragmented. The algorithm should be smarter to handle flash drive autodefrag differently to accommodate the factor of drive life span. Methods To begin exploring the circumstances of fragmentation in BTRFS, several documents and papers were read. The documents that were found covered a broad range of how BTRFS works, but were not specifically about the fragmentation issue or defragmentation algorithms, which were covered briefly. A general knowledge and understanding about how BTRFS works as a whole was gained from resource 1 mainly, but other miscellaneous articles and mailing lists were used as well. Some of the main tactics used to explore the defragmentation algorithm were simply to poke around the Linux source code itself (version 3.12.8). The source code is a big place, so it could be difficult to find where to start. To find the starting point, the git logs for the Linux source code were searched regarding BTRFS defragmentation and autodefrag [5]. Once several functions were found that seemed to be reasonable starting points for exploring defragmentation and autodefreg, those functions were examined for comments and other method calls. The Linux Cross Reference (LXR) [6] was used extensively throughout exploring the source code. Data structures and functions which were related to autodefrag and defragramentation were traced and examined. Results and findings Findings The main starting places in the Linux source code that I found were the functions btrfs_ioctl_defrag(), btrfs_defrag_file() and cow_file_range(). The cow_file_range() function is what performs the COW operations. While performing these operations, it also checks if a file should be defragmented according to the autodefrag policy. The current code will add a file to the list if a write is made that is under 64KB, and if the write is only part of the file.
Load more

Recommended publications
  • Welcome to the Ally Skills Workshop
    @frameshiftllc Welcome to the Ally Skills Workshop Please fill out a name tag & include the pronouns you normally use. Examples: she/her/hers he/him/his they/them/theirs Pronouns Ally Skills Workshop Valerie Aurora http://frameshiftconsulting.com/ally-skills-workshop/ CC BY-SA Frame Shift Consulting LLC, Dr. Sheila Addison, The Ada Initiative @frameshiftllc Format of the workshop ● 30 minute introduction ● 45 minute group discussion of scenarios ● 15 minute break ● 75 minute group discussion of scenarios ● 15 minute wrap-up ~3 hours total @frameshiftllc SO LONG! 2 hour-long workshop: most common complaint was "Too short!" 3 hour-long workshop: only a few complaints that it was too short https://flic.kr/p/7NYUA3 CC BY-SA Toshiyuki IMAI @frameshiftllc Valerie Aurora Founder Frame Shift Consulting Taught ally skills to 2500+ people in Spain, Germany, Australia, Ireland, Sweden, Mexico, New Zealand, etc. Linux kernel and file systems developer for 10+ years Valerie Aurora @frameshiftllc Let’s talk about technical privilege We are more likely to listen to people who "are technical" … but we shouldn’t be "Technical" is more likely to be granted to white men I am using my technical privilege https://frYERZelic.kr/p/ CC BY @sage_solar to end technical privilege! @frameshiftllc What is an ally? Some terminology first: Privilege: an unearned advantage given by society to some people but not all Oppression: systemic, pervasive inequality that is present throughout society, that benefits people with more privilege and harms those with fewer privileges
    [Show full text]
  • Performance Evaluation of Filesystems Compression Features
    UNIVERSITY OF OSLO Department of Informatics Performance Evaluation Of FileSystems Compression Features Master Thesis In the field of Network and System Administration Solomon Legesse Oslo and Akerhus University College (hioa) In collaboration with University of Oslo (UiO) May 20, 2014 1 Performance Evaluation Of FileSystems Compression Features Master Thesis In the field of Network and System Administration Solomon Legesse Oslo and Akerhus University College (hioa) In collaboration with University of Oslo (UiO) May 20, 2014 Abstract The Linux operating system already provide a vast number of filesystems to the user community. In general, having a filesystem that can provide scala- bility, excellent performance and reliability is a requirement, especially in the lights of the very large data size being utilized by most IT data centers. Re- cently modern file systems has begun to include transparent compression as main features in their design strategy. Transparent compression is the method of compressing and decompressing data so that it takes relatively less space. Transparent compression can also improve IO performance by reducing IO traffic and seek distance and has a negative impact on performance only when single-thread I/O latency is critical. Two of the newer filesystem technologies that aim at addressing todays IO challenges are ZFS and Btrfs. Using high speed transparent compression algorithms like LZ4 and LZO with Btrfs and Zfs can greatly help to improve IO performance. The goal of this paper is threefold. 1st, to evaluate the impact of transparent compression on perfor- mance for Btrfs and ZFS, respectively. 2nd, to compare the two file system compression feature on performance.
    [Show full text]
  • The Kernel Report
    The Kernel Report RTLWS 11 edition Jonathan Corbet LWN.net [email protected] “Famous last words, but the actual patch volume _has_ to drop off one day. We have to finish this thing one day." -- Andrew Morton September, 2005 (2.6.14) 2.6.27 -> 2.6.31++ (October 9, 2008 to September 18, 2009) 48,000 changesets merged 2,500 developers 400 employers The kernel grew by 2.5 million lines 2.6.27 -> 2.6.31++ (October 9, 2008 to September 18, 2009) 48,000 changesets merged 2,500 developers 400 employers The kernel grew by 2.5 million lines That come out to: 140 changesets merged per day 7267 lines of code added every day The employer stats None 19% Atheros 2% Red Hat 12% academics 2% Intel 7% Analog Devices 2% IBM 6% AMD 1% Novell 6% Nokia 1% unknown 5% Wolfson Micro 1% Oracle 4% Vyatta 1% consultants 3% HP 1% Fujitsu 2% Parallels 1% Renesas Tech 2% Sun 1% 2.6.27 (October 9, 2008) Ftrace UBIFS Multiqueue networking gspca video driver set Block layer integrity checking 2.6.28 (December 24, 2008) GEM graphics memory manager ext4 is no longer experimental -staging tree Wireless USB Container freezer Tracepoints 2.6.29 (March 23, 2009) Kernel mode setting Filesystems Btrfs Squashfs WIMAX support 4096 CPU support 2.6.30 (June 9) TOMOYO Linux Object storage device support Integrity measurement FS-Cache ext4 robustness fixes Nilfs R6xx/R7xx graphics support preadv()/pwritev() Adaptive spinning Threaded interrupt mutexes handlers 2.6.31 (September 9) Performance counter support Char devices in user space Kmemleak fsnotify infrastructure TTM and Radeon KMS support Storage topology ...about finished? ...about finished? ...so what's left? 2.6.32 (early December) Devtmpfs Lots of block scalability work Performance counter improvements Scheduler tweaks Kernel Shared Memory HWPOISON Networking “Based on all the measurements I'm aware of, Linux has the fastest & most complete stack of any OS.” -- Van Jacobson But..
    [Show full text]
  • A Method of Merging Vmware Disk Images Through File System Unification by Sarah X
    A Method of Merging VMware Disk Images MASSACHUSETTS INSTITUTE through File System Unification OF TECHN4OLOGY by DEC 16 2010 Sarah X. Cheng LIBRARIES S.B., Computer Science and Engineering, Massachusetts Institute of Technology (2004) Submitted to the Department of Electrical Engineering and Computer Science in partial fulfillment of the requirements for the degree of Master of Engineering in Electrical Engineering and Computer Science at the MASSACHUSETTS INSTITUTE OF TECHNOLOGY ACHIVES February 2011 © Massachusetts Institute of Technology 2011. All rights reserved. -7--) Author ...... ...... Department of Electrical Enginer' d Computer Science October 5, 2010 ... s ..... Certified by............. Stephen A. Ward Professor Thesis Supervisor Accepted by........ Dr. Christopher J. Terman Chairman, Master of Engineering Thesis Committee 2 A Method of Merging VMware Disk Images through File System Unification by Sarah X. Cheng Submitted to the Department of Electrical Engineering and Computer Science on October 5, 2010, in partial fulfillment of the requirements for the degree of Master of Engineering in Electrical Engineering and Computer Science Abstract This thesis describes a method of merging the contents of two VMware disk images by merging the file systems therein. Thus, two initially disparate file systems are joined to appear and behave as a single file system. The problem of file system namespace unification is not a new one, with predecessors dating as far back as 1988 to present-day descendants such as UnionFS and union mounts. All deal with the same major issues - merging directory contents of source branches and handling any naming conflicts (namespace de-duplication), and allowing top-level edits of file system unions in presence of read-only source branches (copy-on-write).
    [Show full text]
  • How to Respond to Code of Conduct Reports
    License and attribution Written by Valerie Aurora, based on a short guide written by Mary Gardiner Edited by Annalee Flower Horne https://www.flowerhorne.com/ ​ Copyright © 2019 Valerie Aurora https://valerieaurora.org/ ​ Copyright © 2012 Mary Gardiner https://mary.gardiner.id.au/ ​ Cover image © 2018 Mary Gardiner https://mary.gardiner.id.au/ ​ CC BY-SA 4.0 Valerie Aurora, Mary Gardiner, Annalee Flower Horne, DjangoCon EU, Write the ​ Docs EU 2016, PyGotham 2017 https://creativecommons.org/licenses/by-sa/4.0/ ​ Published by Frame Shift Consulting LLC https://frameshiftconsulting.com ​ ISBN: 9781386922575 Version history 2018-09-15: Version 0.9: Draft release for review 2018-11-26: Version 0.99: Pre-release version for review 2018-11-28: Version 1.0: First edition 2019-01-08: Version 1.1: Fix typos, simplify formatting for better ebook conversion, add/remove links, add email sign-up link, add more incident response guide examples, add ISBN Table of contents Introduction If you are in a hurry How to use this guide About the authors Terminology Chapter 1: Code of conduct theory Purpose of a code of conduct What a code of conduct should contain How a code of conduct works Education Norm-following Attraction and repulsion Deterrence Boundary setting The Paradox of Tolerance What a code of conduct can't do Codes of conduct govern community spaces Violations must have meaningful consequences Codes of conduct must apply to powerful people Visible enforcement is required Summary Chapter 2: Preparing to enforce a code of conduct Publicizing the
    [Show full text]
  • IBM Research Report
    RJ10501 (ALM1207-004) July 9, 2012 Computer Science IBM Research Report BTRFS: The Linux B-tree Filesystem Ohad Rodeh IBM Research Division Almaden Research Center 650 Harry Road San Jose, CA 95120-6099 USA Josef Bacik, Chris Mason FusionIO Research Division Almaden - Austin - Beijing - Cambridge - Haifa - India - T. J. Watson - Tokyo - Zurich LIMITED DISTRIBUTION NOTICE: This report has been submitted for publication outside of IBM and will probably be copyrighted if accepted for publication. It has been issued as a Research Report for early dissemination of its contents. In view of the transfer of copyright to the outside publisher, its distribution outside of IBM prior to publication should be limited to peer communications and specific requests. After outside publication, requests should be filled only by reprints or legally obtained copies of the article (e.g. , payment of royalties). Copies may be requested from IBM T. J. Watson Research Center , P. O. Box 218, Yorktown Heights, NY 10598 USA (email: [email protected]). Some reports are available on the internet at http://domino.watson.ibm.com/library/CyberDig.nsf/home . BTRFS: The Linux B-tree Filesystem Ohad Rodeh Josef Bacik Chris Mason IBM FusionIO FusionIO Abstract BTRFS is a Linux filesystem, headed towards mainline default sta- tus. It is based on copy-on-write, allowing for efficient snapshots and clones. It uses b-trees as its main on-disk data-structure. The de- sign goal is to work well for many use cases and workloads. To this end, much effort has been directed to maintaining even performance as the filesystem ages, rather than trying to support a particular narrow benchmark use case.
    [Show full text]
  • Gender Dimensions of Free and Open Source Development
    THE MAKING OF LABORER SUBJECTIVITY AND KNOWLEDGE IN THE INFORMATION INDUSTRY: GENDER DIMENSIONS OF FREE AND OPEN SOURCE DEVELOPMENT Yeon Ju Oh A Dissertation Submitted to the Graduate College of Bowling Green State University in partial fulfillment of the requirements for the degree of DOCTOR OF PHILOSOPHY August 2013 Committee: Radhika Gajjala, Advisor Deborah G. Wooldridge Graduate Faculty Representative Oliver Boyd-Barrett Victoria S. Ekstrand ii ABSTRACT Radhika Gajjala, Advisor This study examines female software developers as knowledge laborers with a special emphasis on free and open source software (FOSS) development. In examining female developers as knowledge laborers, this study focuses on both labor and knowledge. Women’s low participation in FOSS development is not an issue of recent years, but a consequence of women’s overall status in the computing field over the last three decades. In order to explicate women’s low participation in FOSS development, a broader historical and economic analysis is needed. Thus, this study explores the historical context of computer science education and industry in the 1980s since this is when the groundwork for FOSS development was laid. Furthermore, the power of cultural discourses that maintain and reinforce the gendered construction of FOSS development is discussed to unpack how the gendered construction is interrelated with the labor relations in the knowledge industry. In addition to the labor relations in FOSS development, this study attends to the knowledge produced by FOSS development. Knowledge gains importance as a sum of values of the knowledge producers. Source codes written by software developers turn into products that engage users with certain utility.
    [Show full text]
  • R&S®TCE900 Open Source Acknowledgment
    R&S®TCE900 Series xx9 Transmitter Open Source Acknowledgment 2109.3806.00 – 02 7TA /RL/1/EN 01.00 / 3575.4620.02 M: - T - PAD Open Source Acknowledgment R&S TCE900 Introduction Contents 1 Introduction ......................................................................................... 3 1.1 Disclaimer ..................................................................................................................... 3 1.2 How to obtain the source code .................................................................................. 3 2 Software packages ............................................................................. 4 3 Verbatim license texts ........................................................................ 9 3.1 APACHE-LICENSE-2.0.txt ........................................................................................... 9 3.2 BSD-2-clause.txt ........................................................................................................12 3.3 BSD-3-clause.txt ........................................................................................................13 3.4 BSD-4-clause.txt ........................................................................................................14 3.5 GPLv2.txt ....................................................................................................................14 3.6 GPLv3.txt ....................................................................................................................21 3.7 LGPLv2.1.txt ...............................................................................................................33
    [Show full text]
  • State of the Union When You Don’T Need Union Mounts
    State of the Union When you don't need Union Mounts Jan Blunck Novell [email protected] 30. October 2009 What Union? I Not the European Union ... this is Dresden not Brussels I This is about Filesystems I In particular about Filesystem Namespace Unification What Union? I Not the European Union ... this is Dresden not Brussels I This is about Filesystems I In particular about Filesystem Namespace Unification What Union? I Not the European Union ... this is Dresden not Brussels I This is about Filesystems I In particular about Filesystem Namespace Unification Outline Introduction Where is the Problem? Unioning Filesystems UnionFS Another UnionFS UnionFS-FUSE mini fo Union Mount You probably don't need Union Mounts Device-Mapper Snapshot Delta Filesystem CLIC Filesystem SquashFS Fake Write Support Shared root filessytem - NFS Root Shared root filessytem - XIP Whats left to do than? Thanks Disclaimer I'm the author of the VFS based Union Mount patches. That somehow makes me biased. I'll try my very best though ... Where is the Problem? POSIX Requirements I seek to cookie POSIX is missing I whiteout filetype DT WHITEOUT I topology of mount tree I open (directories) by inode number Where is the Problem? POSIX Requirements I seek to cookie POSIX is missing I whiteout filetype DT WHITEOUT I topology of mount tree I open (directories) by inode number Where is the Problem? I NFS Sucks ftp://ftp.lst.de/pub/people/okir/papers/2006-OLS/ nfs-sucks-slides.pdf Where is the Problem? I NFS Sucks ftp://ftp.lst.de/pub/people/okir/papers/2006-OLS/ nfs-sucks-slides.pdf
    [Show full text]
  • Tlx9 Open Source Acknowledgment
    R&S®TLx9 Series xx9 Transmitter Open Source Acknowledgment 2507.3060.00 – 01 7TA /RL/1/EN 01.00 / 3575.4620.02 M: - T - PAD Open Source Acknowledgment R&S TLx9 Introduction Contents 1 Introduction ......................................................................................... 3 1.1 Disclaimer ..................................................................................................................... 3 1.2 How to obtain the source code .................................................................................. 3 2 Software packages ............................................................................. 4 3 Verbatim license texts ........................................................................ 9 3.1 APACHE-LICENSE-2.0.txt ........................................................................................... 9 3.2 BSD-2-clause.txt ........................................................................................................12 3.3 BSD-3-clause.txt ........................................................................................................13 3.4 BSD-4-clause.txt ........................................................................................................14 3.5 GPLv2.txt ....................................................................................................................14 3.6 GPLv3.txt ....................................................................................................................21 3.7 LGPLv2.1.txt ...............................................................................................................33
    [Show full text]
  • Generating Realistic Impressions for File-System Benchmarking
    Generating Realistic Impressions for File-System Benchmarking Nitin Agrawal, Andrea C. Arpaci-Dusseau and Remzi H. Arpaci-Dusseau Department of Computer Sciences, University of Wisconsin-Madison {nitina, dusseau, remzi}@cs.wisc.edu Abstract of the target usage scenario. Several factors contribute to file-system state, important amongst them are the in- The performance of file systems and related software de- memory state (contents of the buffer cache), the on-disk pends on characteristics of the underlying file-system im- state (disk layout and fragmentation) and the characteris- age (i.e., file-system metadata and file contents). Un- tics of the file-system image (files and directories belong- fortunately, rather than benchmarking with realistic file- ing to the namespace and file contents). system images, most system designers and evaluators rely on ad hoc assumptions and (often inaccurate) rules One well understood contributor to state is the in- memory state of the file system. Previous work has of thumb. Furthermore, the lack of standardization and shown that the contents of the cache can have signifi- reproducibility makes file system benchmarking ineffec- cant impact on the performance results [11]. Therefore, tive. To remedy these problems, we develop Impressions, system initialization during benchmarking typically con- a framework to generate statistically accurate file-system images with realistic metadata and content. Impressions sists of a cache “warm-up” phase wherein the workload is run for some time prior to the actual measurement is flexible, supporting user-specified constraints on vari- phase. Another important factor is the on-disk state of ous file-system parameters using a number of statistical techniques to generate consistent images.
    [Show full text]
  • An Analysis of Compare-By-Hash
    Guidelines for Using Compare-by-hash Val Henson Richard Henderson IBM, Inc. Red Hat, Inc. [email protected] [email protected] Abstract hash collision. Recently, a new technique called compare-by-hash Compare-by-hash made its first appearance in 1996 has become popular. Compare-by-hash is a method in rsync[26], a tool for synchronizing files over of content-based addressing in which data is identi- the network, and has since been used in a num- fied only by the cryptographic hash of its contents. ber of projects, including LBFS[14], a distributed Hash collisions are ignored, with the justification file system; Stanford’s virtual computer migration that they occur less often than many kinds of hard- project[19]; Pastiche[7], an automated backup sys- ware errors. Compare-by-hash is a powerful, ver- tem; OpenCM[21], a configuration management sys- satile tool in the software architect’s bag of tricks, tem; and CASPER[24], a block cache for distributed but it is also poorly understood and frequently mis- file systems. Venti, a block archival storage sys- used. The consequences of misuse range from signif- tem for Plan 9, is described as using compare-by- icant performance degradation to permanent, unre- hash[18] but as implemented it checks for collisions coverable data corruption or loss. The proper use of on writes[9]. Compare-by-hash is also used in a compare-by-hash is a subject of debate[10, 29], but wide variety of document retrieval systems, content- recent results in the field of cryptographic hash func- caching proxies, and many other commercial soft- tion analysis, including the breaking of MD5[28] and ware products.
    [Show full text]