The Full Path to Full-Path Indexing Yang Zhan, The University of North Carolina at Chapel Hill; Alex Conway, Rutgers University; Yizheng Jiao, The University of North Carolina at Chapel Hill; Eric Knorr, Rutgers University; Michael A. Bender, Stony Brook University; Martin Farach-Colton, Rutgers University; William Jannen, Williams College; Rob Johnson, VMware Research; Donald E. Porter, The University of North Carolina at Chapel Hill; Jun Yuan, Stony Brook University https://www.usenix.org/conference/fast18/presentation/zhan This paper is included in the Proceedings of the 16th USENIX Conference on File and Storage Technologies. February 12–15, 2018 • Oakland, CA, USA ISBN 978-1-931971-42-3 Open access to the Proceedings of the 16th USENIX Conference on File and Storage Technologies is sponsored by USENIX. The Full Path to Full-Path Indexing Yang Zhan, Alex Conway†, Yizheng Jiao, Eric Knorr†, Michael A. Bender‡, Martin Farach-Colton†, William Jannen¶, Rob Johnson∗, Donald E. Porter, Jun Yuan‡ The University of North Carolina at Chapel Hill, †Rutgers University, ‡Stony Brook University, ¶Williams College, ∗VMware Research Abstract cographic order by the full-path names of files and di- rectories. With this design, scans of any directory sub- Full-path indexing can improve I/O efficiency for tree (e.g., ls -R or grep -r) should run at near disk workloads that operate on data organized using tradi- bandwidth. The challenge is maintaining full-path order tional, hierarchical directories, because data is placed on as the file system changes. Prior work [15, 22, 23] has persistent storage in scan order. Prior results indicate, shown that the combination of write-optimization [5–7, however, that renames in a local file system with full- 9–11, 36, 43, 44] with full-path indexing can realize effi- path indexing are prohibitively expensive. cient implementations of many file system updates, such This paper shows how to use full-path indexing in a as creating or removing files, but a few operations still file system to realize fast directory scans, writes, and re- have prohibitively high overheads. names. The paper introduces a range-rename mechanism The Achilles’ heel of full-path indexing is the perfor- for efficient key-space changes in a write-optimized dic- mance of renaming large files and directories. For in- tionary. This mechanism is encapsulated in the key-value stance, renaming a large directory changes the path to API and simplifies the overall file system design. every file in the subtree rooted at that directory, which We implemented this mechanism in BetrFS, an in- changes the depth-first search order. Competitive rename kernel, local file system for Linux. This new version, performance in a full-path indexed file system requires BetrFS 0.4, performs recursive greps 1.5x faster and ran- making these changes in an I/O-efficient manner. dom writes 1.2x faster than BetrFS 0.3, but renames The primary contribution of this paper is showing that are competitive with indirection-based file systems for one can, in fact, use full-path indexing in a file system a range of sizes. BetrFS 0.4 outperforms BetrFS 0.3, as without introducing unreasonable rename costs. A file well as traditional file systems, such as ext4, XFS, and system can use full-path indexing to improve directory ZFS, across a variety of workloads. locality—and still have efficient renames. Previous full-path indexing. 1 Introduction The first version of Be- trFS [22, 23] (v0.1), explored full-path indexing. Be- Today’s general-purpose file systems do not fully utilize trFS uses a write-optimized dictionary to ensure fast up- the bandwidth of the underlying hardware. For exam- dates of large and small data and metadata, as well as ple, ext4 can write large files at near disk bandwidth but fast scans of files and directory-tree data and metadata. typically creates small files at less than 3% of disk band- Specifically, BetrFS uses two Be -trees [7, 10] as persis- width. Similarly, ext4 can read large files at near disk tent key-value stores, where the keys are full path names bandwidth, but scanning directories with many small to files and the values are file contents and metadata, re- files has performance that ages over time. For instance, a spectively. Be -trees organize data on disk such that log- git version-control workload can degrade ext4 scan per- ically contiguous key ranges can be accessed via large, formance by up to 15× [14, 55]. sequential I/Os. Be -trees aggregate small updates into At the heart of this issue is how data is organized, or large, sequential I/Os, ensuring efficient writes. indexed, on disk. The most common design pattern for This design established the promise of full-path index- modern file systems is to use a form of indirection, such ing, when combined with Be -trees. Recursive greps run as inodes, between the name of a file in a directory and its 3.8x faster than in the best standard file system. File cre- physical placement on disk. Indirection simplifies imple- ation runs 2.6x faster. Small, random writes to a file run mentation of some metadata operations, such as renames 68.2x faster. or file creates, but the contents of the file system can end However, renaming a directory has predictably miser- up scattered over the disk in the worst case. Cylinder able performance [22, 23]. For example, renaming the groups and other best-effort heuristics [32] are designed Linux source tree, which must delete and reinsert all the to mitigate this scattering. data to preserve locality, takes 21.2s in BetrFS 0.1, as Full-path indexing is an alternative to indirection, compared to 0.1s in btrfs. known to have good performance on nearly all opera- Relative-path indexing. BetrFS 0.2 backed away from tions. File systems that use full-path indexing store data full-path indexing and introduced zoning [54, 55]. Zon- and metadata in depth-first-search order, that is, lexi- ing is a schema-level change that implements relative- USENIX Association 16th USENIX Conference on File and Storage Technologies 123 path indexing. In relative-path indexing, each file or di- unchanged. Our main technical innovation is an efficient rectory is indexed relative to a local neighborhood in the implementation of range rename in a Be -tree. Specifi- directory tree. See Section 2.2 for details. cally, we reduce the cost from the size of the subtree to Zoning strikes a “sweet spot” on the spectrum between the height of the subtree. indirection and full-path indexing. Large file and di- Using range rename, BetrFS 0.4 returns to a sim- rectory renames are comparable to indirection-based file ple schema for mapping file system operations onto systems, and a sequential scan is at least 2x faster than key/value operations; this in turn consolidates all place- inode-based file systems but 1.5x slower than BetrFS 0.1. ment decisions and locality optimizations in one place. There are, however, a number of significant, diffuse The result is simpler code with less ancillary metadata costs to relative-path indexing, which tax the perfor- to maintain, leading to better performance on a range of mance of seemingly unrelated operations. For instance, seemingly unrelated operations. two-thirds of the way through the TokuBench bench- The technical insight behind efficient Be -tree range re- mark, BetrFS 0.2 shows a sudden, precipitous drop in name is a method for performing large renames by direct cumulative throughput for small file creations, which can manipulation of the Be -tree. Zoning shows us that small be attributed to the cost of maintaining zones. key ranges can be deleted and reinserted cheaply. For Perhaps the most intangible cost of zoning is that it large key ranges, range rename is implemented by slic- introduces complexity into the system and thus hides op- ing the tree at the source and destination. Once the source timization opportunities. In a full-path file system, one subtree is isolated, a pointer swing moves the renamed can implement nearly all file system operations as simple section of keyspace to its destination. The asymptotic point or range operations. Adding indirection breaks this cost of such tree surgery is proportional to the height, simple mapping. Indirection generally causes file sys- rather than the size, of the tree. tem operations to map onto more key/value operations Once the Be -tree has its new structure, another chal- and often introduces reads before writes. Because reads lenge is efficiently changing the pivots and keys to their are slower than writes in a write-optimized file system, new values. In a standard Be -tree, each node stores the making writes depend upon reads forgoes some of the full path keys; thus, a straightforward implementation of potential performance benefits of write-optimization. range rename must rewrite the entire subtree. Consider rm -r, for example. With full-path in- We present a method that reduces the work of updat- dexing, one can implement this operation with a single ing keys by removing the redundancy in prefixes shared range-delete message, which incurs almost no latency by many keys. This approach is called key lifting (x5). and requires a single synchronous write to become per- A lifted Be -tree encodes its pivots and keys such that sistent [54, 55]. Using a single range-delete message the values of these strings are defined by the path taken also unlocks optimizations internal to the key/value store, to reach the node containing the string. Using this ap- such as freeing a dead leaf without first reading it from proach, the number of paths that need to be modified in disk.