
FILE SYSTEMS AND STORAGE Evolving Ext4 for Shingled Disks ABUTALIB AGHAYEV, THEODORE TS’O, GARTH GIBSON, AND PETER DESNOYERS Abutalib Aghayev is a PhD ulti-terabyte hard disks today use Shingled Magnetic Recording student in the Computer (SMR), a technique that increases capacity at the expense of more Science Department at costly random writes. We introduce ext4-lazy, a small change to Carnegie Mellon University. M His research interests include the popular Linux ext4 file system that eliminates a major source of random operating systems, file and storage systems, writes—the metadata writeback—significantly improving performance on and, recently, distributed machine learning SMR disks in general, as well as on conventional disks for metadata-heavy systems. [email protected] workloads in particular. In this article, we briefly explain why SMR disks suffer under random writes and how ext4-lazy helps. Theodore Ts’o started working with Linux in September 1991 To cope with the exponential growth of data, as well as to stay competitive with NAND and is the first North American flash-based solid state drives (SSDs), hard disk vendors are researching capacity-increasing Linux kernel developer. He technologies. Shingled Magnetic Recording (SMR) is one such technique that allows disk also served as the tech lead manufacturers to increase areal density with existing fabrication methods. So far, the industry for the MIT Kerberos V5 development team has introduced two kinds of SMR disks: Drive-Managed (DM-SMR) and Host-Managed (HM- and as a chair of the IP Security working group SMR). HM-SMR disks have a novel backward-incompatible interface that requires changes to at the IETF. He previously was CTO for the the I/O stack and, therefore, are not widely deployed. DM-SMR disks, on the other hand, are a Linux Foundation and is currently employed drop-in replacement for Conventional Magnetic Recording (CMR) disks that offer high capac- at Google. Theodore is a Debian Developer, ity with the traditional block interface. Millions of DM-SMR disks have been shipped; in the and maintains the ext4 file system in the Linux rest of the article, therefore, we will use SMR disk as a shorthand for DM-SMR disk. kernel. He is also the maintainer and original If you buy a multi-terabyte disk today, there is a good chance that it is an SMR disk in dis- author of the e2fsprogs userspace utilities for guise, which is easy to tell: unlike CMR disks, SMR disks suffer performance degradation the ext2, ext3, and ext4 file systems. when subjected to continuous random write traffic, as Figure 1 shows. [email protected] One approach to adopting SMR disks is to develop a file system from scratch based on their Garth Gibson is a Professor performance characteristics. But file systems are complex and critical pieces of code that of Computer Science and an take years to mature. Therefore, we take an evolutionary approach to adopting these disks: Associate Dean in the School of we make a small change to the popular Linux file system, ext4, that significantly improves Computer Science at Carnegie its performance on SMR disks by avoiding random metadata writes. Mellon University. Garth’s We introduce a simple technique that we call lazy writeback journaling, and we call a ver- research is split between scalable storage sion of ext4 using our journaling technique ext4-lazy. Like other journaling file systems, by systems and distributed machine learning default ext4 writes metadata twice; as Figure 2a shows, it first writes the metadata block to systems, and he has had his hand in the a temporary location J in the journal and then marks the block as dirty in memory. Once the creation of the RAID taxonomy, the Panasas block has been in memory for long enough, a writeback thread writes the block to its static PanFS parallel file system, the IETF NFS v4.1 location S, resulting in a random write. Although metadata writeback is typically a small parallel NFS extensions, and the USENIX portion of a workload, it results in many random writes. Ext4-lazy, on the other hand, marks Conference on File and Storage Technologies the block as clean after writing it to the journal, to prevent the writeback, and inserts a map- (FAST). [email protected] ping (S, J) to an in-memory map allowing the file system to access the block in the journal, as seen in Figure 2b. Since the journal is written sequentially to a circular log, overwriting a metadata block is not possible. Therefore, ext4-lazy writes an updated block to the head of the log, updating the map and invalidating the old copy of the block. Ext4-lazy uses a large journal so that it can continue writing updated blocks while reclaiming the space from the 12 SUMMER 2017 VOL. 42, NO. 2 www.usenix.org FILE SYSTEMS AND STORAGE Evolving EXT4 for Shingled Disks Peter Desnoyers is an Associate Professor at Northeastern ST5000AS0011 30 University. He worked for ST8000AS0002 10 ST4000LM016 Apple, Motorola, and a WD40NMZW number of startups for 15 years 3 WD5000YS before getting his PhD at the University of Massachusetts, Amherst, in 2007. He received 1 BS and MS degrees from MIT. His main 0.3 focuses are storage, particularly the integration 0.1 of emerging storage technologies, and cloud Throughput (MiB/s) computing. [email protected] 0.03 0.01 0 100 200 300 400 500 Time (s) Figure 1: Throughput of CMR and SMR disks from Table 1 under 4 KiB random write traffic. The CMR disk (WD500YS) has a stable but low throughput under random writes. SMR disks, on the other hand, have a short period of high throughput followed by a continuous period of ultra-low throughput. Type Vendor Model Capacity Form Factor SMR Seagate ST8000AS0002 8 TM 3.5 inch SMR Seagate ST5000AS0011 5 TB 3.5 inch SMR Seagate ST4000LM016 4 TB 2.5 inch SMR Western Digital WD40NMZW 4 TB 2.5 inch CMR Western Digital WD5000YS 500 MB 3.5 inch Table 1: CMR and SMR disks from two vendors used for evaluation invalidated blocks. During mount, it reconstructs the in-memory map from the journal resulting in a modest increase in mount time. Results show that ext4-lazy significantly improves performance on SMR disks in general, as well as on CMR disks for metadata-heavy workloads in particular. Our main contribution to ext4 includes the design, implementation, and evaluation of ext4- lazy on SMR and CMR disks. The change we make is minimally invasive—we modify 80 lines of existing code and introduce the new functionality in additional files totaling 600 lines of C code. As we show in the evaluation section, even on a metadata-light file server benchmark where the metadata writes make up less than 1% of total writes, with stock ext4 the SMR disk appears unresponsive for almost an hour with near-zero throughput. With ext4-lazy, on the other hand, the SMR disk does not suffer such a behavior and completes 1.7–5.4x faster. For directory traversal and metadata-heavy workloads, ext4-lazy achieves 2–13x improvement on both SMR and CMR disks. Background A high-level introduction to SMR technology has been previously presented in ;login: [3]. Readers interested in nitty-gritty details of how an SMR disk works and why it suffers under random writes may refer to the detailed study [1] of one such disk. Here, we give just enough background on SMR disks and ext4 journaling to make the rest of the article understandable. www.usenix.org SUMMER 2017 VOL. 42, NO. 2 13 FILE SYSTEMS AND STORAGE Evolving EXT4 for Shingled Disks are updated using RMW during the idle times, emptying the per- 1 J sistent cache. If the burst of random writes is large enough to fill S 2 Journal the persistent cache, the throughput of the disk drops because every incoming write requires RMW of the corresponding band. Memory Disk Sequential writes, on the other hand, are detected and written (a) Journaling under ext4 directly to bands, bypassing the persistent cache. Ext4 and Journaling S J 2 1 J The ext4 file system evolved from ext2, which was influenced by Fast File System (FFS). Similar to FFS, ext2 divides the disk into Map Journal cylinder groups—or as ext2 calls them, block groups—and tries to Memory Disk put all blocks of a file in the same block group. To further increase (b) Journaling under ext4-lazy locality, the metadata blocks (inode bitmap, block bitmap, and inode table) representing the files in a block group are also placed Figure 2: (a) Ext4 writes a metadata block to disk twice. It first writes within the same block group, as Figure 3a shows. In ext2 the size the metadata block to the journal at some location J and marks it dirty of a block group was limited to 128 MiB—the maximum number in memory. Later, the writeback thread writes the same metadata block of 4 KiB data blocks that a 4 KiB block bitmap can represent. Ext4 to its static location S on disk, resulting in a random write. (b) Ext4-lazy introduced flexible block groups or flex_bgs—a set of contiguous writes the metadata block approximately once to the journal and inserts a mapping (S, J) to an in-memory map so that the file system can find the block groups whose metadata is consolidated in the first 16 MiB metadata block in the journal. of the first block group within the set, as shown in Figure 3b. SMR Ext4 ensures metadata consistency via journaling, but it does As a concrete example, one SMR disk used in our evaluation not implement journaling itself; rather, it uses a generic kernel consists of ≈ 30 MiB bands that are the smallest units that must layer called the Journaling Block Device that runs in a separate be written sequentially.
Details
-
File Typepdf
-
Upload Time-
-
Content LanguagesEnglish
-
Upload UserAnonymous/Not logged-in
-
File Pages8 Page
-
File Size-