Dmdedup: Device Mapper Target for Data Deduplication Appears in the Proceedings of the 2014 Ottawa Linux Symposium (OLS’14) Vasily Tarasov Deepak Jain Geoff Kuenning Stony Brook University & IBM Research Stony Brook University Harvey Mudd College [email protected] [email protected] [email protected] Sonam Mandal Karthikeyani Palanisami Stony Brook University Stony Brook University [email protected] [email protected] Philip Shilane Sagar Trehan Erez Zadok EMC Stony Brook University Stony Brook University [email protected] [email protected] [email protected] Abstract Primary-storage deduplication poses several challenges compared to backup datasets: access locality is less We present Dmdedup, a versatile and practical primary- pronounced; latency constraints are stricter; fewer du- storage deduplication platform suitable for both regular plicates are available (about 2× vs. 10× in backups); users and researchers. Dmdedup operates at the block and the deduplication engine must compete with other layer, so it is usable with existing file systems and appli- processes for CPU and RAM. To facilitate research in cations. Since most deduplication research focuses on primary-storage deduplication, we developed and here metadata management, we designed and implemented present a flexible and fully operational primary-storage a flexible backend API that lets developers easily build deduplication system, Dmdedup, implemented in the and evaluate various metadata management policies. We Linux kernel. In addition to its appealing properties implemented and evaluated three backends: an in-RAM for regular users, it can serve as a basic platform both table, an on-disk table, and an on-disk COW B-tree. We for experimenting with deduplication algorithms and have evaluated Dmdedup under a variety of workloads for studying primary-storage datasets and workloads. and report the evaluation results here. Although it was In earlier studies, investigators had to implement their initially designed for research flexibility, Dmdedup is own deduplication engines from scratch or use a closed- fully functional and can be used in production. Under source enterprise implementation [3, 21, 23]. Dmdedup many real-world workloads, Dmdedup’s throughput ex- is publicly available under the GPL and we submitted ceeds that of a raw block device by 1.5–6×. the code to the Linux community for initial review. Our final goal is the inclusion in the mainline distribution. 1 Introduction Deduplication can be implemented at the file system or block level, among others. Most previous primary- As storage demands continue to grow [2], even continu- storage deduplication systems were implemented in the ing price drops have not reduced total storage costs. Re- file system because file-system-specific knowledge was moving duplicate data (deduplication) helps this prob- available. However, block-level deduplication does not lem by decreasing the amount of physically stored infor- require an elaborate file system overhaul, and allows any mation. Deduplication has often been applied to backup legacy file system (or database) to benefit from dedupli- datasets because they contain many duplicates and rep- cation. Dmdedup is designed as a stackable Linux ker- resent the majority of enterprise data [18, 27]. In recent nel block device that operates at the same layer as soft- years, however, primary datasets have also expanded ware RAID and the Logical Volume Manager (LVM). In substantially [14], and researchers have begun to ex- this paper, we present Dmdedup’s design, demonstrate plore primary storage deduplication [13, 21]. its flexibility, and evaluate its performance, memory us- 1 age, and space savings under various workloads. can be used beneath any block-based file system such as Ext4, GPFS, BTRFS, GlusterFS, etc., allowing re- Most deduplication research focuses on metadata man- searchers to bypass a file system’s limitations and de- agement. Dmdedup has a modular design that allows sign their own block-allocation policies. For that rea- it to use different metadata backends—data structures son, we chose to implement Dmdedup at the block level. for maintaining hash indexes, mappings, and reference Our design means that Dmdedup can also be used with counters. We designed a simple-to-use yet expressive databases that require direct block-device access. API for Dmdedup to interact with the backends. We implemented three backends with different underlying The drawbacks of block-level deduplication are three- data structures: an in-RAM hash table, an on-disk hash fold: (1) it must maintain an extra mapping (beyond the table, and a persistent Copy-on-Write B-tree. In this file system’s map) between logical and physical blocks; paper, we present our experiences and lessons learned (2) useful file-system and application context is lost; while designing a variety of metadata backends, and in- and (3) variable-length chunking is more difficult at clude detailed experimental results. We believe that our the block layer. Dmdedup provides several options for results and open-source deduplication platform can sig- maintaining logical-to-physical mappings. In the future, nificantly advance primary deduplication solutions. we plan to recover some of the lost context using file system and application hints. 2 Design In this section, we classify Dmdedup’s design, discuss Timeliness. Deduplication can be performed in-line the device-mapper framework, and finally present Dmd- with incoming requests or off-line via background edup’s architecture and its metadata backends. scans. In-line deduplication saves bandwidth by avoid- ing repetitive reads and writes on the storage device and 2.1 Classification permits deduplication on a busy system that lacks idle periods. But it risks negatively impacting the perfor- mance of primary workloads. Only a few studies have Levels. Deduplication can be implemented at the ap- addressed this issue [21, 24]. Dmdedup performs inline plication, file system, or block level. Applications can deduplication; we discuss its performance in Section4. use specialized knowledge to optimize deduplication, but modifying every application is impractical. Deduplication in the file system benefits many applica- 2.2 Device Mapper tions. There are three approaches: (1) modifying an ex- isting file system such as Ext3 [15] or WAFL [21]; (2) creating a stackable deduplicating file system either in- The Linux Device Mapper (DM) framework, which has kernel [26] or using FUSE [11,20]; or (3) implementing been part of mainline Linux since 2005, supports stack- a new deduplicating file system from scratch, such as able block devices. To create a new device type, one EMC Data Domain’s file system [27]. Each approach builds a DM target and registers it with the OS. An has drawbacks. The necessary modifications to an ex- administrator can then create corresponding target in- isting file system are substantial and may harm stabil- stances, which appear as regular block devices to the ity and reliability. Developing in-kernel stackable file upper layers (file systems and applications). Targets rest systems is difficult, and FUSE-based systems perform above one or more physical devices (including RAM) poorly [19]. A brand-new file system is attractive but or lower targets. Typical examples include software typically requires massive effort and takes time to reach RAID, the Logical Volume Manager (LVM), and en- the stability that most applications need. Currently, this crypting disks. We chose the DM framework for its per- niche is primarily filled by proprietary products. formance and versatility: standard, familiar tools man- age DM targets. Unlike user-space deduplication solu- Implementing deduplication at the block level is eas- tions [11, 13, 20] DM operates in the kernel, which im- ier because the block interface is simple. Unlike many proves performance yet does not prohibit communica- file-system-specific solutions, block-level deduplication tion with user-space daemons when appropriate [19]. 2 Application tracks the hashes and locations of the chunks; (3) a map- ping between Logical Block Numbers (LBNs) visible to File System upper layers and the Physical Block Numbers (PBNs) Dmdedup Block Device where the data is stored; (4) a space manager that tracks space on the data device, maintains reference counts, al- Deduplication logic locates new blocks, and reclaims unreferenced data; and (5) a chunk store that saves user data to the data device. Hash LBN Space Manager Index Mapping Chunk Store 2.4 Write Request Handling Metadata Device Data Device Figure2 shows how Dmdedup processes write requests. Figure 1: Dmdedup high-level design. 2.3 Dmdedup Components Chunking. The deduplication logic first splits all in- coming requests into aligned, subrequests or chunks with a configurable power-of-two size. Smaller chunks Figure1 depicts Dmdedup’s main components and a allow Dmdedup to detect more duplicates but increase typical setup. Dmdedup is a stackable block device the amount of metadata [14], which can harm perfor- that rests on top of physical devices (e.g., disk drives, mance because of the higher metadata cache-miss rate. RAIDs, SSDs), or stackable ones (e.g., encryption DM However, larger chunks can also hurt performance be- target). This approach provides high configurability, cause they can require read-modify-write operations. To which is useful in both research and production settings. achieve optimal performance, we recommend that Dmd- edup’s chunk size should match the block