Physical Disentanglement in a Container-Based File System Lanyue Lu, Yupu Zhang, Thanh Do, Samer AI-Kiswany Andrea C. Arpaci-Dusseau, Remzi H. Arpaci-Dusseau University of Wisconsin Madison Motivation Isolation in various subsystems ➡ resources: virtual machines, Linux containers ➡ security: BSD jail, sandbox ➡ reliability: address spaces File systems lack isolation ➡ physical entanglement in modern file systems Three problems due to entanglement ➡ global failures, slow recovery, bundled performance Global Failures Definition ➡ a failure which impacts all users of the file system or even the operating system Read-Only ➡ mark the file system as read-only ➡ e.g., metadata corruption, I/O failure Crash ➡ crash the file system or the operating system ➡ e.g., unexpected states, pointer faults Ext3 200 Read-Only Crash 150 100 50 Number of Failure Instances 0 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 Linux 3.X Versions Ext4 Read-Only Crash 400 350 300 250 200 150 100 50 Number of Failure Instances 0 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 Linux 3.X Versions Slow Recovery File system checker ➡ repair a corrupted file system ➡ usually offline Current checkers are not scalable ➡ increasing disk capacities and file system sizes ➡ scan the whole file system ➡ checking time is unacceptably long Scalability of fsck on Ext3 Ext3 1007 1000 800 723 600 476 400 Fsck Time (s) 231 200 0 200GB 400GB 600GB 800GB File-system Capacity Bundled Performance Shared transaction ➡ all updates share a single transaction ➡ unrelated workloads affect each other Consistency guarantee ➡ the same journal mode for all files ➡ limited flexibility for different tradeoffs Bundle Performance on Ext3 180 SQLite Varmail 146.7 150 120 90 76.1 60 Throughput (MB/s) 30 20 1.9 0 Alone Together Server Virtualization VM1 VM2 VM3 VM4 VM5 VD1 VD2 VD3 Shared file system VD4 VD5 All VMs will crash VM1 VM2 VM3 VM4 VM5 metadata VD1 VD2 VD3 Read-Only corruption or Crash VD4 VD5 Virtual Machines VM1 VM2 VM3 100 80 60 fsck: 496s + bootup: 68s 40 20 Throughput (IOPS) 0 0 50 100 150 200 250 300 350 400 450 500 550 600 650 700 Time (Second) Our Solution A new abstraction for disentanglement A disentangled file system: IceFS Three major benefits of IceFS ➡ failure isolation ➡ localized recovery ➡ specialized journaling Motivation IceFS Evaluation Conclusion A New Abstraction: Cube What is a cube ➡ an independent container for a group of files ➡ physically isolated Interface ➡ create a cube: mkdir(cube_flag) ➡ delete a cube: rmdir() ➡ add files to a cube ➡ remove files to a cube ➡ set cube attributes Cube Example Cube 2 Cube 1 / a c2 c1 b d A Disentangled File System No shared physical resources ➡ no shared metadata: e.g., block groups ➡ no shared disk blocks or buffers No access dependency ➡ partition linked lists or trees ➡ avoid directory hierarchy dependency No bundled transactions ➡ use separate transactions ➡ enable customized journaling modes IceFS A prototype based on Ext3 in Linux 3.5 Disentanglement ➡ directory indirection ➡ transaction splitting Directory Indirection Cube 2 Cube 1 / a c2 c1 b d Pathname matching for cubes’ top directory paths Cubes’ dentries are pinned in memory Transaction Splitting App 1 App 2 App 3 App 1 App 2 App 3 on-disk on-disk journal journal in-memory tx preallocated tx committed tx (a) Ext3/Ext4 (b) IceFS Separate transactions from different cubes commit to the disk journal in parallel Benefits of Disentanglement Localized reactions to failures ➡ per-cube read-only and crash Localized recovery ➡ only check faulty cubes ➡ offline and online Specialized journaling ➡ parallel journaling ➡ diverse journal modes ➡ no journal, no fsync, writeback, ordered, data Motivation IceFS Evaluation Conclusion Fast Recovery Ext3 IceFS 1007 1000 800 723 600 476 400 Fsck Time (s) 231 200 91 122 35 64 0 200GB 400GB 600GB 800GB File-system Capacity Specialized Journaling OR: ordered SQLite Varmail 250 220.3 NJ: no journal 200 150 120.6 125.4 103.4 100 76.1 Throughput (MB/s) 50 1.9 9.8 5.6 0 OR OR OR OR NJ OR OR NJ Ext3 IceFS Server Virtualization VM1 VM2 VM3 VM4 VM5 metadata VD1 VD2 VD3 corruption Shared file system VD4 VD5 Server Virtualization VM1 VM2 VM3 100 IceFS-Offline fsck: 35s 80 + bootup: 67s 60 40 20 0 0 50 100 150 200 250 300 350 400 450 500 550 600 650 700 100 IceFS-Online fsck: 74s 80 + bootup: 39s Throughput (IOPS) 60 40 20 0 0 50 100 150 200 250 300 350 400 450 500 550 600 650 700 Time (Second) Conclusion File systems lack physical isolation Our contribution ➡ a new abstraction: cube ➡ a disentangled file system: IceFS ➡ demonstrate its benefits: ➡ isolated failures ➡ localized recovery ➡ specialized journaling ➡ improving both reliability and performance Questions ? IceFS Disk Layout cube inode number, cube pathname orphan inode list, cube attributes SB S0 S1 S2 Sn bg bg bg bg bg cube 0 cube 1 cube 2 Each cube has one sub-super block (Si) and its own block groups (bg) .