The NILFS2 Filesystem: Review and Challenges

Home , Data loss, NILFS, Tux3

Ryusuke KONISHI NTT Cyberspace Laboratories NTT Corporation — NILFS Introduction — FileSystem Design — Development Status — Wished features & Challenges

Copyright (C) 2009 NTT Corporation 2 — NILFS is the Linux file system supporting “continuous snapshotting” üProvides versioning capability of entire file system üCan retrieve previous states before operation mistake even restores files mistakenly overwritten or destroyed just a few seconds ago. üMerged into the mainline kernel 2.6.30

Human error Hardware 26% failure 59%

Unprotected by redundant drives. Mostly preventable with basic high-integrity system Source: Ontrack Data Recovery, Inc. (Redundant configuration) Including office PC.The data is based on actual data recoveries performed by Ontrack.

Copyright (C) 2009 NTT Corporation 4 ISSUES IN BACKUP — Changes after the last backup are not safe — But, frequent backups place burden on the system as well as the interval is limited by the backup time.

Incrementally done

Hard drive Backup Discretely scheduled e.g. per 8 hours, per day Backup

Backup

Some applications (esp. for system software) Restoration is cumbersome are sensitive to coherency among files (and often fails or unhelpful)

Copyright (C) 2009 NTT Corporation 5 — Adopt Log-structured File System approach to continually save data on disk. — Checkpoints are created every time user makes a change, and each checkpoint is mutable to snapshots later on. Instantly Ordinary file systems NILFS taken Backup CP CP A CP B B → → → CP C A B C CP D → → CP E E D E F F Accessible ex. any time Previous data is Previous data is per 8 hours overridden preserved on disk

Copyright (C) 2009 NTT Corporation 6 Maximum File System Instant Writable Retroactive Incremental Number of Snapshotting Snapshots Backup (solution) Snapshots Snapshots NTFS Optional Volume 64 Third party Shadow Copy product

ZFS Unlimited*1 √ √ √

Btrfs Unlimited*1 √ √ Planned

NILFS2 Unlimited*1 √ √ Requested

Apple Time Thinned out ― Backup Machine automatically √ Solutions CDP Unlimited*1 ― ― ― √

*1:No practical limits (bounded by disk capacity)

Application view: File A (modified) File B (appended) A A’ A B B’

On disk images: B-Tree intermediate blocks Metadata blocks (inodes, …) A B modified blocks

A B A’ B’

x255 CPFILE SUFILE DAT Data blocks checkpoint Segment Disk block information usage address translation *1 B-tree depth varies with the number of data blocks

Files, Directories, Symbolic links cno=100 IFILE cno=101 cno=102 contains cno=103 ino=123 ino=124 ino=125 ino=126 Inodes cno=104 cno=105

Copyright (C) 2009 NTT Corporation 9 — Segments ü Disk space is allocated or freed per segment ü Each segment is filled with logs — Logs ü Organize delta of data and metadata per file ü Composea new version of metadata hierarchy every checkpoint

Super Segments Logs Super block block 0 1 2 3 N-1 N

Metadata files file-A file-B file-C file-D IFILE CPFILE SUFILE DAT

Data blocks B-tree nodes Summary blocks (per log) Super Root ( Changed blocks only ) Block

The super block points to the most recent log. *1 The pointer is periodically updated. incomplete series of logs is ignored Super root block Super block Summary block to be chosen next segment next segment next segment

log log log log log

Scan order of logs

*1Series of logs may not have the super root block. This type of variant is allowed for optimizations to make synchronous write operation faster.

Copyright (C) 2009 NTT Corporation 11 — Creates new disk space to continue writing logs ü Essential function of Log structured File Systems — A disk block is in-use if it belongs to a snapshot or recent checkpoints; unused blocks are freed with their checkpoints

Preserving checkpoints as snapshots

Protection CP SS SS period CP CP CP CP CP CP CP CP CP CP CP A checkpoint which user marked as SNAPSHOT Not all unprotected checkpoints will be Recent checkpoints deleted in a shot of GC

segment usage information checkpoint information Cleanerd execute GC Virtual block address (Standalone daemon) information ioctl() Userland

Log summary SUFILE CPFILE DAT GC cache Log writer Kernel L L L L

Segment 101 Segment 102 Segment 103 Segment 1000 Segment 1001 On disk S L L D L S D L D D S L L

Live blocks Dead blocks

S L L S L L L L

Copyright (C) 2009 NTT Corporation 13 block addressing — Issue for moving disk blocks ü Must rewrite b-tree node blocks and inodeshaving a pointer to moved blocks ü Disk blocks are pointed from many parent blocks because NILFS makes numerous versions — Solution ü Use virtual (i.e. indirect) block numbers instead of real disk block numbers

Block Disk Address mapping Translation File offset Virtual Block Disk Block (per block) Number Number

offset=256 vblocknr=1235 blocknr= 10002 Example. DAT File #1234 4000

#1235 10002 Extra lookup is needed but DAT is cached as a file B-tree lookup Table reference

— Cleanerddetermines if each disk block is LIVE or DEAD from DAT

Snapshots Recent checkpoints Virtual Block Numbers DAT cno cno cno le64 start cno>= 800 vblocknr=1234 200 2 120 280 le64 end LIVE! 200 < 280 < 300 300 le64 start vblocknr=1235 300 le64 end LIVE! 300 < 800 < 1000 1000

le64 start vblocknr=1440 300 DEAD le64 end Log 500 Summary vblocknr vblocknr vblocknr Block(s) 1234 1235 1440

Virtual block numbers of the payload blocks are written in the summary

Copyright (C) 2009 NTT Corporation 15 — Achievements üSnapshots Automatically and continuously taken Mountable as read-only file systems Mountable concurrently with the writable mount (convenient for online backup) Quick listing Easy administration

üOnline disk space reclamation Can maintain multiple snapshots

Copyright (C) 2009 NTT Corporation 16 — Achievements üOther Features Quick recovery on-mount after system crash B-tree based file and meta data management 64-bit data structures; support many files, large files and disks Block sizes smaller than page size (e.g. 1KB or 2KB) Redundant super blocks (automatic switch) 64-bit on-disk timestamps which are free of the year 2038 problem Nano second timestamps

— Performance issues üDirectory operations üWrite performance üOptimization for silicon disks (esp. for SSD)

Copyright (C) 2009 NTT Corporation 18 Compilebench (kernel 2.6.31-rc8) 180 Ext3 F a st 160 Btrfs 140 Under investigation. Nilfs2 120 Applying ext3 hash tree or b-tree based directory management is discussed 100 in the NILFS community. 80 60 A v g . M B /s ec 40 20 0 initial create patch compile clean read delete stat S low create tree tree tree

Hardware specs: Processor: Pentium Dual-Core CPU E5200 @ 2.49GHz, Chipset: Intel 4 Series Chipset + ICH10R, Memory: 2989MB, Disk: ST3500620AS

T h ro ug p ut 40

0 S low seq write seq rewrite seq read

Hardware specs: Processor: Pentium Dual-Core CPU E5200 @ 2.49GHz, Chipset: Intel 4 Series Chipset + ICH10R, Memory: 2989MB, Disk: ST3500620AS

Copyright (C) 2009 NTT Corporation 20 — Faster and robust online backup like ZFS ü Back up checkpoints instead of usual files ü Similar features are planned for btrfs, TUX3, and the Device Mapper (dm replication)

The reader extracts delta between The writer appends the delta two checkpoints and streams it to a passive NILFS partition over the network Replicator Replicator sshconnection (writer) (reader) CP CP CP CP CP CP CP CP CP CP CP CP

— How to extract delta between two checkpoints ? üTwo approaches Scan logs in creation order (just gets delta from logs) Scan DAT to gather blocks changed during given period üHave pros and cons The former seems to be efficient, but has a limit due to GC. Replicator may use either or both of these methods — Rollback on the destination file system üNeeded before starting replication especially to thin out the backups with GC

Copyright (C) 2009 NTT Corporation 22 — Better garbage collector is much needed üBetter data retention policy to prevent disk full üSelf-regulating speed üSmarter selection algorithm of target segments to reduce I/O — Further chance of optimization and enhancement üBackground data verification üDefragmentation üDe-duplication üBackground disk format upgrade

Copyright (C) 2009 NTT Corporation 23 — NILFS is in the mainline kernel üYou can go back in time just before you scream “ ” üInstant failure recovery. Simple administration. üPotential for innovative application ü… and most importantly, WORKING STABLY :) — Contribution is welcome üVarious topics in GC, snapshot tools, and time-oriented tools. üLet’s drop the (EXPERIMENTAL) flag!

— Mailing-list üusers (at) nilfs.org üusers-ja(at) nilfs.org

— Contact Information üRyusuke KONISHI