ExtFUSE Extension framework for File systems in User space Ashish Bijlani, Umakishore Ramachandran Georgia Institute of Technology 1 Kernel vs User File Systems • Examples • Examples – Ext4, OverlayFS, etc. – EncFS, Gluster, etc. • Pros • Pros – Native performance – Improved security/reliability – Easy to develop/debug/ • Cons maintain – Poor security/reliability – Not easy to develop/ • Cons debug/maintain ––PoorPoor performance! performance! 2 File Systems in User Space (FUSE) struct fuse_lowlevel_ops ops { • State-of-the-art framework .lookup = handle_lookup, .access = NULL, – All file system handlers .getattr = handle_getattr, implemented in user space .setattr = handle_setattr, .open = handle_open, .read = handle_read, • Over 100+ FUSE file systems .readdir = handle_readdir, – Stackable: Android .write = handle_write, SDCardFS, EncFS, etc. // more handlers … .getxattr = handle_getxattr, – Network: GlusterFS, Ceph, .rename = handle_rename, Amazon S3FS, etc. .symlink = handle_symlink, .flush = NULL, } 3 FUSE Architecture 4’ FUSE Daemon Application Over the L I B F U S E network User 4 Stackable 1 Kernel VFS 2 3 5 FUSE Driver QUEUE 6 Lower FS (e.g., EXT4) 4 FUSE Architecture lookup() FUSE Daemon open(“/mnt/foo/bar”) getattr() setattr() L I B F U S E Application open() User read() 4 1 Kernel readdir() write() VFS … rename() 2 lookup(“foo”) symlink() close() 3 5 FUSE Driver getxattr() QUEUE C O N T E X T setxattr() S W I T C H 6 Lower FS (e.g., EXT4) 5 FUSE Performance “cd linux-4.18; make tinyconfig; make -j4” • Intel i5-3350 quad core, Ubuntu 16.04.4 LTS • Linux 4.11.0, LibFUSE commit # 386b1b, StackFS (w/ EXT4) 50 39.74 Opts Enabled 38 30.91 -o max_write=128K 25 -o splice_read -o splice_write 13 28.57% overhead! Time (sec) -o splice_move 0 entry_timeout > 0 Native (EXT4) FUSE attr_timeout > 0 6 6 # Req received by FUSE • “cd linux-4.18; make tinyconfig; make -j4” 400K Regular Optimized 350K 1 2 300K 3 250K 4 200K atimeVFS changes issues duringgetxattr() 150K 4x fewer read() invalidate # Requests for each write() for 100K lookup()s cachedreading attributes security labels 50K 2’ 1’ 0K Lookup Getattr Rename Setattr Create Open Release Getxattr Mkdir Unlink Opendir Readdir Releasedir Read Write 7 eBPF Overview • Pseudo machine architecture Clang/LLVM bytecode BPF C program • C code compiled into BPF code syscall() • Verified and loaded into kernel user • Executed under VM runtime kernel Verifier • Evolved as a generic kernel extension framework bpf virtual machine • Used by trace, perf, net subsystems BPF Map sandbox key-value data struct • Shared BPF maps with user space Kernel functions 8 ExtFUSE • Extension framework for File systems in User space –Register “thin” extensions - handle requests in kernel • Avoid user space context switch! –Share data between FUSE daemon and extensions using BPF maps • Cache metadata in the kernel 9 ExtFUSE Architecture BPF Handlers FUSE Daemon Application L I B E x t F U S E L I B F U S E User 4 1 Kernel 7 0’ Cache Load VFS Meta- BPF data Code Deliver req to 2 extension 3’ 3 BPF VM FUSE Driver QUEUE 4’ 5 BPF Map Serve from cache 6 Lower FS (e.g., EXT4) 10 ExtFUSE Applications • BPF code to proactively cache/invalidate meta-data in kernel – Applies potentially to all FUSE file systems – e.g., Gluster readdir ahead results could be cached • BPF code to perform custom filtering or perm checks – e.g., Android SDCardFS uid checks in lookup(), open() • BPF code to directly forward I/O requests to lower FS in kernel – e.g., install/remove target file descriptor in BPF map 11 ExtFUSE Example struct bpf_map_def map = { .type = BPF_MAP_TYPE_HASH, .key_size = sizeof(u64), // ino (param 0) .value_size = sizeof(struct fuse_attr_out), .max_entries = MAX_NUM_ATTRS, // 2 << 16 }; // getattr() kernel extension - cache attrs int getattr(struct extfuse_args *args) { u32 key = bpf_extfuse_read(args, PARAM0); u64 *val = bpf_map_lookup_elem(map, &key); if (val) bpf_extfuse_write(args, PARAM0, val); } 12 ExtFUSE Example • Invalidate cached attrs from kernel extensions. E.g., // setattr() kernel extension - invalidate attrs int setattr(struct extfuse_args *args) { u32 key = bpf_extfuse_read(args, PARAM0); if (val) bpf_map_delete_elem(map, &key); } • Cache attrs from FUSE daemon – Insert into map on atime change • Similarly, cache lookup()s and xattr()s, symlink()s 13 ExtFUSE Performance • “cd linux-4.18; make tinyconfig; make -j4” • Intel i5-3350 quad core, SSD, Ubuntu 16.04.4 LTS • Linux 4.11.0, LibFUSE commit # 386b1b, StackFS (w/ EXT4) 40 Overhead 30 Optimized Latency: 28.57% ExtFUSE Latency: 2.16% 20 Only 2.16% overhead! ExtFUSE Memory: 50MB Time (sec) 10 (worst case) 0 Cached: lookup, attr, xattr Native Optimized ExtFUSE Passthrough: read, write 14 # Req received by FUSE • “cd linux-4.18; make tinyconfig; make -j4” 400K Optimized ExtFUSE 350K 300K 250K 200K No read/ 150K Very few Very few write() reqs # Requests 100K getattr()s getxattr()s 50K 0K Lookup Getattr Rename Setattr Create Open Release Getxattr Mkdir Unlink Opendir Readdir Releasedir Read Write 15 Conclusion • ExtFUSE framework safely executes “thin” file system handlers in the kernel. • Developers can use ExtFUSE to • Cache metadata requests • Directly pass I/O requests to lower FS. • Insert custom security checks in the kernel. • We ported four FUSE file systems to ExtFUSE, including Android SDcardFS and show significant performance improvements. 16 Thank You! • Open Source Summit ’18 • Presentation slides • Linux Plumbers Conference ’18 talk • Presentation video • Project page –https://extfuse.github.io 17.