Unionfs: User- and Community-Oriented Development of a Unification File System David Quigley, Josef Sipek, Charles P. Wright, and Erez Zadok Stony Brook University {dquigley,jsipek,cwright,ezk}@cs.sunysb.edu Abstract If a file exists in multiple branches, the user sees only the copy in the higher-priority branch. Unionfs allows some branches to be read-only, Unionfs is a stackable file system that virtually but as long as the highest-priority branch is merges a set of directories (called branches) read-write, Unionfs uses copy-on-write seman- into a single logical view. Each branch is as- tics to provide an illusion that all branches are signed a priority and may be either read-only writable. This feature allows Live-CD develop- or read-write. When the highest priority branch ers to give their users a writable system based is writable, Unionfs provides copy-on-write se- on read-only media. mantics for read-only branches. These copy- on-write semantics have lead to widespread There are many uses for namespace unifica- use of Unionfs by LiveCD projects including tion. The two most common uses are Live- Knoppix and SLAX. In this paper we describe CDs and diskless/NFS-root clients. On Live- our experiences distributing and maintaining CDs, by definition, the data is stored on a read- an out-of-kernel module since November 2004. only medium. However, it is very convenient As of March 2006 Unionfs has been down- for users to be able to modify the data. Uni- loaded by over 6,700 unique users and is used fying the read-only CD with a writable RAM by over two dozen other projects. The total disk gives the user the illusion of being able to number of Unionfs users, by extension, is in the modify the CD. Maintaining an identical sys- tens of thousands. tem configuration across multiple diskless sys- tems is another application of Unionfs. One simply needs to build a read-only system im- 1 Introduction age, and create a union for each diskless node. Unionfs is based on the FiST stackable file sys- Unionfs is a stackable file system that allows tem templates, which provide support for lay- users to specify a series of directories (also ering over a single directory [12]. As shown known as branches) which are presented to in Figure 1(a), the kernel’s VFS is responsi- users as one virtual directory even though the ble for dispatching file-system–related system branches can come from different file systems. calls to the appropriate file system. To the VFS, This is commonly referred to as namespace a stackable file system appears as if it were a unification. Unionfs uses a simple priority sys- standard file system, but instead of storing or tem which gives each branch a unique priority. retrieving data, a stackable file system passes 350 • Unionfs: User- and Community-Oriented Development of a Unification File System User Process User Process rename() User rename() User vfs_rename() vfs_rename() Virtual File System Virtual File System fist_rename() unionfs_rename() Kernel Kernel FiST Unionfs nfs_rename() RW RO tmpfs_rename() nfs_rename() NFS tmpfs ... NFS (a) Wrapfs layers over a single directory. (b) Unionfs layers over multiple directories. Figure 1: User processes issue system calls, which the kernel’s virtual file system (VFS) directs to stackable file systems. Stackable file systems in turn pass the calls down to lower-level file systems (e.g., tmpfs or NFS). calls down to lower-level file systems that are as a research project [10, 11]. We released responsible for data storage and retrieval. In Unionfs as a standalone kernel module because this scenario, NFS is used as the lower-level that was the most expedient way for users to file system, but any file system can be used begin using it and it required less initial ef- to store the data (e.g., Ext2, Ext3, Reiserfs, fort on our part. Unionfs was quickly adopted SQUASHFS, isofs, and tmpfs). To the lower- by several LiveCDs such as SLAX [7] (De- level file systems, a stackable file system ap- cember 2004) and Knoppix [5] (March 2005). pears as if it were the VFS. This makes stack- As of March 2006, Unionfs has been down- able file system development difficult, because loaded by over 6,700 users from 81 countries the file system must adhere to the conventions and is distributed as part of other projects. Our both of file systems (for processing VFS calls) mailing list currently has 336 subscribers with and of the VFS (for making VFS calls). 53 of them subscribed to our CVS update list. Unionfs is an integral part of several LiveCDs, As shown in Figure 1(b), Unionfs extends the so the actual number of Unionfs users is much FiST templates to layer over multiple direc- larger. tories, unify directory contents, and perform copy-on-write. In this example, Unionfs is Maintaining Unionfs outside of the kernel has layered over two branches: (1) a read-write both benefits and complications. By maintain- tmpfs file system and (2) a read-only NFS file ing the tree outside of the kernel, our user base system. The contents of these file systems are is expanded: users can still use their vendor’s virtually merged, and if operations on the NFS kernel, and we are able to support several ker- file system return the read-only file system er- nel versions. We were also able to release ror code (EROFS) then Unionfs transparently Unionfs more frequently than the kernel. How- copies the files to the tmpfs branch. ever, this makes our code more complex since we must deal with changing interfaces between We originally released Unionfs in November kernel versions. It also raises questions about 2004, after roughly 18 months of development the point at which support for a particular older 2006 Linux Symposium, Volume Two • 351 kernel version should be dropped. At this point, 2 Use Cases Unionfs has become established enough that we are moving towards a release that is ready for mainline kernel submission. We have identified three primary use cases for Unionfs. All of these common use cases lever- age Unionfs’s copy-on-write semantics. The first and most prevalent use of Unionfs is in Unionfs has complex allocation schemes (par- LiveCDs. The second is using Unionfs to pro- ticularly for dentry and inode objects), and vide a common base for several NFS-mounted makes more use of kmalloc than other file machines. The third is to use Unionfs for snap- systems. One hurdle we had to overcome shotting. was lack of useful memory-allocation debug- ging support. The memory-allocation debug- ging code in recent -mm kernels does not pro- LiveCDs. LiveCDs allow users to boot Linux vide sufficient debugging information. In our without modifying any data on a hard disk. approach, we log kmalloc and dentry alloca- This has several advantages: tions, and then post-process the log to locate memory leaks and other errors. • Users can try Linux without committing to it [5, 7]. In our efforts to move toward kernel inclusion • Special-purpose open-source software can we have come across many aspects that conflict be distributed to non-technical users (e.g., with maintaining an out-of-kernel module. One for music composition [4]). of the main issues is the ability to separate re- • System administrators can rescue ma- search code from practical code. Features such chines more easily [2]. as persistent inodes and atomically performing certain operations increase code complexity, • Many similar machines can be set up with- conflicting with the mantra “less code is better out installing software (e.g., in a cluster code.” We also had to change the way we sepa- environment [9], or at events that require rate file system components to provide simpler certain software). and more easily maintainable code. In addition to this, we also have to keep up with changes in kernel interfaces such as the change of locking The simplest use of Unionfs for LiveCDs uni- primitives introduced in Linux 2.6.16. fies a standard read-only ISO9660 file system with a higher-priority read-write tmpfs file system. Current versions of Knoppix [5] use such a configuration, which allows users to in- The rest of this paper is organized as follows. stall and reconfigure programs. In Section 2 we describe Unionfs use cases. In Section 3 we describe the challenges of main- Knoppix begins its boot sequence by loading taining an out-of-tree module. In Section 4 we an initial RAM disk (initrd) image of an describe some limitations of Unionfs. In Sec- Ext2 file system and then executing a shell tion 5 we present a brief performance evalua- script called /linuxrc. The linuxrc script tion of Unionfs. Finally, we conclude in Sec- first mounts the /proc and /sys file systems. tion 6. Next, Knoppix loads various device drivers 352 • Unionfs: User- and Community-Oriented Development of a Unification File System (e.g., SCSI, IDE, USB, FireWire) and mounts to the SQUASHFS images, the highest-priority a compressed ISO9660 image on /KNOPPIX. branch is a read-write tmpfs which provides After the Knoppix image is mounted, the the illusion that the CD is read-write. Unionfs module is loaded. Next a tmpfs file system is mounted on /ramdisk. Once SLAX uses the pivot_root system call so Unionfs is mounted, this RAM disk becomes that the root file system is indeed Unionfs, the destination for all of the changes to the CD- whereas Knoppix creates symbolic links to ROM. Next, the directory /UNIONFS is cre- provide the illusion of a Unionfs-rooted CD.