puffs - Pass-to-Userspace Framework File System Antti Kantee <[email protected].fi> Helsinki University of Technology ABSTRACT Fault tolerant and secure operating systems are a worthwhile goal. Aknown method for accomplishing fault tolerance and security is isolation. This means running separate operating system services in separate protection domains so that theycannot interfere with each other,and can communicate only via well-defined messaging inter- faces. Isolation and message passing brings inherent overhead when compared to ser- vices doing communication by accessing each others memory directly.Toaddress this, the ultimate goal would be to be able to run the kernel subsystems in separate domains during development and testing, but have a drop-in availability to makethem run in ker- nel mode for performance critical application scenarios. Still today,most operating sys- tems are written purely with C and some assembly using the monolithic kernel approach, where all operating system code runs within a single protection domain. Asingle error in anysubsystem can bring the whole operating system down. This work presents puffs -the Pass-to-Userspace Framework File System - shipped with the NetBSD Operating System. It is a framework for implementing file systems out- side of the kernel in a separate protection domain in a user process. The implementation is discussed in-depth for a kernel programmer audience. The benefits in implementation simplicity and increased security and fault tolerance are argued to outweigh the measured overhead when compared with a classic in-kernel file system. Aconcrete result of the work is a completely BSD-licensed sshfs implementation. Keywords:userspace file systems, robust and secure operating systems, message-passing subsystems, BSD-licensed sshfs 1. Introduction protection domain with the popular argument "Microkernels have won",isafamous being an advantage in performance. Even if we quote from the Tanenbaum - Torvalds debate from were to disregard research which states that the the early 90’s. Microkernel operating systems are performance difference is irrelevant [2], we might associated with running the operating system ser- be willing to makeatradeofffor a more robust vices, such as file systems and networking proto- system. cols, in separate domains, and component com- Aseparate argument is that we do not need munication via message passing through channels to see issues only in black-and-white. An operat- instead of direct memory references. This is ing system’score can be monolithic with the known as isolation and provides an increase in associated tradeoffs, but offer the interfaces to system security and reliability in case of a misbe- implement some services in separate domains. having component [1]; at worst the component An HTTP server or an NFS server can be imple- can corrupt only itself instead of the entire sys- mented either as part of the monolithic kernel or tem. However, most contemporary operating sys- as a separate user process, eventhough theyboth tems still run all services inside a single have their "correct" locations of implementation. There is obviously room for both a microkernel The best known userspace file system and a monolithic kernel approach within the same framework is FUSE, Filesystem in USErspace operating system. Another relevant argument is [6]. It supports already hundreds of file systems the use of inline assembly in an operating system: written against it. On a technical level, puffs is almost everyone agrees that it is wrong, yet not fairly similar to FUSE, since theyboth export using it makes the system less performant. similar virtual file system interfaces to userspace. Clearly,performance is not everything. However, the are differences already currently in, This work presents puffs,the Pass-To- for example, pathname handling and concurrency Userspace Framework File System for NetBSD. control. The differences are expected to growas puffs provides an interface similar to the kernel the puffs project reaches future goals. Even so, virtual file system interface, vfs [3], to a user providing a source compatible interface with process. puffs attaches itself to the kernel vfs FUSE is an important goal to leverage all the layer.Itpasses requests it receivesfrom the vfs existing file systems (see Chapter 5). In the sum- interface in the kernel to userspace, waits for a mer of 2005 FUSE was available only for Linux, result and provides the caller with the result. buthas since been ported to FreeBSD in the Applications and the rest of the kernel outside of Fuse4BSD [7] project. Aderivate project of the the vfs module cannot distinguish a file system FreeBSD porting effort, MacFUSE [8], recently implemented on top of puffs from a file system added support for Mac OS X. Adownside from implemented purely in the kernel. For the the BSD point-of-viewisthat userspace library userspace implementation a library,libpuffs, is for FUSE is available only under LGPL and that provided. libpuffs not only provides a program- file systems written on top of it have a tendency ming interface to implement the file system on, of being GPL-licensed. butalso includes convenience routines commonly Apart from frameworks merely exporting required for implementing file systems. the Unix-style vfs/vnode interface to userspace puffs is envisioned to be a step in moving for file system implementation, there are systems towards a more flexible NetBSD operating sys- which completely redesign the whole concept. tem. It clearly adds a microkernel touch with the Plan 9 is Bell Labs’ operating system where the associated implications for isolation and robust- adage "everything is a file" really holds: there are ness, but also provides an environment in which no special system calls for services likethere are programming a file system is much easier than on Unix-style operating systems, where, for compared to the same task done in the kernel. example, opening a network connection requires a And instead of just creating a userspace file sys- special type of system call. Plan 9 was also tem framework, the lessons learned from doing so designed to be a distributed operating system, so will be turned upside down and the whole system all the file operations are encoded in such a way will also be improvedtobetter facilitate creating that a remote machine can decode them. As a functionality such as puffs.The latter part, how- roughly equivalent counterpart to the Unix virtual ev er, isout of the scope of this paper. file system, Plan 9 provides the 9P [9] transport protocol, which is used by clients to communicate Related Work with file servers. 9P has been adapted to for example Linux [10], but the greater problem with There are several other packages available 9P is that it is relatively different from the for building file systems in userspace. When this (Net)BSD vfs interface and it makes some project was begun in the summer of 2005, the assumptions about file systems in general not only option available for BSD was nnpfs, which is valid on Unix [10]. Therefore, it was not directly supplied as part of the Arla [4] AFS implementa- considered for the userspace library interface. tion. Arla is a portable implementation of AFS. It relies on a small kernel module, nnpfs, which DragonFly BSD has started putting forth attaches to the host operating system’skernel and effort in creating a VFS transport protocol, which, provides an interface for the actual userspace AFS like9P, would be suitable for distributed environ- implementation to talk to. Ahuge drawback was ments in which the server can exist on a different that at the time it only supported caching on a file network node than the client [11]. It is also level. Since, it has developed block levelcaching usable for implementing a file system in and some documentation on howtowrite file sys- userspace, but is a huge undertaking and restruc- tems on top of it [5]. tures much of the kernel file system code. The main reason for writing a framework from scratch is that the ultimate goal of the work puffs architecture is not to develop a userspace file system frame- work, but rather to improve the flexibility and robustness of the operating system itself. While taking a more flexible route such as that of 9P file server (4) may eventually prove tobethe right thing to do, it libpuffs (3) is easier to takensmall steps in reaching a goal and keep the system functional all the time. Cur- user rently,especially the kernel side of puffs is very /dev/puffs (2) lightweight and tries to be a good kernel citizen in not modifying the rest of the kernel. The ultimate puffs vfs module (1) goal is to gradually change this in creating a more kernel secure and reliable operating system. kernel Paper Contents user syscall Chapter 2 discusses the architecture and implementation of puffs on an in-depth technical application level. Chapter 3presents a fewfile systems built on top of puffs.Itdiscusses experiences in devel- oping them. Chapter 4 presents performance measurements and analyses the measured results. The vfs layer is made up of twoseparate Chapter 5 contains work being done currently and interfaces: the actual virtual file system interface outlines some future visions for development. and the vnode interface. The former deals with Finally,Chapter 6 provides conclusions. calls involving file system leveloperations, such as mount and unmount, while the latter always involves an operation on a file; the vnode or vir- 2. puffs Architecture tual node is an abstract, i.e. virtual, representation puffs is made up of four separate compo- of a file. nents (see figure): Vnodes are treated as reference counted 1. VFS attachment, including virtual mem- objects by the kernel. Once the reference count ory subsystem and page cache integra- for a vnode drops to zero, it is movedtothe freel- tion.