TFS: A Transparent File System for Contributory Storage James Cipar Mark D. Corner Emery D. Berger Department of Computer Science University of Massachusetts Amherst Amherst, MA 01003 {jcipar, mcorner, emery}@cs.umass.edu Abstract cations in wide use include computing efforts like Fold- ing@home [17] and anonymous publishing and content Contributory applications allow users to donate unused distribution such as Freenet [8]. The research commu- resources on their personal computers to a shared pool. nity has also developed a number of contributory appli- Applications such as SETI@home, Folding@home, and cations, including distributed backup and archival stor- Freenet are now in wide use and provide a variety of ser- age [30], server-less network file systems [1], and dis- vices, including data processing and content distribution. tributed web caching [11]. However, the adoption of However, while several research projects have proposed storage-based contributory applications has been limited contributory applications that support peer-to-peer stor- compared to those that are CPU-based. age systems, their adoption has been comparatively lim- Two major barriers impede broader participation in ited. We believe that a key barrier to the adoption of contributory storage systems. First, existing contribu- contributory storage systems is that contributing a large tory storage systems degrade normal application perfor- quantity of local storage interferes with the principal user mance. While transparency—the effect that system per- of the machine. formance is as if no contributory application is running— To overcome this barrier, we introduce the Transparent has been the goal of other OS mechanisms for network File System (TFS). TFS provides background tasks with bandwidth [34], main memory [7], and disk schedul- large amounts of unreliable storage—all of the currently ing [19], previous work on contributory storage systems available space—without impacting the performance of has ignored its local performance impact. In particu- ordinary file access operations. We show that TFS al- lar, as more storage is allocated, the performance of the lows a peer-to-peer contributory storage system to pro- user’s file system operations quickly degrades [20]. vide 40% more storage at twice the performance when Second, despite the fact that end-user hard drives are compared to a user-space storage mechanism. We an- often half empty [10, 16], users are generally reluctant alyze the impact of TFS on replication in peer-to-peer to relinquish their free space. Though disk capacity has storage systems and show that TFS does not appreciably been steadily increasing for many years, users view stor- increase the resources needed for file replication. age space as a limited resource. For example, three of the Freenet FAQs express the implicit desire to donate less 1 Introduction disk space [12]. Even when users are given the choice to limit the amount of storage contribution, this option Contributory applications allow users to donate unused requires the user to decide a priori what is a reasonable resources from their personal computers to a shared pool. contribution. Users may also try to donate as little as pos- These applications harvest idle resources such as CPU sible while still taking advantage of the services provided cycles, memory, network bandwidth, and local storage to by the contributory application, thus limiting its overall serve a common distributed system. These applications effectiveness. are distinct from other peer-to-peer systems because the Contributions: This paper presents the Transparent resources being contributed are not directly consumed File System (TFS), a file system that can contribute by the contributor. For instance, in Freenet [8], all 100% of the idle space on a disk while imposing a neg- users contribute storage, and any user may make use ligible performance penalty on the local user. TFS oper- of the storage, but there is no relationship between ates by storing files in the free space of the file system user data and contributed storage. Contributory appli- so that they are invisible to ordinary files. In essence, normal file allocation proceeds as if the system were not Instead of using static limits, one could use a dynamic contributing any space at all. We show in Section 5 that system that monitors the amount of storage used by lo- TFS imposes nearly no overhead on the local user. TFS cal applications. The contributory storage system could achieves this both by minimizing interference with the then use a significantly greater portion of the disk, while file system’s block allocation policy and by sacrificing yielding space to the local user as needed. Possible persistence for contributed space: normal files may over- approaches include the watermarking schemes found in write contributed space at any time. TFS takes several Elastic Quotas [18] and FS2 [16]. A contributory storage steps that limit this unreliability, but because contribu- system could use these approaches as follows: whenever tory applications are already designed to work with un- the current allocation exceeds the maximum watermark reliable machines, they behave appropriately in the face set by the dynamic contribution system, it could delete of unreliable files. Furthermore, we show that TFS does contributory files until the contribution level falls below not appreciably impact the bandwidth needed for repli- a lower watermark. cation. Users typically create little data in the course of However, if the watermarks are set to comprise all a day [4], thus the erasure of contributed storage is neg- free space on the disk, the file system is forced to delete ligible when compared to the rate of machine failures. files synchronously from contributed storage when writ- TFS is especially useful for replicated storage systems ing new files to disk. In this case, the performance of the executing across relatively stable machines with plentiful disk would be severely degraded, similar to the synchro- bandwidth, as in a university or corporate network. This nous cleaning problem in LFS [31]. For this reason, Elas- environment is the same one targeted by distributed stor- tic Quotas and FS2 use more conservative watermarks age systems such as FARSITE [1]. As others have shown (e.g., at most 85%), allowing the system to delete files previously, for high-failure modes, such as wide-area lazily as needed. Internet-based systems, the key limitation is the band- Choosing a proper watermark leaves the system de- width between nodes, not the total storage. The band- signer with a trade-off between the amount of storage width needed to replicate data after failures essentially contributed and local performance. At one end of the limits the amount of storage the network can use [3]. spectrum, the system can contribute little space, limiting In a stable network, TFS offers substantially more stor- its usefulness. At the other end of the spectrum, local age than dynamic, user-space techniques for contributing performance suffers. storage. To see why local performance suffers, consider the fol- Organization: In Section 2, we first provide a de- lowing: as a disk fills, the file system’s block allocation tailed explanation of the interference caused by contrib- algorithm becomes unable to make ideal allocation de- utory applications, and discuss current alternatives for cisions, causing fragmentation of the free space and al- contributing storage. Second, we present the design of located files. This fragmentation increases the seek time TFS in Section 3, focusing on providing transparency to when reading and writing files, and has a noticeable ef- normal file access. We describe a fully operating imple- fect on the performance of disk-bound processes. In an mentation of TFS. We then explain in Section 4 the in- FFS file system, throughput can drop by as much as 77% teraction between machine reliability, contributed space, in a file system that is only 75% full versus an empty and the amount of storage used by a contributory stor- file system [32]—the more storage one contributes, the age system. Finally, we demonstrate in Section 5 that the worse the problem becomes. The only way to avoid this performance of our TFS prototype is on par with the file is to maintain enough free space on the disk to allow the system it was derived from, and up to twice as fast as allocation algorithm to work properly, but this limits con- user-space techniques for contributing storage. tribution to only a small portion of the disk. Though some file systems provide utilities to defrag- 2 Interference from Contributing Storage ment their disk layout, these utilities are ineffective when there is insufficient free space on the file system. For in- All contributory applications we are aware of are config- stance, the defragmentation utility provided with older ured to contribute a small, fixed amount of storage—the versions of Microsoft Windows will not even attempt to contribution is small so as not to interfere with normal defragment a disk if more than 85% is in use. On mod- machine use. This low level of contribution has little im- ern Windows systems, the defragmentation utility will pact on file system performance and files will generally run when the disk is more than 85% full, but will give only be deleted by the contributory system, not because a warning that there is not enough free space to defrag- the user needs storage space. However, such small, fixed- ment properly [22]. When one wants to contribute all of size contributions limit contribution to small-scale stor- the free space on the disk, they will be unable to meet age systems. these requirements of the defragmentation utility. 60 tuning it. Accordingly, deviating from that policy can re- sult in a loss of performance. The presence of data on the 50 file system can be viewed as an obstruction which causes 40 a deviation from the default allocation policy.