Suvfs: a Virtual File System in Userspace That Supports Large Files
Total Page:16
File Type:pdf, Size:1020Kb
2013 IEEE International Conference on Big Data suvfs: A virtual file system in userspace that supports large files Wasim Ahmad Bhat S.M.K. Quadri Department of Computer Sciences Department of Computer Sciences University of Kashmir University of Kashmir Email: [email protected] Email: [email protected] Abstract—As a consequence of voluminous growth and prolif- space is wasted, and 4) still there is an upper limit. eration of digital data, the file system size and count limitations In this paper, we propose a novel approach to overcome have become problematic. The fact is that file systems use fixed the file size limitation of file systems without instrumenting length fields within their metadata structures to keep track of the file system by using the concept of virtual-unification volume size, file size and file count, and hence need design (and to present a large virtual file that spans over a number of source) modification to cope up with this growth. In this paper, legitimate sized physical files. The proposal, namely suvfs, is we propose a Scalable User-space Virtual File System, namely suvfs, which when mounted on top of any file system extends its implemented as a virtual file system using FUSE framework capability to store and process large files not natively supported [1] and can be layered on top of any file system in user-space by the file system. It works by exploiting the concept of virtual to extend its capability to support large files. Furthermore, unification to present a large virtual file that spans over a number we empirically evaluated the performance overhead added of legitimate sized physical files. It does so without modifying by the suvfs algorithms by layering it on top of FAT32 file user applications, system libraries, system calls and even file system. After exercising the file system with Sprite LFS large- systems. We implemented it using FUSE framework & evaluated file micro benchmark, the results indicate that the average it for performance overhead added by layering it over FAT32 growth of performance overhead added by suvfs algorithms file system. The results indicate that there is a very minimal with doubling the file size is 0.51% for reading, 6.73% for performance hit encountered by FAT32 with suvfs mounted on writing and 9.54% for deletion. top for sequential reading, while as for sequential writing & deletion the performance hit increases with file size but is largely due to FUSE framework & not by suvfs. II. BACKGROUND &RELATED WORK Virtual Unification provides a merged view of several I. INTRODUCTION directories, without physically merging them. This allows the files to remain physically separate, but appear as if they reside File System Scalability is defined as the ability of a file in one location. As such, Virtual Unification presents a large system to support large volumes, large files, large directories directory wherein there is no limit on the number of files and and large number of files while still providing I/O performance. directories it contains. Conversely, Virtual Unification presents Thus, scalability is a two part term: Scalable to Capacity a file system that is scalable to store and process large number and Scalable to Performance. To what extent a file system of files and directories even though they may actually reside at is scalable to capacity, depends on how it stores information different locations within the file system or may even belong to about files, directories and the file system itself. As an ex- different file systems. Furthermore, as it can merge directories ample, if the file size is stored as 32-bit integer, then no file from different file systems into one, the file system so created on that file system can exceed 232 bytes (4 GB). The fact is is scalable to large volume size if the specified directories that file systems use fixed length fields within their on-disk are mount points of different file systems. Many proposals for metadata structures to keep track of volume size, file size, file virtual-unification of directories (simple or mount-points) have count, and so on. Thus being fixed in length, they are not surfaced from time to time; however the following text briefly scalable and hence limit the file system. An obvious solution discusses the most influential and popular designs. to this problem would be the usage of dynamic length fields. Plan-9 developed by Bell Labs can connect different ma- In fact, the limitation on filename length in ext2 file system chines, file servers, and networks, and offers a binding ser- was mitigated by replacing the fixed length filename field with vice that enables multiple directories to be grouped under a a dynamic one. However, in general there are three problems common namespace [2]. Similarly, 3DFS, also developed by associated with this approach; 1) it hinders the performance as AT&T Bell Labs for source code management, maintains a per- extra computations are to be performed, 2) an extra metadata process table that contains directories and a location in the file structure is added which demands space and management, and system that the directories overlay [3]. This technique is called 3) because very little space is allocated to metadata structures, viewpathing, and it presents a view of directories stacked over they can’t grow dynamically after certain limit. one another. Although, the Translucent File System Another possible solution would be to keep a large fixed length (TFS) released in SunOS 4.1 [4] also provides a viewpathing field whose length would never be possibly defeated. This idea solution like 3DFS, but is an improved one as it better has four limitations; 1) the memory requirements are high, 2) adheres to UNIX semantics when deleting a file. Moreover, a lot of space is occupied by metadata structures, 3) a lot of Union-Mounts, which were implemented on 4.4BSD-Lite, 978-1-4799-1293-3/13/$31.00 ©2013 IEEE 7 merge directories and their trees to provide a unified view [5]. for processing. The operations performed on this large virtual This structure, called the union-stack, permits directories to file are reflected in the associated fragments. Third, to avoid be dynamically added either to the top or the bottom of the individual access to the fragments and to exclude them from view. Unionfs proposed by Wright et al. [6] is the most directory listing, suvfs filters fragments from simple files using improved union file system as it maintains UNIX semantics the magic string in their filenames. while offering advanced unification features such as dynamic Furthermore, to ensure that the split and unification is trans- insertion and removal of namespaces at any point in the parent to user applications, suvfs uses file system layering to unified view. Unionfs allows users to specify a series of perform transparent splitting and unification, thus avoiding any directories and a mount point which presents the union of design or source modification of user applications, libraries, specified directories. Although, originally intended for name- system calls, file systems, etc. Layering allows trapping, pre- space unification, union file systems can also be used for processing and post-processing of file system syscalls targeted snapshotting, by marking some data sources read-only and to the below mounted file system. As such, layering is trans- then utilizing copy-on-write for the read-only sources [7]. parent to both; user applications on one side and native file Finally, mhddfs is another union file system developed by systems on other side. This mitigates the design goals of suvfs D.E. Oboukhov that allows to unite several mount points to enumerated above at 2 and 3. the single one [8]. mhddfs simulates one big file system suvfs is implemented as a file system layer in user-space using by combining several hard drives or network file systems. As FUSE framework [1]. This design decision mitigates design such, mhddfs can be argued as to be a file system scalable to goal enumerated above at 4. Figure 1 shows the design of large volume size and file count in the same way as other union suvfs using FUSE framework. file systems can be. However, mhddfs goes a step further by supporting load balancing. In mhddfs, if an overflow File System Operations suvfs arises while writing to some unified mount point, the file content already written will be transferred to another unified mount point containing enough of free space for the file. The transferring is processed on-the-fly, fully transparent for the application that is writing. large virtual file associated fragments It is worth mentioning here that no proposal (based on concept User-Mode of virtual unification) that directly claims to have overcome any size or count limitation of the file systems exists. Never- VFS theless, this paper proposes the one. FUSE File System III. DESIGN OF suvfs Kernel-Mode suvfs is a scalable user-space virtual file system that breaks the maximum file size limitation of any file system when Fig. 1. Design of suvfs using FUSE framework mounted on top of it. suvfs can extend capability of any file system to handle large file sizes without modifying the design and source of the file system. Also, with suvfs on top, there is IV. IMPLEMENTATION OF suvfs logically no upper limit on the file size. Therefore, the design suvfs is implemented using FUSE framework [1]. FUSE goals of suvfs are as follows: is an acronym for Filesystem in UserSpacE and is used to 1) To extend the capability of a file system to handle develop full-fledged file systems and to extend existing ones. large files with logically no upper limit on the file The file systems so created run in the user space.