File-System Implementation Electrical and Computer Engineering Stephen Kim ([email protected]) ECE/IUPUI RTOS & APPS 1 File System Implementation n File System Structure n File System Implementation n Directory Implementation n Allocation Methods n Free-Space Management n Efficiency and Performance n Recovery n Log-Structured File Systems n NFS ECE/IUPUI RTOS & APPS 2 1 File-System Structure (1) n File system resides on secondary storage u disks, diskette, CD-ROM, DVD u flash memory – with or without disk interfaces u virtual disk, or memory disk n File system design problems u to define how the file system should look to the use t file and its attribute t operation or command allowed on a file u to design algorithm and data structure to map the logical file system to the physical storage devices ECE/IUPUI RTOS & APPS 3 File-System Structure (2) n File system organized into layers. u I/O Control layer – consisting of device drivers and interrupt handlers to transfer data b/w the main memory and the storage system. It translates high-level commands to low-level hardware-specific instructions u basic file system – to issue generic command to the device driver u file-organization module – to translate a logical address to a physical address, and free-space manager u logical file system – to manage metadata information (file-system structure, but not the content), to manage the directory system and symbolic file name, to maintain file structure via file control block ECE/IUPUI RTOS & APPS 4 2 File-System Structure (3) n File control block – storage structure consisting of information about a file. u ownership, permissions, location of the file contents, … n Existing file systems u UFS – Unix file system u ext, or ext2 – Linux file system u FAT, FAT32 – DOS file system u NTFS – Windows NT or its successors file systems ECE/IUPUI RTOS & APPS 5 File-System Implementation (1) n file system consists of on-disk structure and in-memory structure n on-disk structure u boot control block – information needed by the system to boot an OS from that disk. Typically the first block of a partition. Called boot block in UFS; partition boot sector in NTFS u partition control block – partition detail s.t. the # of blocks, the size of the blocks, free-block count, free- block pointers, and so forth. superblock (in UFS) or master file table (in NTFS) u directory structure u file control block – file’s detail. file permission, ownership, size, location of the data block. inode ECE/IUPUI(UFS) RTOS & APPS 6 3 File-System Implementation (2) n in-memory structure u used for file-system management and performance improvement via caching u in-memory partition table – information about each mounted partition u in-memory directory structure – holds the directory information of recently accessed directories u system-wide open-file table – a copy of file control block of each open file u per-process open-file table - has a pointer to the proper entry in system-wide open-file table, a pointer to the current location of the file for read/write operations ECE/IUPUI RTOS & APPS 7 File-System Implementation (3) n to creat a file u allocate a new FCB, u read the proper directory into memory u update it with the new file name and FCB u write it back to the disk n In Unix, a directory is treated exactly as a file, but in Windows, a directory is treated separate entities from files ECE/IUPUI RTOS & APPS 8 4 File-System Implementation (4) n the open system call u first search the system-wide open-file table to see if the file is already in use u if it is, a per-process open-file table entry is created pointing to the existing system-wide open-file table, and increment the system-wide entry’s open count is incremented u if not, a system-wide open-file table entry is created, a per-process open-file table entry is created pointing to the system-wide open-file table entry ECE/IUPUI RTOS & APPS 9 File-System Implementation (5) n Once a file is opened with a file name, u the file name is not needed to keep in open-file table. u instead, the open system call returns a pointer or unique ID for the per-process table u it is called a file descriptor in Unix, or a file handle in Windows(DOS) u All successive operations on the file is accomplished using the file descriptor. ECE/IUPUI RTOS & APPS 10 5 File-System Implementation (6) n When a process closes the file, u the per-process table entry is removed, and u the system-wide entry’s open count is decremented n When all users opened the file close it, u the updated file information is copied back to the on-disk directory structure and the system- wide open-file table entry is removed. ECE/IUPUI RTOS & APPS 11 In-Memory File System Structures ECE/IUPUI RTOS & APPS 12 6 Partitions and Mounting n Each partition can be either u raw format with no file system. For example, t Unix swap space can use a raw partition using its own internal format. t Boot information is also stored in raw format. Why? u formatted file system n boot loader understands multiple file system and multiple operating system in term of how-to-start n root partition contains the OS kernel, and is mounted at boot time. u other partitions can be mounted at boot time automatically by boot script, or manually later ECE/IUPUI RTOS & APPS 13 Virtual File Systems n Virtual File Systems (VFS) provide an object- oriented way of implementing file systems. u abstraction of file system based on file- representation structure (vnode) u to support unilaternal interface to many different file systems including NFS n VFS allows the same system call interface (API) to be used for different types of file systems. n The API is to the VFS interface, rather than any specific type of file system ECE/IUPUI RTOS & APPS 14 7 Schematic View of Virtual File System ECE/IUPUI RTOS & APPS 15 Directory Implementation n Linear list of file names with pointer to the data blocks. u linear search required u simple to program u time-consuming to execute u How-to reuse the directory entry t special file name t special character t Boolean bit (used or unused) t free directory pool t compact – moving the last entry to the empty slot. n Hash Table – linear list with hash data structure. u decreases directory search time u collisions – two file names hash to the same location u overflow – resizing and remapping the directory structure, ECE/IUPUI RTOS & APPS 16 8 Allocation Methods n An allocation method refers to how disk blocks are allocated for files: n Contiguous allocation n Linked allocation n Indexed allocation ECE/IUPUI RTOS & APPS 17 Contiguous Allocation n Each file occupies a set of contiguous blocks on the disk. n Simple – only starting location (block #) and length (number of blocks) are required. n sequential and direct access can be supported. n Wasteful of space (dynamic storage-allocation problem). u first fit, best fit, random fit, … n External fragmentation u compression – all free space into one contiguous space n Files cannot grow. u overestimate the space needed t results in another internal fragmentation ECE/IUPUI RTOS & APPS 18 9 Contiguous Allocation of Disk Space ECE/IUPUI RTOS & APPS 19 Extent-Based Systems n Many newer file systems (I.e. Veritas File System) use a modified contiguous allocation scheme. n Extent-based file systems allocate disk blocks in extents. n An extent is a contiguous block of disks. Extents are allocated for file allocation. A file consists of one or more extents. ECE/IUPUI RTOS & APPS 20 10 Linked Allocation (1) n Each file is a linked list of disk blocks: blocks may be scattered anywhere on the disk. n The directory contains a pointer to the first and last blocks of the file. n Advantage u No external fragmentation u Free-space management system – no waste of space u Files can grow ECE/IUPUI RTOS & APPS 21 Linked Allocation (2) ECE/IUPUI RTOS & APPS 22 11 Linked Allocation (3) n Disadvantages u practically sequential access only. Inefficient direct- access u space required for the pointers u reliability t once a pointer is disrupted, the error results in linking into the free-space or another file n FAT (File Allocation Table), a variation u used in MS-DOS u set aside a portion of disk for pointers u FAT should be cached for performance. Otherwise, serious disk head seek. ECE/IUPUI RTOS & APPS 23 File-Allocation Table ECE/IUPUI RTOS & APPS 24 12 Indexed Allocation (1) n Put all pointers together into the index block. n Each file has its own index block. n The directory contains the address of the index block n Random access n Dynamic access without external fragmentation n overhead of index block ECE/IUPUI RTOS & APPS 25 Example of Indexed Allocation ECE/IUPUI RTOS & APPS 26 13 Indexed Allocation (2) n Mapping from logical to physical in a file of maximum size of 256K words and block size of 512 words. We need only 1 block for index block. 256K / 512 = 512 maximum entries n What should be the size of index block? u As small as possible not to waste space u As large as possible to support a large file size ECE/IUPUI RTOS & APPS 27 Indexed Allocation, Linked Scheme n Link several index blocks n No limit on size of file ECE/IUPUI RTOS & APPS 28 14 Indexed Allocation, Two-level index n a first-level index block to point into a set of second-level index blocks n The second-level index block points to the file blocks.