File-System Implementation

Electrical and Computer Engineering Stephen Kim ([email protected])

ECE/IUPUI RTOS & APPS 1

File System Implementation n Structure n File System Implementation n Implementation n Allocation Methods n Free-Space Management n Efficiency and Performance n Recovery n Log-Structured File Systems n NFS

ECE/IUPUI RTOS & APPS 2

1 File-System Structure (1) n File system resides on secondary storage

u disks, diskette, CD-ROM, DVD

u – with or without disk interfaces

u virtual disk, or memory disk n File system design problems

u to define how the file system should look to the use

t file and its attribute

t operation or command allowed on a file

u to design algorithm and data structure to map the logical file system to the physical storage devices

ECE/IUPUI RTOS & APPS 3

File-System Structure (2) n File system organized into layers. u I/O Control layer – consisting of device drivers and interrupt handlers to transfer data b/w the main memory and the storage system. It translates high-level commands to low-level hardware-specific instructions u basic file system – to issue generic command to the device driver u file-organization module – to translate a logical address to a physical address, and free-space manager u logical file system – to manage metadata information (file-system structure, but not the content), to manage the directory system and symbolic file name, to maintain file structure via

ECE/IUPUI RTOS & APPS 4

2 File-System Structure (3) n File control block – storage structure consisting of information about a file.

u ownership, permissions, location of the file contents, … n Existing file systems

u UFS –

u ext, or – file system

u FAT, FAT32 – DOS file system

u NTFS – Windows NT or its successors file systems

ECE/IUPUI RTOS & APPS 5

File-System Implementation (1) n file system consists of on-disk structure and in-memory structure n on-disk structure u boot control block – information needed by the system to boot an OS from that disk. Typically the first block of a partition. Called boot block in UFS; partition in NTFS u partition control block – partition detail s.t. the # of blocks, the size of the blocks, free-block count, free- block pointers, and so forth. superblock (in UFS) or master file table (in NTFS) u directory structure u file control block – file’s detail. file permission, ownership, size, location of the data block.

ECE/IUPUI(UFS) RTOS & APPS 6

3 File-System Implementation (2) n in-memory structure

u used for file-system management and performance improvement via caching

u in-memory partition table – information about each mounted partition

u in-memory directory structure – holds the directory information of recently accessed directories

u system-wide open-file table – a copy of file control block of each open file

u per-process open-file table - has a pointer to the proper entry in system-wide open-file table, a pointer to the current location of the file for read/write operations

ECE/IUPUI RTOS & APPS 7

File-System Implementation (3) n to creat a file

u allocate a new FCB,

u read the proper directory into memory

u update it with the new file name and FCB

u write it back to the disk n In Unix, a directory is treated exactly as a file, but in Windows, a directory is treated separate entities from files

ECE/IUPUI RTOS & APPS 8

4 File-System Implementation (4) n the open system call u first search the system-wide open-file table to see if the file is already in use u if it is, a per-process open-file table entry is created pointing to the existing system-wide open-file table, and increment the system-wide entry’s open count is incremented u if not, a system-wide open-file table entry is created, a per-process open-file table entry is created pointing to the system-wide open-file table entry

ECE/IUPUI RTOS & APPS 9

File-System Implementation (5) n Once a file is opened with a file name,

u the file name is not needed to keep in open-file table.

u instead, the open system call returns a pointer or unique ID for the per-process table

u it is called a file descriptor in Unix, or a file handle in Windows(DOS)

u All successive operations on the file is accomplished using the file descriptor.

ECE/IUPUI RTOS & APPS 10

5 File-System Implementation (6) n When a process closes the file,

u the per-process table entry is removed, and

u the system-wide entry’s open count is decremented n When all users opened the file close it,

u the updated file information is copied back to the on-disk directory structure and the system- wide open-file table entry is removed.

ECE/IUPUI RTOS & APPS 11

In-Memory File System Structures

ECE/IUPUI RTOS & APPS 12

6 Partitions and Mounting n Each partition can be either u raw with no file system. For example, t Unix swap space can use a raw partition using its own internal format. t Boot information is also stored in raw format. Why? u formatted file system n boot loader understands multiple file system and multiple in term of how-to-start n root partition contains the OS kernel, and is mounted at boot time. u other partitions can be mounted at boot time automatically by boot script, or manually later

ECE/IUPUI RTOS & APPS 13

Virtual File Systems n Virtual File Systems (VFS) provide an object- oriented way of implementing file systems.

u abstraction of file system based on file- representation structure (vnode)

u to support unilaternal interface to many different file systems including NFS n VFS allows the same system call interface (API) to be used for different types of file systems. n The API is to the VFS interface, rather than any specific type of file system

ECE/IUPUI RTOS & APPS 14

7 Schematic View of

ECE/IUPUI RTOS & APPS 15

Directory Implementation n Linear list of file names with pointer to the data blocks. u linear search required u simple to program u time-consuming to execute u How-to reuse the directory entry t special file name t special character t Boolean bit (used or unused) t free directory pool t compact – moving the last entry to the empty slot. n Hash Table – linear list with hash data structure. u decreases directory search time u collisions – two file names hash to the same location u overflow – resizing and remapping the directory structure,

ECE/IUPUI RTOS & APPS 16

8 Allocation Methods n An allocation method refers to how disk blocks are allocated for files: n Contiguous allocation n Linked allocation n Indexed allocation

ECE/IUPUI RTOS & APPS 17

Contiguous Allocation n Each file occupies a set of contiguous blocks on the disk. n Simple – only starting location (block #) and length (number of blocks) are required. n sequential and direct access can be supported. n Wasteful of space (dynamic storage-allocation problem).

u first fit, best fit, random fit, … n External fragmentation

u compression – all free space into one contiguous space n Files cannot grow.

u overestimate the space needed

t results in another internal fragmentation

ECE/IUPUI RTOS & APPS 18

9 Contiguous Allocation of Disk Space

ECE/IUPUI RTOS & APPS 19

Extent-Based Systems n Many newer file systems (I.e. ) use a modified contiguous allocation scheme. n -based file systems allocate disk blocks in extents. n An extent is a contiguous block of disks. Extents are allocated for file allocation. A file consists of one or more extents.

ECE/IUPUI RTOS & APPS 20

10 Linked Allocation (1) n Each file is a linked list of disk blocks: blocks may be scattered anywhere on the disk. n The directory contains a pointer to the first and last blocks of the file. n Advantage u No external fragmentation

u Free-space management system – no waste of space u Files can grow

ECE/IUPUI RTOS & APPS 21

Linked Allocation (2)

ECE/IUPUI RTOS & APPS 22

11 Linked Allocation (3) n Disadvantages u practically sequential access only. Inefficient direct- access u space required for the pointers u reliability t once a pointer is disrupted, the error results in linking into the free-space or another file n FAT (), a variation u used in MS-DOS u set aside a portion of disk for pointers u FAT should be cached for performance. Otherwise, serious disk head seek.

ECE/IUPUI RTOS & APPS 23

File-Allocation Table

ECE/IUPUI RTOS & APPS 24

12 Indexed Allocation (1) n Put all pointers together into the index block. n Each file has its own index block. n The directory contains the address of the index block n Random access n Dynamic access without external fragmentation n overhead of index block

ECE/IUPUI RTOS & APPS 25

Example of Indexed Allocation

ECE/IUPUI RTOS & APPS 26

13 Indexed Allocation (2) n Mapping from logical to physical in a file of maximum size of 256K words and block size of 512 words. We need only 1 block for index block. 256K / 512 = 512 maximum entries n What should be the size of index block?

u As small as possible not to waste space

u As large as possible to support a large

ECE/IUPUI RTOS & APPS 27

Indexed Allocation, Linked Scheme n Link several index blocks n No limit on size of file

ECE/IUPUI RTOS & APPS 28

14 Indexed Allocation, Two-level index n a first-level index block to point into a set of second-level index blocks n The second-level index block points to the file blocks. n For example, given 2048- block and 4-byte word, what is the maximum file size the two-level index scheme can support?

ECE/IUPUI RTOS & APPS 29

Indexed Allocation, Combined Scheme n multiple level of index block in the file’s inode

u direct blocks

u single indirect blocks

u double indirect block

u triple indirect block n For given 2048-byte block and 4-byte word, what is the maximum file size the above combined scheme can support?

ECE/IUPUI RTOS & APPS 30

15 Combined Scheme in UNIX

ECE/IUPUI RTOS & APPS 31

Free-Space Management n To keep track of free disk blocks, the OS maintains a free-space list. n To create a new file, the OS search the free-space list for the required amount of space, allocate that space to the new file. n Methods of implementing the free-space list

u bit vector

u linked list

u grouping

u counting

ECE/IUPUI RTOS & APPS 32

16 Free-Space Mgmt, Bit Vector (1) n Each block is represented by 1 bit n Advantage u simple to implement u efficient to find the first free block, or n consecutive free block because most of processors support efficient bit-manipulation instruction. n Block number calculation (number of bits per word) *(number of 0-value words) +offset of first 1 bit

0 1 2 n-1 0 Þ block[i] free … bit[i] = 1 Þ block[i] occupied 678

ECE/IUPUI RTOS & APPS 33

Free-Space Mgmt, Bit Vector (2) n Inefficient if the entire vector is not in main memory n Space requirement of bit vector. Example: Suppose that we have 1 GB hard disk, 4KB block size is supported. How much space is needed for maintaining a free-space list using the bit vector method? How about 10 GB hard disk? n The bit-vector method is inefficient if the entire vector is not kept in main memory.

ECE/IUPUI RTOS & APPS 34

17 Free-Space Mgmt, Linked List n To link together all the free disk blocks, by keeping a pointer to the first block in a special location on the disk and caching it in memory. n The first block contains a pointer to the next block, which contains a pointer to the next block, and so forth (See the next figure) n Advantage u No waste of space (simply a pointer to the first block) n Disadvantage u Not efficient to traverse the list (do we need?) u Cannot get contiguous space easily u Front blocks used more heavily that back-blocks. t Flash memory has a lifetime of rewriting

t ECE/IUPUI Solution ? RTOS & APPS 35

Linked Free Space List on Disk

ECE/IUPUI RTOS & APPS 36

18 Free-Space Mgmt, Grouping n A free block contains n pointers to free blocks n The last block among n pointers contains another n free blocks. n Example Suppose that the size of a disk block and the size of a pointer are 4KB and 4-byte, the system is implement the grouped linked list for free-space list. What is the number of disk IO to obtain 10,000 blocks?

ECE/IUPUI RTOS & APPS 37

Free-Space Mgmt, Counting n Observation u When disk space is allocated, several contiguous blocks will be needed at a time u When disk space is de-allocated, several contiguous blocks may be freed at a time. n Run-length of free blocks u keep the address of the first free blocks, and u the number n of free contiguous blocks following the first block. n Each entry in the free-space list consists of disk address and a count.

ECE/IUPUI RTOS & APPS 38

19