11/8/17 Where Are We? • Covered: COS 318: Operating Systems "! Management of CPU & concurrency "! Management of main memory & virtual memory "! Management of I/O devices File Structure •! Currently --- File Systems "! This lecture: File Structure Jaswinder Pal Singh Computer Science Department •! Then: Princeton University •! Naming and directories (http://www.cs.princeton.edu/courses/cos318/) •! Efficiency and performance •! Reliability and protection The File System Abstraction File System •! Open, close, read, write … named files, arranged in folders or directories ◆! Naming "! File name and directory File naming Physical Reality File System Abstraction ◆! File access File access "! Read, write, other operations block oriented byte oriented (char stream) Buffer cache ◆! Buffer cache "! Reduce client/server disk I/Os Management physical sector #’s named files Disk allocation ◆! Disk allocation Disk Drivers no protection users protected from ◆! Layout, mapping files to blocks each other ◆! Security, protection, reliability, durability data might be corrupted robust to machine failures ◆! Management tools if machine crashes 4 1 11/8/17 Topics Typical File Attributes ◆! File system structure •! Name ◆! Disk allocation and i-nodes •! Type – needed for systems that support different types ◆! Directory and link implementations •! Location – pointer to file location on device. ◆! Physical layout for performance •! Size – current file size. •! Protection – controls who can read, write, execute •! Time, date, and user identification – data for protection, security, and usage monitoring •! Information about files are kept in the directory structure, which is maintained on the disk 2 Typical Layout of a Disk Partition File Types – Name, Extension ◆! Boot block Boot block "! Code to load and boot OS File Type Usual extension Function Executable exe, com, bin or ready-to-run machine- ◆! Super-block defines a file system Superblock none language program "! File system info: type, no of blocks, ... Object obj, o complied, machine language, not linked "! File metadata File metadata area Source code c, p, pas, 177, source code in various (i-nodes in Unix) "! Information about / ptr to free blocks asm, a languages Batch bat, sh commands to the "! Location of descriptor of root directory Directory data command interpreter ◆! File metadata Text txt, doc textual data doc uments "! Each descriptor describes a file Word processor wp, tex, rrf, etc. various word-processor formats ◆! Directories File data Library lib, a libraries of routines "! Directory data (directory and file names) Print or view ps, dvi, gif ASCII or binary file ◆! File data Archive arc, zip, tar related files grouped "! Data blocks into one file, sometimes compressed. 3 2 11/8/17 Typical File Operations Open A File: Open(fd, name, access) ◆! Various checking (directory and file name lookup, authenticate) •! Create ◆! Copy the file descriptors into the in-memory data structure •! Write ◆! Create an entry in the open file table (system wide) •! Read ◆! Create an entry in PCB File system •! Reposition within file – file seek ◆! Return user a pointer to “file descriptor” info Process File •! Delete control Open-file table descriptors File block (system-wide) (Metadata) •! Truncate metadata •! Open(Fi) – search the directory structure on disk for Directories entry Fi, and move the content of entry to memory. •! Close (Fi) – move the content of entry Fi in memory to Open directory structure on disk. file File data pointer . array . 5 Translating from user to system view File system design constraints •! User wants to read 10 bytes from file starting at byte 2? •! For small files: "! Seek byte 2, fetch the block, read 10 bytes "! Small blocks for storage efficiency "! Files used together should be stored together •! User wants to write 10 bytes to file starting at byte 2? "! Seek byte 2, fetch the block, write 10 bytes, write out block •! For large files: "! Contiguous allocation for sequential access •! Everything inside file system is in whole size blocks "! Efficient lookup for random access "! Even getc and putc buffers 4096 bytes •! May not know at file creation whether file will become •! From now on, file is collection of blocks. small or large 3 11/8/17 File usage patterns File system design •! How do users access files? •! Data structures "! Sequential: bytes read in order "! Directories: file name -> file metadata "! Random: read/write element out of middle of arrays •! Store directories as files "! Content-based access: find me next byte starting with “COS318” "! File metadata: used to find file data blocks •! How are files used? "! Free map: list of free disk blocks "! Most files are small "! Large files use up most of the disk space •! How do we organize these data structures? "! Most transfers are small "! Large files account for most of the bytes transferred •! Bad news "! Need everything to be efficient Data Structures for Storage Allocation Data structures for disk management •! A file header for each file (part of the file meta-data) ◆! A File "! Disk sectors associated with each file "! Metadata "! A list of data blocks •! A data structure to track free space on disk ◆! Free space data structure "! Bit map "! Bit map indicating the status of 11111111111111111000000000000000 •! 1 bit per block (sector) disk blocks 00000111111110000000000111111111 … •! blocks numbered in cylinder-major order, why? "! Linked list that chains free "! blocks together 11000001111000111100000000000000 Linked list "! Others? "! Buddy system Free "! … link •! What about allocation for the blocks associated with a file? •! Let’s look at some ways of keeping addr link … track of file data size addr size 6 4 11/8/17 Contiguous Allocation Linked Files ◆! Allocate contiguous blocks on storage ◆! File structure (Alto) File header "! Bitmap: find N contiguous 0’s 3 "! File metadata points to 1st block on storage "! Linked list: find a region (size >= N) "! A block points to the next ◆! File metadata "! Last block has a NULL pointer "! First block in file ◆! Pros "! Number of blocks . "! Can grow files dynamically ◆! Pros "! File data tracked similarly to free "! Fast sequential access list of blocks "! Easy random access File A File B "! Doesn’t waste space ◆! Cons ◆! Cons null "! External fragmentation "! Random access: bad (what if file C needs 4 blocks) "! Unreliable: losing a block means "! Hard to grow files losing the rest 7 8 Linked files (cont’d) File Allocation Table (FAT) •! Idea is to keep the linked list metadata foo 217 (pointers) in memory, rather than on disk •! Allocation table at beginning of each volume 0 ◆! N entries for N blocks ◆! Want to keep it in memory •! File structure (MS-DOS) 217 619 "! A file is a linked list of blocks "! File metadata points to first block of file 399 "! The entry of first block points to next, … EOF •! Pros "! Simple 619 399 •! Cons "! Random access: still not good "! Wastes space - table for each file expensive to keep in memory FAT Allocation Table 8 5 11/8/17 DEMOS (Cray-1) Single-level Indexed File File metadata ◆! Idea •! User declares max size size0 "! Try contiguous allocation size1 •! File header holds array of pointers "! Allow non-contiguous Disk size to disk blocks 9 File header blocks ◆! File structure "! Small file metadata has 10 (base,size) pointers •! Pros: "! Big file has 10 indirect pointers "! Can grow up to a limit ◆! Pros & Cons size0 "! Random access is fast size "! Can grow (max 10GB) 1 "! No external fragmentation "! Fragmentation size9 •! Cons: size 0 "! Clumsy to grow beyond limit size1 size0 size1 "! Still lots of seeks size9 size9 11 Single-level indexed files (cont’d) Multi-level Indexed Files ! outer-index index table file 6 11/8/17 Hybrid Multi-level Indexed Files (Unix) Original Unix i-node ◆! 13 Pointers in a header ◆! Mode: file type, protection bits, setuid, setgid bits "! 10 direct pointers data ◆! Link count: no. of directory entries pointing to this file "! 11: 1-level indirect ◆! Uid: uid of the file owner "! 12: 2-level indirect data ◆! Gid: gid of the file owner "! 13: 3-level indirect 1 .2 . ◆! File size ◆! Pros & Cons 11 . 12 . data ◆! Times (access, modify, change) "! In favor of small files 13 "! Can grow . "! Limit is 16G . data ◆! 10 pointers to data blocks "! Can have lots of seeking ◆! Single indirect pointer . ◆! Double indirect pointer . data ◆! Triple indirect pointer 12 13 Extents Naming Files ◆! An extent is a variable number of Can name files via: blocks Block offset ◆! Main idea length ◆! Index (i-node number): Not easy for users to specify "! A file is a number of extents Starting block ◆! Text name: Need to map it to index "! XFS uses 8Kbyte blocks ◆! "! Max extent size is 2M blocks Icon: Need to map it to index or to text and then to index ◆! Index nodes need to have . "! Block offset ◆! Directories "! Length ◆! Table of file name, file index pairs "! Starting block ◆! Map name to file index (where to find the header) •! Microsoft NTFS, Linux EXT4, … ◆! Pros: little metadata, fast seq ◆! A directory is itself stored as a file access, can grow over time, less fragmentation ◆! Cons: external fragmentation still problem 14 15 7 11/8/17 Naming Tricks Directory Organization Examples •! Bootstrapping: Where do you start looking? ◆! Flat (CP/M) "! Root directory "! All files are in one directory "! inode #2 on the system ◆! Hierarchical (Unix) "! 0 and 1 used for other purposes "! /u/cos318/foo •! Special names: "! Directory is stored in a file containing (name, i-node) pairs "! Root directory: