Recall: A Little Queuing Theory: Some Results • Assumptions: CS162 – System in equilibrium; No limit to the queue Operating Systems and – Time between successive arrivals is random and memoryless Systems Programming Queue Server Lecture 19 Arrival Rate Service Rate μ=1/Tser File Systems (Con’t), • Parameters that describe our system: MMAP, Buffer Cache – : mean number of arriving customers/second –Tser: mean time to service a customer (“m1”) –C: squared coefficient of variance = 2/m12 –μ: service rate = 1/Tser th April 11 , 2019 –u: server utilization (0u1): u = /μ = Tser • Parameters we wish to compute: Prof. Ion Stoica –Tq: Time spent in queue http://cs162.eecs.Berkeley.edu –Lq: Length of queue = Tq (by Little’s law) • Results: –Memoryless service distribution (C = 1): » Called M/M/1 queue: Tq = Tser x u/(1 – u) –General service distribution (no restrictions), 1 server: » Called M/G/1 queue: Tq = Tser x ½(1+C) x u/(1 – u)) 4/11/19 Kubiatowicz CS162 © UCB Spring 2019 Lec 19.2
Recall: FAT Properties FAT Assessment – Issues • File is collection of disk blocks • Time to find block (large files) ?? (Microsoft calls them “clusters”) FAT Disk Blocks FAT Disk Blocks • FAT is array of integers mapped • Block layout for file ??? 1-1 with disk blocks 0: 0: 0: 0: File #1 File #1 – Each integer is either: • Sequential Access ??? » Pointer to next block in file; or 31: File 31, Block 0 31: File 31, Block 0 » End of file flag; or File 31, Block 1 File 31, Block 1 » Free block flag File 63, Block 1 • Random Access ??? File 63, Block 1 • File Number is index of root of block list for the file • Fragmentation ??? File 31, Block 3 File 31, Block 3 – Follow list to get block # – MSDOS defrag tool File 63, Block 0 File 63, Block 0 – Directory must map name to 63: 63: block number at start of file File #2 File 31, Block 2 • Small files ??? File #2 File 31, Block 2 • But: Where is FAT stored? – Beginning of disk, before the data blocks N-1: N-1: • Big files ??? N-1: N-1: – Usually 2 copies (to handle errors) 4/11/19 Kubiatowicz CS162 © UCB Spring 2019 Lec 19.3 4/11/19 Kubiatowicz CS162 © UCB Spring 2019 Lec 19.4 What about the Directory? Directory Structure (cont’d)
• Essentially a file containing • How many disk accesses to resolve “/my/book/count”?
4/11/19 Kubiatowicz CS162 © UCB Spring 2019 Lec 19.5 4/11/19 Kubiatowicz CS162 © UCB Spring 2019 Lec 19.6
Many Huge FAT Security Holes! Characteristics of Files • FAT has no access rights – No way, even in principle, to track ownership of data • FAT has no header in the file blocks – No way to enforce control over data, since all processes have access of FAT table – Just follow pointer to disk blocks • Just gives an index into the FAT to read data – (file number = block number) – Could start in middle of file or access deleted data
4/11/19 Kubiatowicz CS162 © UCB Spring 2019 Lec 19.7 4/11/19 Kubiatowicz CS162 © UCB Spring 2019 Lec 19.8 Characteristics of Files Characteristics of Files • Most files are small, growing numbers of files • Most of the space is occupied by the rare big ones over time
4/11/19 Kubiatowicz CS162 © UCB Spring 2019 Lec 19.9 4/11/19 Kubiatowicz CS162 © UCB Spring 2019 Lec 19.10
Unix File System (1/2) Unix File System (2/2)
• Original inode format appeared in BSD 4.1 • Metadata associated with the file – Berkeley Standard Distribution Unix – Rather than in the directory that points to it – Part of your heritage! • UNIX Fast File System (FFS) BSD 4.2 Locality Heuristics: – Similar structure for Linux Ext2/3 – Block group placement • File Number is index into inode arrays – Reserve space • Multi-level index structure • Scalable directory structure – Great for little and large files – Asymmetric tree with fixed sized blocks
4/11/19 Kubiatowicz CS162 © UCB Spring 2019 Lec 19.11 4/11/19 Kubiatowicz CS162 © UCB Spring 2019 Lec 19.12 Administrivia Inode Structure • Watch for group evalutions – We are asking for additional information about both project 1 and • inode metadata project 2. Tell about your group dynamics » Not about getting points for yourself, but about evaluating partners – Please collaborate with your group! (potentially a zero-sum game if you are not contributing to your group’s success!) • Final Midterm: 5/2 – All topics up to and including that last lecture before then – You can bring 3 sheets of notes (both sides) • Help to name new planet: https://2007or10.name – You have until May 10th, 2019 – In the Kuiper Belt, beyond Neptune, smaller than Pluto and Eris – Currently named “2007 OR10” » Gonggong: A Chinese water god with red hair and a serpent-like tail » Holle: A European winter goddess of fertility, rebirth and women » Vili: A Nordic deity. Together with his brothers Odin and Ve, he defeated frost giant Ymir and used Ymir’s body to create universe 4/11/19 Kubiatowicz CS162 © UCB Spring 2019 Lec 19.13 4/11/19 Kubiatowicz CS162 © UCB Spring 2019 Lec 19.14
File Attributes Data Storage
• Small files: 12 pointers direct to data blocks • inode metadata Direct pointers
4kB blocks sufficient for files up to 48KB
User Group 9 basic access control bits - UGO x RWX Setuid bit - execute at owner permissions rather than user Setgid bit - execute at group’s permissions
4/11/19 Kubiatowicz CS162 © UCB Spring 2019 Lec 19.15 4/11/19 Kubiatowicz CS162 © UCB Spring 2019 Lec 19.16 Data Storage UNIX BSD 4.2 (1984) (1/2)
• Large files: 1,2,3 level indirect pointers • Same as BSD 4.1 (same file header and triply indirect blocks), except incorporated ideas from Cray Operating Indirect pointers System: - point to a disk block – Uses bitmap allocation in place of freelist containing only pointers - 4 kB blocks => 1024 ptrs – Attempt to allocate files contiguously => 4 MB @ level 2 – 10% reserved disk space => 4 GB @ level 3 48 KB => 4 TB @ level 4 – Skip-sector positioning (mentioned later) +4 MB
+4 GB
+4 TB
4/11/19 Kubiatowicz CS162 © UCB Spring 2019 Lec 19.17 4/11/19 Kubiatowicz CS162 © UCB Spring 2019 Lec 19.18
UNIX BSD 4.2 (1984) (2/2) Attack of the Rotational Delay • Problem 2: Missing blocks due to rotational delay • Problem: When create a file, don’t know how big it will – Issue: Read one block, do processing, and read next block. In meantime, disk has continued turning: missed next block! Need 1 become (in UNIX, most writes are by appending) revolution/block! – How much contiguous space do you allocate for a file? Skip Sector – In BSD 4.2, just find some range of free blocks » Put each new file at the front of different range Track Buffer » To expand a file, you first try successive blocks in (Holds complete track) bitmap, then choose new range of blocks – Solution1: Skip sector positioning (“interleaving”) – Also in BSD 4.2: store files from same directory near » Place the blocks from one file on every other block of a track: give time each other for processing to overlap rotation » Can be done by OS or in modern drives by the disk controller – Solution 2: Read ahead: read next block right after first, even if • Fast File System (FFS) application hasn’t asked for it yet » This can be done either by OS (read ahead) – Allocation and placement policies for BSD 4.2 » By disk itself (track buffers) - many disk controllers have internal RAM that allows them to read a complete track • Modern disks + controllers do many things “under the covers” – Track buffers, elevator algorithms, bad block filtering 4/11/19 Kubiatowicz CS162 © UCB Spring 2019 Lec 19.19 4/11/19 Kubiatowicz CS162 © UCB Spring 2019 Lec 19.20 Where are inodes Stored? Where are inodes Stored? • In early UNIX and DOS/Windows’ FAT file system, • Later versions of UNIX moved the header information to be headers stored in special array in outermost cylinders closer to the data blocks – Often, inode for file stored in same “cylinder group” as parent directory of the file (makes an ls of that directory run fast) • Header not stored anywhere near the data blocks • Pros: – To read a small file, seek to get header, seek back to data – UNIX BSD 4.2 puts bits of file header array on many cylinders – For small directories, can fit all data, file headers, etc. in same • Fixed size, set when disk is formatted cylinder no seeks! – At formatting time, a fixed number of inodes are created – File headers much smaller than whole block (a few hundred – Each is given a unique number, called an “inumber” bytes), so multiple headers fetched from disk at same time – Reliability: whatever happens to the disk, you can find many of the files (even if directories disconnected) • Part of the Fast File System (FFS) – General optimization to avoid seeks
4/11/19 Kubiatowicz CS162 © UCB Spring 2019 Lec 19.21 4/11/19 Kubiatowicz CS162 © UCB Spring 2019 Lec 19.22
4.2 BSD Locality: Block Groups 4.2 BSD Locality: Block Groups • File system volume is divided into a set of block groups • First-Free allocation of new file blocks – Close set of tracks – To expand file, first try successive • Data blocks, metadata, and free blocks in bitmap, then space interleaved within block group choose new range of blocks – Few little holes at start, big – Avoid huge seeks between sequential runs at end of group user data and system structure – Avoids fragmentation • Put directory and its files in common block group – Sequential layout for big files • Important: keep 10% or more free! – Reserve space in the Block Group
4/11/19 Kubiatowicz CS162 © UCB Spring 2019 Lec 19.23 4/11/19 Kubiatowicz CS162 © UCB Spring 2019 Lec 19.24 UNIX 4.2 BSD FFS First Fit Block Allocation UNIX 4.2 BSD FFS •Pros – Efficient storage for both small and large files – Locality for both small and large files – Locality for metadata and data – No defragmentation necessary!
• Cons – Inefficient for tiny files (a 1 byte file requires both an inode and a data block) – Inefficient encoding when file is mostly contiguous on disk – Need to reserve 10-20% of free space to prevent fragmentation • Fills in the small holes at the start of block group • Avoids fragmentation, leaves contiguous free space at end
4/11/19 Kubiatowicz CS162 © UCB Spring 2019 Lec 19.25 4/11/19 Kubiatowicz CS162 © UCB Spring 2019 Lec 19.26
Linux Example: Ext2/3 Disk Layout A bit more on directories • Disk divided into block • Stored in files, can be read, but typically don’t groups – System calls to access directories /usr – Provides locality – open / creat traverse the structure – Each group has two block- – mkdir /rmdir add/remove entries sized bitmaps (free /usr/lib blocks/inodes) – link / unlink (rm) /usr/lib4.3 – Block sizes settable » Link existing file to a directory at format time: • Not in FAT ! 1K, 2K, 4K, 8K… »Forms a DAG • Actual inode structure similar • When can file be deleted? /usr/lib/foo to 4.2 BSD – Maintain ref-count of links to the file – with 12 direct pointers – Delete after the last reference is gone /usr/lib4.3/foo • Ext3: Ext2 with Journaling • libc support – Several degrees of – DIR * opendir (const char *dirname) protection with comparable overhead – struct dirent * readdir (DIR *dirstream) • Example: create a file1.dat – int readdir_r (DIR *dirstream, struct dirent *entry, under /dir1/ in Ext3 struct dirent **result) 4/11/19 Kubiatowicz CS162 © UCB Spring 2019 Lec 19.27 4/11/19 Kubiatowicz CS162 © UCB Spring 2019 Lec 19.28 Links Large Directories: B-Trees (dirhash)
• Hard link in FreeBSD, NetBSD, OpenBSD – Sets another directory entry to contain the file number for the file – Creates another name (path) for the file – Each is “first class”
• Soft link or Symbolic Link or Shortcut – Directory entry contains the path and name of the file – Map one name to another name
4/11/19 Kubiatowicz CS162 © UCB Spring 2019 Lec 19.29 4/11/19 Kubiatowicz CS162 © UCB Spring 2019 Lec 19.30
NTFS NTFS • Master File Table • New Technology File System (NTFS) – Database with Flexible 1KB entries for metadata/data – Default on Microsoft Windows systems – Variable-sized attribute records (data or metadata) – Extend with variable depth tree (non-resident) • Variable length extents • Extents – variable length contiguous regions – Rather than fixed blocks – Block pointers cover runs of blocks • Everything (almost) is a sequence of
• Directories organized in B-tree structure by default http://ntfs.com/ntfs-mft.htm 4/11/19 Kubiatowicz CS162 © UCB Spring 2019 Lec 19.31 4/11/19 Kubiatowicz CS162 © UCB Spring 2019 Lec 19.32 NTFS Small File NTFS Medium File
Create time, modify time, access time, Owner id, security specifier, flags (RO, hidden, sys)
data attribute
Attribute list
4/11/19 Kubiatowicz CS162 © UCB Spring 2019 Lec 19.33 4/11/19 Kubiatowicz CS162 © UCB Spring 2019 Lec 19.34
NTFS Multiple Indirect Blocks
4/11/19 Kubiatowicz CS162 © UCB Spring 2019 Lec 19.35 4/11/19 Kubiatowicz CS162 © UCB Spring 2019 Lec 19.36 Memory Mapped Files Recall: Who Does What, When?
• Traditional I/O involves explicit transfers between buffers Process virtual address physical address in process address space to/from regions of a file page# instruction MMU frame# – This involves multiple copies into caches in memory, plus PT system calls offset retry page fault exception frame# • What if we could “map” the file directly into an empty region of our address space Operating System offset update PT entry – Implicitly “page it in” when we read it Page Fault Handler – Write it and “eventually” page it out load page from disk • Executable files are treated this way when we exec the process!!
scheduler
4/11/19 Kubiatowicz CS162 © UCB Spring 2019 Lec 19.37 4/11/19 Kubiatowicz CS162 © UCB Spring 2019 Lec 19.38
Using Paging to mmap() Files mmap() system call
Process virtual address physical address page# instruction MMU PT frame# offset retry page fault exceptionRead File contents Operating System from memory!Create PT entries Page Fault Handler for mapped region as “backed” by file • May map a specific region or let the system find one for you – Tricky to know where the holes are • Used both for manipulating files and for sharing between File processes scheduler mmap() file to region of VAS 4/11/19 Kubiatowicz CS162 © UCB Spring 2019 Lec 19.39 4/11/19 Kubiatowicz CS162 © UCB Spring 2019 Lec 19.40 An mmap() Example Sharing through Mapped Files VAS 1 VAS 2 #include
/* map the file */ mfile = mmap(0, 10000, PROT_READ|PROT_WRITE, MAP_FILE|MAP_SHARED, myfd, 0); stack stack if (mfile == MAP_FAILED) {perror("mmap$ cat failed"); test exit(1);} This is line one printf("mmap at : %16lx\n", (long unsignedThiLet's int) write mfile); over its line three puts(mfile); This is line four OS OS strcpy(mfile+20,"Let's write over it"); 0xFFF… 0xFFF… close(myfd); return 0; • Also: anonymous memory between parents and children } – no file backing – just swap space 4/11/19 Kubiatowicz CS162 © UCB Spring 2019 Lec 19.41 4/11/19 Kubiatowicz CS162 © UCB Spring 2019 Lec 19.42
File System Caching File System Caching (con’t) • Key Idea: Exploit locality by caching data in memory • Cache Size: How much memory should the OS allocate to the – Name translations: Mapping from pathsinodes buffer cache vs virtual memory? – Disk blocks: Mapping from block addressdisk content – Too much memory to the file system cache won’t be able to • Buffer Cache: Memory used to cache kernel resources, run many applications at once including disk blocks and name translations – Too little memory to file system cache many applications may – Can contain “dirty” blocks (blocks yet on disk) run slowly (disk caching not effective) • Replacement policy? LRU – Solution: adjust boundary dynamically so that the disk access – Can afford overhead of timestamps for each disk block rates for paging and file access are balanced – Advantages: • Read Ahead Prefetching: fetch sequential blocks early » Works very well for name translation – Key Idea: exploit fact that most common file access is » Works well in general as long as memory is big enough to sequential by prefetching subsequent disk blocks ahead of accommodate a host’s working set of files. current read request (if they are not already in memory) – Disadvantages: – Elevator algorithm can efficiently interleave groups of prefetches » Fails when some application scans through file system, thereby from concurrent applications flushing the cache with data used only once »Example: find . –exec grep foo {} \; – How much to prefetch? • Other Replacement Policies? » Too many imposes delays on requests by other applications – Some systems allow applications to request other policies » Too few causes many seeks (and rotational delays) among – Example, ‘Use Once’: concurrent file requests » File system can discard blocks as soon as they are used 4/11/19 Kubiatowicz CS162 © UCB Spring 2019 Lec 19.43 4/11/19 Kubiatowicz CS162 © UCB Spring 2019 Lec 19.44 File System Caching (con’t) Important “ilities” • Delayed Writes: Writes to files not immediately sent out to • Availability: the probability that the system can accept and disk process requests – Instead, write() copies data from user space buffer to kernel – Often measured in “nines” of probability. So, a 99.9% buffer (in cache) probability is considered “3-nines of availability” » Enabled by presence of buffer cache: can leave written file blocks – Key idea here is independence of failures in cache for a while • Durability: the ability of a system to recover data despite » If some other application tries to read data before written to disk, faults file system will read from cache – This idea is fault tolerance applied to data – Flushed to disk periodically (e.g. in UNIX, every 30 sec) – Doesn’t necessarily imply availability: information on pyramids – Advantages: was very durable, but could not be accessed until discovery of » Disk scheduler can efficiently order lots of requests Rosetta Stone » Disk allocation algorithm can be run with correct size value for a • Reliability: the ability of a system or component to perform its file required functions under stated conditions for a specified » Some files need never get written to disk! (e..g temporary scratch files written /tmp often don’t exist for 30 sec) period of time (IEEE definition) – Usually stronger than simply availability: means that the – Disadvantages system is not only “up”, but also working correctly » What if system crashes before file has been written out? – Includes availability, security, fault tolerance/durability » Worse yet, what if system crashes before a directory file has been written out? (lose pointer to inode!) – Must make sure data survives system crashes, disk crashes, other problems 4/11/19 Kubiatowicz CS162 © UCB Spring 2019 Lec 19.45 4/11/19 Kubiatowicz CS162 © UCB Spring 2019 Lec 19.46
File System Summary (1/2) File System Summary (2/2) • File System: • 4.2 BSD Multilevel index files – Transforms blocks into Files and Directories – Inode contains ptrs to actual blocks, indirect blocks, double – Optimize for size, access and usage patterns indirect blocks, etc. – Maximize sequential access, allow efficient random access – Optimizations for sequential access: start new files in open – Projects the OS protection and security regime (UGO vs ACL) ranges of free blocks, rotational optimization • File defined by header, called “inode” • File layout driven by freespace management • Naming: translating from user-visible names to actual sys – Integrate freespace, inode table, file blocks and dirs into resources block group – Directories used for naming for local file systems • Deep interactions between mem management, file system, – Linked or tree structure stored in files sharing • Multilevel Indexed Scheme – mmap(): map file or anonymous segment to memory – inode contains file info, direct pointers to blocks, indirect • Buffer Cache: Memory used to cache kernel resources, blocks, doubly indirect, etc.. including disk blocks and name translations – NTFS: variable extents not fixed blocks, tiny files data is in – Can contain “dirty” blocks (blocks yet on disk) header 4/11/19 Kubiatowicz CS162 © UCB Spring 2019 Lec 19.47 4/11/19 Kubiatowicz CS162 © UCB Spring 2019 Lec 19.48