Linux Metadata Linux Metadata Where is Metadata Stored? struct stat { Metadata in the File dev_t st_dev; /* device */ Metadata in the Directory ino_t st_ino; /* inode */ Crash Recovery mode_t st_mode; /* protection */ The Unix Filesystem nlink_t st_nlink; /* number of hard links File Operations uid_t st_uid; /* user ID of owner */ File System Layout The Windows FAT gid_t st_gid; /* group ID of owner */ File System dev_t st_rdev; /* device type (if inode Dump/Restore off_t st_size; /* total size, in bytes blksize_t st_blksize; /* blocksize for filesystem blkcnt_t st_blocks; /* number of blocks allocated time_t st_atime; /* time of last access time_t st_mtime; /* time of last modification time_t st_ctime; /* time of last status }; 1 / 42 Where is Metadata Stored? Linux Metadata ■ Where is Metadata In the file? Stored? Metadata in the File ■ In the directory entry? Metadata in the Directory ■ Elsewhere? Crash Recovery ■ The Unix Filesystem Split? File Operations File System Layout The Windows FAT File System Dump/Restore 2 / 42 Metadata in the File Linux Metadata Where is Metadata ■ (Sort of) done by Apple: resource and data Stored? Metadata in the File forks Metadata in the Directory ■ Not very portable — when you copy the file Crash Recovery The Unix Filesystem to/from other systems, what happens to the File Operations metadata? File System Layout ■ No standardized metadata exchange format The Windows FAT File System Dump/Restore 3 / 42 Metadata in the Directory Linux Metadata ■ Where is Metadata Speeds access to metadata Stored? Metadata in the File ■ Makes hard links difficult — need to keep Metadata in the Directory copies of the metadata synchronized Crash Recovery ■ The Unix Filesystem Makes directories larger; often, one doesn’t File Operations need the metadata File System Layout ■ Many newer systems keep at least a few bits of The Windows FAT File System metadata in the directory, notably file type — Dump/Restore knowing if something is or isn’t a directory speeds up tree walks considerably 4 / 42 Crash Recovery Linux Metadata ■ Where is Metadata Must ensure that file systems are in a Stored? Metadata in the File consistent state after a system crash Metadata in the Directory ■ Example: don’t write out directory entry Crash Recovery Crash Recovery before the metadata Repairing Damage Log-Structured File ■ Example: File systems are generally trees, not Systems Modern Disks graphs; make sure things always point The Unix Filesystem somewhere sane File Operations ■ File System Layout What if the file has blocks but the freelist The Windows FAT File System hasn’t been updated? Dump/Restore ■ Principle: order writes to ensure that the disk is always in a safe state 5 / 42 Repairing Damage Linux Metadata ■ Where is Metadata At boot-time, run a consistency checker except Stored? Metadata in the File after a clean shutdown Metadata in the Directory ■ Example: fsck (Unix) and scandisk Crash Recovery Crash Recovery (Windows) Repairing Damage Log-Structured File ■ Force things to a safe state; move any Systems Modern Disks allocated but unreferenced blocks and files to a The Unix Filesystem recovery area File Operations File System Layout The Windows FAT File System Dump/Restore 6 / 42 Log-Structured File Systems Linux Metadata ■ Where is Metadata Instead of overwriting data, append the Stored? Metadata in the File changes to a journaling area Metadata in the Directory ■ The file system is thus always consistent, as Crash Recovery Crash Recovery long as writes are properly ordered. Repairing Damage Log-Structured File ■ Hmm — is that a reasonable assumption? Systems Modern Disks The Unix Filesystem File Operations File System Layout The Windows FAT File System Dump/Restore 7 / 42 Modern Disks Linux Metadata ■ Where is Metadata Modern disks do a lot of buffering Stored? Metadata in the File ■ Cache size on new Seagate drives: 2-16M bytes Metadata in the Directory ■ The drive will reorder writes to optimize seek Crash Recovery Crash Recovery times Repairing Damage Log-Structured File ■ If a bad block has been relocated, you can’t Systems Modern Disks even predict when seeks will occur; only the The Unix Filesystem drive knows File Operations ■ File System Layout What if the power fails when data is buffered? The Windows FAT File System Dump/Restore 8 / 42 Linux Metadata Where is Metadata Stored? Metadata in the File Metadata in the Directory Crash Recovery The Unix Filesystem The Unix Filesystem From the Process Directories The Unix Filesystem Finding a File . and .. I-Nodes What’s in an I-Node? Disk Blocks in the I-Node Multiple Layers of Indirection File Operations File System Layout The Windows FAT File System Dump/Restore 9 / 42 The Unix Filesystem Linux Metadata ■ Where is Metadata Let’s take a high-level look at the Unix file Stored? Metadata in the File system Metadata in the Directory ■ Though details differ (a lot) for, say, Windows, Crash Recovery The Unix Filesystem at a high level things are pretty similar The Unix Filesystem ■ From the Process We’ll discuss the actual code paths next time Directories ■ Finding a File Note: all modern operating systems support . and .. I-Nodes multiple file system types; differences hidden What’s in an I-Node? by abstraction layer. Disk Blocks in the I-Node Multiple Layers of Indirection File Operations File System Layout The Windows FAT File System Dump/Restore 10 / 42 From the Process Linux Metadata ■ Where is Metadata The process has two crucial directory Stored? Metadata in the File attributes, the current root and the current Metadata in the Directory working directory Crash Recovery ■ The Unix Filesystem Paths that start with a / (known as absolute The Unix Filesystem From the Process paths) start from the current root; those that Directories Finding a File do not start with a / (relative paths) start . and .. I-Nodes from the current working directory What’s in an ■ I-Node? (Roots other than the real root are a security Disk Blocks in the I-Node Multiple Layers of mechanism; we’ll discuss that later in the Indirection semester.) File Operations File System Layout The Windows FAT File System Dump/Restore 11 / 42 Directories Linux Metadata ■ Where is Metadata On some Unix systems, directories can be read Stored? Metadata in the File like any other file with read(); on Linux, you Metadata in the Directory must (and on other Unix systems you should) Crash Recovery The Unix Filesystem use readdir() The Unix Filesystem ■ From the Process A directory entry consists of a variable-length Directories Finding a File name and an i-node number . and .. ■ I-Nodes (What’s an i-node?) What’s in an ■ I-Node? By convention, on most Unix systems the first Disk Blocks in the I-Node . .. Multiple Layers of two entries in a directory are and — the Indirection current and parent directories File Operations ■ File System Layout Don’t count on them being there; they’re not The Windows FAT File System guaranteed by the spec! Dump/Restore 12 / 42 Finding a File Linux Metadata ■ Where is Metadata Find the next component in the pathname Stored? Metadata in the File ■ Read the current directory looking for it Metadata in the Directory ■ If there’s another component to the path name Crash Recovery The Unix Filesystem and this element is a directory, repeat, starting The Unix Filesystem From the Process from this element Directories ■ Finding a File When we’re done, the result is an i-node . and .. I-Nodes number What’s in an I-Node? Disk Blocks in the I-Node Multiple Layers of Indirection File Operations File System Layout The Windows FAT File System Dump/Restore 13 / 42 . and .. Linux Metadata ■ Where is Metadata Note how . and .. work Stored? Metadata in the File ■ . has the i-node number of the current Metadata in the Directory directory — you start again from this node for Crash Recovery The Unix Filesystem the next component The Unix Filesystem ■ From the Process It’s just another directory; to (this part of) the Directories Finding a File kernel, there’s nothing special about it . and .. ■ I-Nodes The same is true for .. — it “happens” to What’s in an I-Node? point up a level in the directory tree. Disk Blocks in the I-Node ■ Multiple Layers of Following a search path does not rely on the Indirection directory structure being a tree! That’s File Operations File System Layout primarily needed for orderly tree walks. The Windows FAT ■ File System (Mental exercise: symbolic links do introduce Dump/Restore the possiblity of loops. How can this be dealt with?) 14 / 42 I-Nodes Linux Metadata ■ Where is Metadata What’s an i-node? Stored? Metadata in the File ■ An i-node holds most of the metadata for a Metadata in the Directory Unix file — on classic Unix, it holds all but the Crash Recovery The Unix Filesystem i-node number itself The Unix Filesystem ■ From the Process The i-list is a disk-resident array i-nodes Directories ■ Finding a File The i-node number is just an index into this . and .. I-Nodes array What’s in an ■ I-Node? Looked at another way, a directory entry maps Disk Blocks in the I-Node Multiple Layers of a name to an array entry Indirection ■ Files are actually described by i-nodes, not by File Operations File System Layout names The Windows FAT File System Dump/Restore 15 / 42 What’s in an I-Node? Linux Metadata ■ Where is Metadata All of the fields from the stat structure Stored? Metadata in the File ■ A few other pieces of user-settable metadata Metadata in the Directory (flags) Crash Recovery ■ The Unix Filesystem Disk block information — where on disk the The Unix Filesystem From the Process file resides? Directories ■ Finding a File How many blocks is enough? .