File System Issues Role of Files • What is the role of files? • Persistance - long-lived - data for What is the file abstraction? posterity • File naming. How to find the file we want? à non-volitile storage media Sharing files. Controlling access to files. à semantically meaningful (memorable) • Performance issues - how to deal with the names bottleneck of disks? What is the “right” way to optimize file access? Abstractions *File Abstractions User view Addressbook, record for Duke CPS • UNIX-like files Application – Sequence of bytes – Operations: open (create), close, read, write, fid addrfile ->fid, byte range* seek bytes File System • Memory mapped files block# device, block # – Sequence of bytes Disk Subsystem – Mapped into address space surface, cylinder, sector – Page fault mechanism does data transfer • Named, Possibly typed 1 Functions of File System Functions of Device Subsystem • (Directory subsystem) Map filenames to fileids- In general, deal with device characteristics open (create) syscall. Create kernel data structures. • Translate block numbers (the abstraction of Maintain naming structure (unlink, mkdir , rmdir) device shown to file system) to physical disk addresses. • Determine layout of files and metadata on disk in terms of blocks. Disk block allocation. Bad Device specific (subject to change with upgrades blocks. in technology) intelligent placement of blocks. • Handle read and write system calls • Schedule (reorder?) disk operations • Initiate I/O operations for movement of blocks to/from disk. • Maintain buffer cache Know your Workload! Unix File Syscalls User grp others • File usage patterns should influence design int fd, num, success, bufsize; rwx rwx rwx decisions. Do things differently depending: char data[bufsize]; long offset, pos; 111 100 000 – How large are most files? How long-lived? fd = open (filename, mode [,permissions]); Read vs. write activity. Shared often? success = close (fd); O_RDONLY – Different levels “see” a different workload. O_WRONLY pos = lseek (fd, offset, mode); O_RDWR • Feedback loop O_CREAT num = read (fd, data, bufsize); O_APPEND num = write (fd, data, bufsize); ... Usage patterns File System Relative to design and impl observed today beginning, current position, end of file 2 File System Data Structures UNIX Inodes System-wide System-wide Data Block Addr Open file table File descriptor table File Data blocks Process 3 ... 3 3 ... 3 descriptor Attributes in-memory r-w pos, mode copy of inode File data Addr ... ptr to on-disk inode r-w pos, mode Block 1 ... stdin 2 ... array stdout ... 2 stderr ... process ptr - Decoupling meta-data 1 pos 2 per file from directory entries 2 pos 1 Pathname Resolution cps110 File Sharing Between Parent/Child Directory node “cps110/current/Proj/proj3” File current main(int argc, char *argv[]) { current inode# Attributes Directory node File char c; Attributes int fdrd , fdwt, fdpriv; Proj inode# if (( fdrd = open(argv[1], O_RDONLY)) == -1) exit(1); if (( fdwt = creat([argv[2], 0666)) == -1) File exit(1); Attributes File fork(); Attributes if (( fdpriv = open([ argv[3], O_RDONLY)) == -1) exit(1); Proj proj3 while (TRUE) { data file Directory node if (read(fdrd , &c, 1) != 1) exit(0); proj3 inode# write(fdwt, &c, 1); } index node of wd } 3 Sharing Open File Instances File System Data Structures System-wide System-wide user ID Open file table File descriptor table process ID parent process group ID shared seek parent PID offset in shared Process signal state siblings file table entry descriptor children in-memory r-w pos, mode copy of inode ptr to on-disk inode user ID shared file r-w pos, mode process ID (inodeorvnode) process group ID child parent PID stdin signal state siblings children array stdout process file system open stderr process process ptr descriptors file table - forked process’s objects per file openafterfork Process descriptor Memory Mapped Files Nachos File fd = open (somefile, consistent_mode); Syscalls/Operations pa = mmap(addr, len, prot, flags, fd, Create(“zot”); offset); R, W, X, OpenFileId fd; none fd + offset fd = Open(“zot”); Close(fd); pa Shared, len Private, Fixed, char data[bufsize]; Limitations: len Noreserve Write(data, count, fd); 1. small, fixed-size files and directories Read(data, count, fd); 2. single disk with a single directory 3. stream files only: no seek syscall 4. file size is specified at creation time VAS Reading performed by Load instr. 5. no access control, etc. 4 Goals of File Naming Naming Structures • Foremost function - to find files (e.g., in open() ), • Flat name space - 1 system-wide table, Map file name to file object. – Unique naming with multiple users is hard. • To store meta-data about files. Name conflicts. • To allow users to choose their own file names – Easy sharing, need for protection without undue name conflict problems. • Per-user name space • To allow sharing. – Protection by isolation, no sharing • Convenience: short names, groupings. – Easy to avoid name conflicts • To avoid implementation complications – Register identifies with directory to use to resolve names, possibility of user-settable (cd) Naming Structures Full Naming Network* Terry A Naming network • /Jamie/joe/project/D • Component names - pathnames grp1 root • /Jamie/d – Absolute pathnames - from a designated root Jo • /Jamie/joe/jam/proj1 – Relative pathnames - from a working directory TA /C – Each name carries how to resolve it. joe Jamie • (relative from Terry) • Short names to files anywhere in the network project jam A produce cycles, but convenience in naming B d proj1 • (relative from Jamie) things. D E C d D project * not Unix 5 Full Naming Network* Meta-Data Terry A • /Jamie/joe/project/D • File size • Location of file - grp1 root • /Jamie/d • File type which device Jo • /Jamie/joe/jam/proj1 • Protection - access • Location of TA /C control information individual blocks of joe Jamie the file on disk. project • (relative from Terry) • History: jam A creation time, • Owner of file B d last modification, • Group(s) of users proj1 • (relative from Jamie) D E last access. associated with file C d D project Why? * Unix Operations on Directories Restricting to a Hierarchy (UNIX) • Problems with full naming network • link (oldpathname, newpathname) - – What does it mean to “delete” a file? make entry pointing to file – Meta-data interpretation • unlink (filename) - remove entry pointing to file • Eliminating cycles • mknod (dirname, type, device) - used – allows use of reference counts for (e.g. by mkdir utility function) to create a reclaiming file space directory (or named pipe, or special file) – avoids garbage collection • getdents(fd, buf, structsize) - reads dir entries 6 A Typical Unix File Tree Garbage Collection Each volume is a set of directories and files; a host’s file tree Terry is the set of directories and files visible to processes on A a given host. File trees are built by grafting / grp1 X root volumes from different devices or from network servers. Jo bin etc tmp usr vmunix X TA In Unix, the graft operation is joe XJamie Series of the privileged mount system call, ls sh project users project unlinks and each volume is afilesystem. packages jam mount point B d proj1 coveredDir D E mount (coveredDir, volume) C coveredDir: directory pathname D project volume: device volume root contents become visible at pathname coveredDir A Typical Unix File Tree Each volume is a set of directories and files; a host’s file tree Reclaiming Convenience is the set of directories and files visible to processes on a given host. • Symbolic links - indirect files File trees are built by grafting / volumes from different devices filename maps, not to file object, but to or from network servers. bin etc tmp usr vmunix another pathname In Unix, the graft operation is – allows short aliases the privileged mount system call, ls sh project users and each volume is afilesystem. – slightly different semantics packages mount point • Search path rules (volumecoveredDir root) mount (coveredDir, volume) tex emacs /usr/project/packages/coveredDir/emacs 7 Unix File Naming (Hard Links) Unix Symbolic (Soft) Links directory A directory B A Unix file may have multiple names. 0 wind: 18 • Unix files may also be named by symbolic (soft) links. rain: 32 0 – A soft link is a file containing a pathname of some other file. hail: 48 sleet: 48 Each directory entry naming the file is called a hard link. symlink system call symlink (existing name, new name) inodelink directory A directory B Each inode contains a reference count allocate a new file (inode) with type symlink count = 2 0 wind: 18 showing how many hard links name it. initialize file contents with existing name 0 inode48 rain: 32 create directory entry for new file with new name hail: 48 sleet: 67 linksystem call unlinksystem call (“remove”) The target of the link may be link (existing name, new name) unlink(name) removed at any time, leaving create a new name for an existing file destroy directory entry inodelink ../A/hail/0 a dangling reference. increment inode link count decrement inode link count count = 1 if count = 0 and file is not in active use inode48 inode67 How should the kernel free blocks (recursively) and on -disk inode handle recursive soft links? Convenience, but not performance! Access Control for Files UNIX access control • Access control lists - detailed list • Each file carries its access control with it. attached to file of users allowed rwx rwx rwx setuid (denied) access, including kind of access allowed/denied. Owner Group Everybody else When bit set, it