Carnegie Mellon Carnegie Mellon Today ¢ Unix I/O System-Level I/O ¢ Metadata, sharing, and redirec?on ¢ Standard I/O 15-213 / 18-213: Introduc2on to Computer Systems ¢ RIO (roBust I/O) package th 15 Lecture, Mar. 8, 2012 ¢ Closing remarks Instructors: Todd C. Mowry & Anthony Rowe 1 2 Carnegie Mellon Carnegie Mellon Unix Files Unix File Types ¢ A Unix file is a sequence of m bytes: ¢ Regular file § File containing user/app data (binary, text, whatever) § B0 , B1 , .... , Bk , .... , Bm-1 § OS does not know anything about the format § other than “sequence of bytes”, akin to main memory ¢ All I/O devices are represented as files: ¢ Directory file § /dev/sda2 (/usr disk par22on) § A file that contains the names and locaons of other files § /dev/tty2 (terminal) ¢ Character special and Block special files § Terminals (character special) and disks (block special) ¢ Even the kernel is represented as a file: ¢ FIFO (named pipe) § /dev/kmem (kernel memory image) § A file type used for inter-process communicaon § /proc (kernel data structures) ¢ Socket § A file type used for network communicaon between processes 3 4 1 Carnegie Mellon Carnegie Mellon Unix I/O Opening Files ¢ Key Features ¢ § Elegant mapping of files to devices allows kernel to export simple Opening a file informs the kernel that you are geUng ready to interface called Unix I/O access that file § Important idea: All input and output is handled in a consistent and int fd; /* file descriptor */ uniform way if ((fd = open("/etc/hosts", O_RDONLY)) < 0) { perror("open"); ¢ Basic Unix I/O operaons (system calls): exit(1); § Opening and closing files } § open()and close() § Reading and wri2ng a file ¢ Returns a small iden?fying integer file descriptor § read() and write() § fd == -1 indicates that an error occurred § Changing the current file posi/on (seek) ¢ Each process created by a Unix shell begins life with three open § indicates next offset into file to read or write files associated with a terminal: § lseek() § 0: standard input B0 B1 • • • Bk-1 Bk Bk+1 • • • § 1: standard output § 2: standard error Current file posi?on = k 5 6 Carnegie Mellon Carnegie Mellon Closing Files Reading Files ¢ Reading a file copies Bytes from the current file posi?on to ¢ Closing a file informs the kernel that you are finished memory, and then updates file posion accessing that file char buf[512]; int fd; /* file descriptor */ int fd; /* file descriptor */ int retval; /* return value */ int nbytes; /* number of bytes read */ if ((retval = close(fd)) < 0) { /* Open file fd ... */ perror("close"); /* Then read up to 512 bytes from file fd */ exit(1); if ((nbytes = read(fd, buf, sizeof(buf))) < 0) { } perror("read"); exit(1); } ¢ Closing an already closed file is a recipe for disaster in threaded programs (more on this later) ¢ Returns numBer of Bytes read from file fd into buf ¢ Moral: Always check return codes, even for seemingly § Return type ssize_t is signed integer benign func?ons such as close() § nbytes < 0 indicates that an error occurred § Short counts (nbytes < sizeof(buf) ) are possible and are not errors! 7 8 2 Carnegie Mellon Carnegie Mellon Wri?ng Files Simple Unix I/O example ¢ Wring a file copies bytes from memory to the current file ¢ Copying standard in to standard out, one byte at a me posion, and then updates current file posion char buf[512]; int main(void) int fd; /* file descriptor */ { int nbytes; /* number of bytes read */ char c; int len; /* Open the file fd ... */ /* Then write up to 512 bytes from buf to file fd */ while ((len = read(0 /*stdin*/, &c, 1)) == 1) { if ((nbytes = write(fd, buf, sizeof(buf)) < 0) { if (write(1 /*stdout*/, &c, 1) != 1) { perror("write"); exit(20); exit(1); } } } if (len < 0) { printf (“read from stdin failed”); exit (10); ¢ Returns numBer of Bytes wriXen from buf to file fd } § nbytes < 0 indicates that an error occurred exit(0); § As with reads, short counts are possible and are not errors! } 9 10 Carnegie Mellon Carnegie Mellon On Short Counts Today ¢ Short counts can occur in these situaons: ¢ Unix I/O § Encountering (end-of-file) EOF on reads ¢ Metadata, sharing, and redirec?on § Reading text lines from a terminal ¢ Standard I/O § Reading and wri2ng network sockets or Unix pipes ¢ RIO (roBust I/O) package ¢ Closing remarks ¢ Short counts never occur in these situaons: § Reading from disk files (except for EOF) § Wri2ng to disk files 11 12 3 Carnegie Mellon Carnegie Mellon Example of Accessing File Metadata File Metadata /* statcheck.c - Querying and manipulating a file’s meta data */ ¢ Metadata is data about data, in this case file data #include "csapp.h" unix> ./statcheck statcheck.c ¢ Per-file metadata maintained By kernel int main (int argc, char **argv) type: regular, read: yes § accessed by users with the stat and fstat funcons { unix> chmod 000 statcheck.c struct stat stat; unix> ./statcheck statcheck.c char *type, *readok; type: regular, read: no /* Metadata returned by the stat and fstat functions */ unix> ./statcheck .. struct stat { Stat(argv[1], &stat); type: directory, read: yes dev_t st_dev; /* device */ if (S_ISREG(stat.st_mode)) unix> ./statcheck /dev/kmem ino_t st_ino; /* inode */ type = "regular"; type: other, read: yes mode_t st_mode; /* protection and file type */ else if (S_ISDIR(stat.st_mode)) nlink_t st_nlink; /* number of hard links */ type = "directory"; uid_t st_uid; /* user ID of owner */ else gid_t st_gid; /* group ID of owner */ type = "other"; dev_t st_rdev; /* device type (if inode device) */ off_t st_size; /* total size, in bytes */ if ((stat.st_mode & S_IRUSR)) /* OK to read?*/ unsigned long st_blksize; /* blocksize for filesystem I/O */ readok = "yes"; else unsigned long st_blocks; /* number of blocks allocated */ readok = "no"; time_t st_atime; /* time of last access */ time_t st_mtime; /* time of last modification */ time_t st_ctime; /* time of last change */ printf("type: %s, read: %s\n", type, readok); exit(0); }; } statcheck.c 13 14 Carnegie Mellon Carnegie Mellon Repeated Slide: Opening Files How the Unix Kernel Represents Open Files ¢ Opening a file informs the kernel that you are geUng ready to ¢ Two descriptors referencing two dis?nct open disk files. access that file Descriptor 1 (stdout) points to terminal, and descriptor 4 int fd; /* file descriptor */ points to open disk file if ((fd = open("/etc/hosts", O_RDONLY)) < 0) { Descriptor taBle Open file taBle v-node table perror("open"); exit(1); [one table per process] [shared by all processes] [shared by all processes] } File A (terminal) stdin fd 0 File access stdout fd 1 ¢ Returns a small iden?fying integer file descriptor File pos File size Info in stderr fd 2 stat refcnt=1 File type § fd 3 ... fd == -1 indicates that an error occurred ... struct fd 4 File B (disk) File access File size File pos File type refcnt=1 ... ... 15 16 4 Carnegie Mellon Carnegie Mellon File Sharing How Processes Share Files: Fork() ¢ Two dis?nct descriptors sharing the same disk file through ¢ A child process inherits its parent’s open files two disnct open file table entries § Note: situaon unchanged by exec funcons (use fcntl to change) § E.g., Calling open twice with the same filename argument ¢ Before fork() call: Descriptor taBle Open file taBle v-node table Descriptor taBle Open file taBle v-node table [one table per process] [shared by all processes] [shared by all processes] [one table per process] [shared by all processes] [shared by all processes] File A (disk) File A (terminal) stdin fd 0 File access stdin fd 0 File access stdout fd 1 stdout fd 1 File pos File size File pos File size stderr fd 2 stderr fd 2 refcnt=1 File type refcnt=1 File type fd 3 ... fd 3 ... ... ... fd 4 fd 4 File B (disk) File B (disk) File access File size File pos File pos File type refcnt=1 refcnt=1 ... ... ... 17 18 Carnegie Mellon Carnegie Mellon How Processes Share Files: Fork() I/O Redirec?on ¢ A child process inherits its parent’s open files ¢ Ques?on: How does a shell implement I/O redirec?on? ¢ Aer fork(): unix> ls > foo.txt § Child’s table same as parent’s, and +1 to each refcnt Descriptor taBle Open file taBle v-node table ¢ Answer: By calling the dup2(oldfd, newfd) funcon [one table per process] [shared by all processes] [shared by all processes] § Copies (per-process) descriptor table entry oldfd to entry newfd Parent File A (terminal) fd 0 File access fd 1 Descriptor taBle Descriptor taBle File pos File size fd 2 before dup2(4,1) aer dup2(4,1) refcnt=2 File type fd 3 ... ... fd 4 fd 0 fd 0 fd 1 fd 1 Child File B (disk) a b File access fd 0 fd 2 fd 2 File size fd 1 File pos fd 3 fd 3 fd 2 File type refcnt=2 ... fd 3 ... fd 4 b fd 4 b fd 4 19 20 5 Carnegie Mellon Carnegie Mellon I/O Redirec?on Example I/O Redirec?on Example (cont.) ¢ Step #1: open file to which stdout should be redirected ¢ Step #2: call dup2(4,1) § Happens in child execu2ng shell code, before exec § cause fd=1 (stdout) to refer to disk file pointed at by fd=4 Descriptor taBle Open file taBle v-node table Descriptor taBle Open file taBle v-node table [one table per process] [shared by all processes] [shared by all processes] [one table per process] [shared by all processes] [shared by all processes] File A File A stdin fd 0 File access stdin fd 0 File access stdout fd 1 stdout fd 1 File pos File size File pos File size stderr fd 2 stderr fd 2 refcnt=1 File type refcnt=0 File type fd 3 ... fd 3 ... ... ... fd 4 fd 4 File B File B File access File access File size File size File pos File pos File type File type refcnt=1 ..