DOCSLIB.ORG

The Design and Implementation of the 4.4BSD Operating System

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
The Design and Implementation of the 4.4BSD Operating System The Design and Implementation of the 4.4BSD Operating System Marshall Kirk McKusick Keith Bostic Michael J. Karels John S. Quarterman The Design and Implementation of the 4.4BSD Operating System by Marshall Kirk McKusick, Keith Bostic, Michael J. Karels, and John S. Quarterman Revision: 76bf5f73 Copyright © 1996 Addison-Wesley Longman, Inc The second chapter of the book, The Design and Implementation of the 4.4BSD Operating System is excerpted here with the permission of the publisher. No part of it may be further reproduced or distributed without the publisher's express written permission. The rest of the book explores the concepts introduced in this chapter in incredible detail and is an excellent reference for anyone with an interest in BSD UNIX. More information about this book is available from the publisher, with whom you can also sign up to receive news of related titles. Information about BSD courses is available from Kirk McKusick. ii Table of Contents 2. Design Overview of 4.4BSD ......................................................................................................... 1 2.1. 4.4BSD Facilities and the Kernel ........................................................................................ 1 2.2. Kernel Organization ....................................................................................................... 2 2.3. Kernel Services ............................................................................................................. 4 2.4. Process Management ...................................................................................................... 4 2.5. Memory Management ..................................................................................................... 6 2.6. I/O System ................................................................................................................... 8 2.7. Filesystems ................................................................................................................. 11 2.8. Filestores .................................................................................................................... 14 2.9. Network Filesystem ...................................................................................................... 15 2.10. Terminals .................................................................................................................. 15 2.11. Interprocess Communication ......................................................................................... 16 2.12. Network Communication .............................................................................................. 16 2.13. Network Implementation .............................................................................................. 17 2.14. System Operation ....................................................................................................... 17 List of Figures 2.1. Process lifecycle .................................................................................................................... 4 2.2. A small filesystem ................................................................................................................ 12 List of Tables 2.1. Machine-independent software in the 4.4BSD kernel ..................................................................... 2 2.2. Machine-dependent software for the HP300 in the 4.4BSD kernel ..................................................... 3 Chapter 2. Design Overview of 4.4BSD 2.1. 4.4BSD Facilities and the Kernel The 4.4BSD kernel provides four basic facilities: processes, a filesystem, communications, and system startup. This section outlines where each of these four basic services is described in this book. 1. Processes constitute a thread of control in an address space. Mechanisms for creating, terminating, and other- wise controlling processes are described in Chapter 4. The system multiplexes separate virtual-address spaces for each process; this memory management is discussed in Chapter 5. 2. The user interface to the filesystem and devices is similar; common aspects are discussed in Chapter 6. The filesystem is a set of named les, organized in a tree-structured hierarchy of directories, and of operations to manipulate them, as presented in Chapter 7. Files reside on physical media such as disks. 4.4BSD supports several organizations of data on the disk, as set forth in Chapter 8. Access to les on remote machines is the subject of Chapter 9. Terminals are used to access the system; their operation is the subject of Chapter 10. 3. Communication mechanisms provided by traditional UNIX systems include simplex reliable byte streams be- tween related processes (see pipes, Section 11.1), and notification of exceptional events (see signals, Section 4.7). 4.4BSD also has a general interprocess-communication facility. This facility, described in Chapter 11, uses access mechanisms distinct from those of the filesystem, but, once a connection is set up, a process can access it as though it were a pipe. There is a general networking framework, discussed in Chapter 12, that is normally used as a layer underlying the IPC facility. Chapter 13 describes a particular networking implementation in detail. 4. Any real operating system has operational issues, such as how to start it running. Startup and operational issues are described in Chapter 14. Sections 2.3 through 2.14 present introductory material related to Chapters 3 through 14. We shall define terms, mention basic system calls, and explore historical developments. Finally, we shall give the reasons for many major design decisions. 2.1.1. The Kernel The kernel is the part of the system that runs in protected mode and mediates access by all user programs to the underlying hardware (e.g., CPU, disks, terminals, network links) and software constructs (e.g., filesystem, network protocols). The kernel provides the basic system facilities; it creates and manages processes, and provides functions to access the filesystem and communication facilities. These functions, called system calls appear to user processes as library subroutines. These system calls are the only interface that processes have to these facilities. Details of the system-call mechanism are given in Chapter 3, as are descriptions of several kernel mechanisms that do not execute as the direct result of a process doing a system call. A kernel in traditional operating-system terminology, is a small nucleus of software that provides only the minimal facilities necessary for implementing additional operating-system services. In contemporary research operating systems -- such as Chorus [11], Mach [1], Tunis [3], and the V Kernel [2] -- this division of functionality is more than just a logical one. Services such as filesystems and networking protocols are implemented as client application processes of the nucleus or kernel. The 4.4BSD kernel is not partitioned into multiple processes. This basic design decision was made in the earliest versions of UNIX. The rst two implementations by Ken Thompson had no memory mapping, and thus made no hardware-enforced distinction between user and kernel space [9]. A message-passing system could have been im- plemented as readily as the actually implemented model of kernel and user processes. The monolithic kernel was chosen for simplicity and performance. And the early kernels were small; the inclusion of facilities such as net- Kernel Organization working into the kernel has increased its size. The current trend in operating-systems research is to reduce the kernel size by placing such services in user space. Users ordinarily interact with the system through a command-language interpreter, called a shell, and perhaps through additional user application programs. Such programs and the shell are implemented with processes. De- tails of such programs are beyond the scope of this book, which instead concentrates almost exclusively on the kernel. Sections 2.3 and 2.4 describe the services provided by the 4.4BSD kernel, and give an overview of the latter's design. Later chapters describe the detailed design and implementation of these services as they appear in 4.4BSD. 2.2. Kernel Organization In this section, we view the organization of the 4.4BSD kernel in two ways: 1. As a static body of software, categorized by the functionality offered by the modules that make up the kernel 2. By its dynamic operation, categorized according to the services provided to users The largest part of the kernel implements the system services that applications access through system calls. In 4.4BSD, this software has been organized according to the following: • Basic kernel facilities: timer and system-clock handling, descriptor management, and process management • Memory-management support: paging and swapping • Generic system interfaces: the I/O, control, and multiplexing operations performed on descriptors • The filesystem: les, directories, pathname translation, le locking, and I/O buer management • Terminal-handling support: the terminal-interface driver and terminal line disciplines • Interprocess-communication facilities: sockets • Support for network communication: communication protocols and generic network facilities, such as routing Table 2.1. Machine-independent software in the 4.4BSD kernel Category Lines of code Percentage of kernel headers 9,393 4.6 initialization 1,107 0.6 kernel facilities 8,793 4.4 generic interfaces 4,782 2.4 interprocess communication 4,540 2.2 terminal handling 3,911 1.9 virtual
Load more

Recommended publications
  • Twenty Years of Berkeley Unix : from AT&T-Owned to Freely
    Twenty Years of Berkeley Unix : From AT&T-Owned to Freely Redistributable Marshall Kirk McKusick Early History Ken Thompson and Dennis Ritchie presented the first Unix paper at the Symposium on Operating Systems Principles at Purdue University in November 1973. Professor Bob Fabry, of the University of California at Berkeley, was in attendance and immediately became interested in obtaining a copy of the system to experiment with at Berkeley. At the time, Berkeley had only large mainframe computer systems doing batch processing, so the first order of business was to get a PDP-11/45 suitable for running with the then-current Version 4 of Unix. The Computer Science Department at Berkeley, together with the Mathematics Department and the Statistics Department, were able to jointly purchase a PDP-11/45. In January 1974, a Version 4 tape was delivered and Unix was installed by graduate student Keith Standiford. Although Ken Thompson at Purdue was not involved in the installation at Berkeley as he had been for most systems up to that time, his expertise was soon needed to determine the cause of several strange system crashes. Because Berkeley had only a 300-baud acoustic-coupled modem without auto answer capability, Thompson would call Standiford in the machine room and have him insert the phone into the modem; in this way Thompson was able to remotely debug crash dumps from New Jersey. Many of the crashes were caused by the disk controller's inability to reliably do overlapped seeks, contrary to the documentation. Berkeley's 11/45 was among the first systems that Thompson had encountered that had two disks on the same controller! Thompson's remote debugging was the first example of the cooperation that sprang up between Berkeley and Bell Labs.
    [Show full text]
  • Absolute BSD—The Ultimate Guide to Freebsd Table of Contents Absolute BSD—The Ultimate Guide to Freebsd
    Absolute BSD—The Ultimate Guide to FreeBSD Table of Contents Absolute BSD—The Ultimate Guide to FreeBSD............................................................................1 Dedication..........................................................................................................................................3 Foreword............................................................................................................................................4 Introduction........................................................................................................................................5 What Is FreeBSD?...................................................................................................................5 How Did FreeBSD Get Here?..................................................................................................5 The BSD License: BSD Goes Public.......................................................................................6 The Birth of Modern FreeBSD.................................................................................................6 FreeBSD Development............................................................................................................7 Committers.........................................................................................................................7 Contributors........................................................................................................................8 Users..................................................................................................................................8
    [Show full text]
  • The Release Engineering of Freebsd 4.4
    The Release Engineering of FreeBSD 4.4 Murray Stokely [email protected] Wind River Systems Abstract different pace, and with the general assumption that they This paper describes the approach used by the FreeBSD re- have first gone into FreeBSD-CURRENT and have been lease engineering team to make production-quality releases thoroughly tested by our user community. of the FreeBSD operating system. It details the methodol- In the interim period between releases, nightly snap- ogy used for the release of FreeBSD 4.4 and describes the shots are built automatically by the FreeBSD Project build tools available for those interested in producing customized machines and made available for download from ftp: FreeBSD releases for corporate rollouts or commercial pro- //stable.FreeBSD.org. The widespread availabil- ductization. ity of binary release snapshots, and the tendency of our user community to keep up with -STABLE development with CVSup and “make world”[8] helps to keep FreeBSD- 1 Introduction STABLE in a very reliable condition even before the qual- ity assurance activities ramp up pending a major release. The development of FreeBSD is a very open process. Bug reports and feature requests are continuously sub- FreeBSD is comprised of contributions from thousands of mitted by users throughout the release cycle. Problem people around the world. The FreeBSD Project provides reports are entered into our GNATS[9] database through anonymous CVS[1] access to the general public so that email, the send-pr(1) application, or via a web-based form. others can have access to log messages, diffs between de- In addition to the multitude of different technical mailing velopment branches, and other productivity enhancements lists about FreeBSD, the FreeBSD quality-assurance mail- that formal source code management provides.
    [Show full text]
  • [11] Case Study: Unix
    [11] CASE STUDY: UNIX 1 . 1 OUTLINE Introduction Design Principles Structural, Files, Directory Hierarchy Filesystem Files, Directories, Links, On-Disk Structures Mounting Filesystems, In-Memory Tables, Consistency IO Implementation, The Buffer Cache Processes Unix Process Dynamics, Start of Day, Scheduling and States The Shell Examples, Standard IO Summary 1 . 2 INTRODUCTION Introduction Design Principles Filesystem IO Processes The Shell Summary 2 . 1 HISTORY (I) First developed in 1969 at Bell Labs (Thompson & Ritchie) as reaction to bloated Multics. Originally written in PDP-7 asm, but then (1973) rewritten in the "new" high-level language C so it was easy to port, alter, read, etc. Unusual due to need for performance 6th edition ("V6") was widely available (1976), including source meaning people could write new tools and nice features of other OSes promptly rolled in V6 was mainly used by universities who could afford a minicomputer, but not necessarily all the software required. The first really portable OS as same source could be built for three different machines (with minor asm changes) Bell Labs continued with V8, V9 and V10 (1989), but never really widely available because V7 pushed to Unix Support Group (USG) within AT&T AT&T did System III first (1982), and in 1983 (after US government split Bells), System V. There was no System IV 2 . 2 HISTORY (II) By 1978, V7 available (for both the 16-bit PDP-11 and the new 32-bit VAX-11). Subsequently, two main families: AT&T "System V", currently SVR4, and Berkeley: "BSD", currently 4.4BSD Later standardisation efforts (e.g.
    [Show full text]
  • Utilizing Zfs for the Storage of Acquired Data*
    UTILIZING ZFS FOR THE STORAGE OF ACQUIRED DATA* C. Pugh, P. Henderson, K. Silber, T. Carroll, K. Ying Information Technology Division Princeton Plasma Physics Laboratory (PPPL) Princeton, NJ [email protected] Abstract— Every day, the amount of data that is acquired from command, no longer does a system administrator have to guess plasma experiments grows dramatically. It has become difficult how large a project will grow. No longer does a project have to for systems administrators to keep up with the growing demand be interrupted in order to grow a partition, or move data to a for hard drive storage space. In the past, project storage has larger partition. been supplied using UNIX filesystem (ufs) partitions. In order to increase the size of the disks using this system, users were II. FINDING A SOLUTION required to discontinue use of the disk, so the existing data could be transferred to a disk of larger capacity or begin use of a The administrators at Princeton Plasma Physics completely new and separate disk, thus creating a segmentation Laboratory (PPPL) were struggling with meeting the demands of data storage. of ever growing acquired project data. Under their current system of exporting a UNIX file system (ufs) partition via With the application of ZFS pools, the data capacity Network File System protocol (NFS), any project at the lab woes are over. ZFS provides simple administration that could request a partition for the storage of acquired project data eliminates the need to unmount to resize, or transfer data to a which could be automatically mounted on our internal “portal” larger disk.
    [Show full text]
  • Linux Metadata
    Linux Metadata Linux Metadata Where is Metadata Stored? struct stat { Metadata in the File dev_t st_dev; /* device */ Metadata in the Directory ino_t st_ino; /* inode */ Crash Recovery mode_t st_mode; /* protection */ The Unix Filesystem nlink_t st_nlink; /* number of hard links File Operations uid_t st_uid; /* user ID of owner */ File System Layout The Windows FAT gid_t st_gid; /* group ID of owner */ File System dev_t st_rdev; /* device type (if inode Dump/Restore off_t st_size; /* total size, in bytes blksize_t st_blksize; /* blocksize for filesystem blkcnt_t st_blocks; /* number of blocks allocated time_t st_atime; /* time of last access time_t st_mtime; /* time of last modification time_t st_ctime; /* time of last status }; 1 / 42 Where is Metadata Stored? Linux Metadata ■ Where is Metadata In the file? Stored? Metadata in the File ■ In the directory entry? Metadata in the Directory ■ Elsewhere? Crash Recovery ■ The Unix Filesystem Split? File Operations File System Layout The Windows FAT File System Dump/Restore 2 / 42 Metadata in the File Linux Metadata Where is Metadata ■ (Sort of) done by Apple: resource and data Stored? Metadata in the File forks Metadata in the Directory ■ Not very portable — when you copy the file Crash Recovery The Unix Filesystem to/from other systems, what happens to the File Operations metadata? File System Layout ■ No standardized metadata exchange format The Windows FAT File System Dump/Restore 3 / 42 Metadata in the Directory Linux Metadata ■ Where is Metadata Speeds access to metadata Stored? Metadata
    [Show full text]
  • Distributed File Service Messages and Codes
    z/OS Version 2 Release 3 Distributed File Service Messages and Codes IBM SC23-6885-30 Note Before using this information and the product it supports, read the information in “Notices” on page 295. This edition applies to Version 2 Release 3 of z/OS (5650-ZOS) and to all subsequent releases and modifications until otherwise indicated in new editions. Last updated: 2019-03-26 © Copyright International Business Machines Corporation 1996, 2018. US Government Users Restricted Rights – Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp. Contents About this document..............................................................................................v How to send your comments to IBM......................................................................vii Summary of changes...........................................................................................viii Chapter 1. Introduction......................................................................................... 1 Chapter 2. IOENnnnnnt: General DFS messages..................................................... 5 Chapter 3. IOEPnnnnnt: DFS kernel (dfskern) and general DFS error messages..... 27 Chapter 4. IOEWnnnnnt: SMB File/Print Server messages.................................... 59 Chapter 5. IOEXnnnnnt: File Exporter (dfsexport).................................................89 Chapter 6. IOEZnnnnnt: zFS messages............................................................... 101 Chapter 7. IOEZHnnnnt: zFS Health Checker messages.......................................213
    [Show full text]
  • Berkeley DB from Wikipedia, the Free Encyclopedia
    Berkeley DB From Wikipedia, the free encyclopedia Berkeley DB Original author(s) Margo Seltzer and Keith Bostic of Sleepycat Software Developer(s) Sleepycat Software, later Oracle Corporation Initial release 1994 Stable release 6.1 / July 10, 2014 Development status production Written in C Operating system Unix, Linux, Windows, AIX, Sun Solaris, SCO Unix, Mac OS Size ~1244 kB compiled on Windows x86 Type Embedded database License AGPLv3 Website www.oracle.com/us/products/database/berkeley-db /index.html (http://www.oracle.com/us/products/database/berkeley- db/index.html) Berkeley DB (BDB) is a software library that provides a high-performance embedded database for key/value data. Berkeley DB is written in C with API bindings for C++, C#, PHP, Java, Perl, Python, Ruby, Tcl, Smalltalk, and many other programming languages. BDB stores arbitrary key/data pairs as byte arrays, and supports multiple data items for a single key. Berkeley DB is not a relational database.[1] BDB can support thousands of simultaneous threads of control or concurrent processes manipulating databases as large as 256 terabytes,[2] on a wide variety of operating systems including most Unix- like and Windows systems, and real-time operating systems. "Berkeley DB" is also used as the common brand name for three distinct products: Oracle Berkeley DB, Berkeley DB Java Edition, and Berkeley DB XML. These three products all share a common ancestry and are currently under active development at Oracle Corporation. Contents 1 Origin 2 Architecture 3 Editions 4 Programs that use Berkeley DB 5 Licensing 5.1 Sleepycat License 6 References 7 External links Origin Berkeley DB originated at the University of California, Berkeley as part of BSD, Berkeley's version of the Unix operating system.
    [Show full text]
  • Copyright © 1992, by the Author(S). All Rights Reserved
    Copyright © 1992, by the author(s). All rights reserved. Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission. AN IMPLEMENTATION OF A LOG-STRUCTURED FILE SYSTEM FOR UNIX by Margo Seltzer, Keith Bostic, Marshall Kirk McKusick, and Carl Staelin Memorandum No. UCB/ERL M92/134 3 December 1992 AN IMPLEMENTATION OF A LOG-STRUCTURED FILE SYSTEM FOR UNIX by Margo Seltzer, Keith Bostic, Marshall Kirk McKusick, and Carl Staelin Memorandum No. UCB/ERL M92/134 3 December 1992 ELECTRONICS RESEARCH LABORATORY College of Engineering University of California, Berkeley 94720 AN IMPLEMENTATION OF A LOG-STRUCTURED FILE SYSTEM FOR UNIX by Margo Seltzer, Keith Bostic, Marshall Kirk McKusick, and Carl Staelin Memorandum No. UCB/ERL M92/134 3 December 1992 ELECTRONICS RESEARCH LABORATORY College of Engineering University ofCalifornia, Berkeley 94720 An Implementation ofa Log-Structured File System for UNIX1 Margo Seltzer —University ofCalifornia, Berkeley Keith Bostic - University ofCalifornia, Berkeley Marshall Kirk McKusick —University ofCalifornia, Berkeley Carl Staelin - Hewlett-Packard Laboratories ABSTRACT Research results [ROSE91] demonstrate that a log-structured file system (LFS) offers the potential for dramatically improved write performance, faster recovery time, and faster file creation and deletion than traditional UNIX file systems. This paper presents a redesign and implementation of the Sprite [ROSE91] log-structured file system that is more robust and integrated into the vnode interface [KLEI86].
    [Show full text]
  • Design and Implementation of the Spad Filesystem
    Charles University in Prague Faculty of Mathematics and Physics DOCTORAL THESIS Mikul´aˇsPatoˇcka Design and Implementation of the Spad Filesystem Department of Software Engineering Advisor: RNDr. Filip Zavoral, Ph.D. Abstract Title: Design and Implementation of the Spad Filesystem Author: Mgr. Mikul´aˇsPatoˇcka email: [email protected]ff.cuni.cz Department: Department of Software Engineering Faculty of Mathematics and Physics Charles University in Prague, Czech Republic Advisor: RNDr. Filip Zavoral, Ph.D. email: Filip.Zavoral@mff.cuni.cz Mailing address (advisor): Dept. of Software Engineering Charles University in Prague Malostransk´en´am. 25 118 00 Prague, Czech Republic WWW: http://artax.karlin.mff.cuni.cz/~mikulas/spadfs/ Abstract: This thesis describes design and implementation of the Spad filesystem. I present my novel method for maintaining filesystem consistency — crash counts. I describe architecture of other filesystems and present my own de- sign decisions in directory management, file allocation information, free space management, block allocation strategy and filesystem checking algorithm. I experimentally evaluate performance of the filesystem. I evaluate performance of the same filesystem on two different operating systems, enabling the reader to make a conclusion on how much the performance of various tasks is affected by operating system and how much by physical layout of data on disk. Keywords: filesystem, operating system, crash counts, extendible hashing, SpadFS Acknowledgments I would like to thank my advisor Filip Zavoral for supporting my work and for reading and making comments on this thesis. I would also like to thank to colleague Leo Galamboˇsfor testing my filesystem on his search engine with 1TB RAID array, which led to fixing some bugs and improving performance.
    [Show full text]
  • The Unix Operating System SE 101 Spiros Mancoridis What Is an OS?
    The Unix Operating System SE 101 Spiros Mancoridis What is an OS? An operating system (OS) is software that manages the resources of a computer Like most managers, the OS aims to manage its resources in a safe and efficient way Examples of computer resources are: CPU, RAM, disk memory, printers, displays, keyboard, mouse, etc The OS also isolates users and application programmers from the underlying computer Operating Systems Microsoft Windows Unix OS Architecture Without an OS, every application would have to implement some part of this software hierarchy ... Unix A popular multi-user, multi-tasking OS Attributes: stability, portability, security Created at Bell Labs by Dennis Ritchie and Ken Thompson (won the ACM Turing Award in 1983) Unix is considered one of the greatest achievements in computer science Has been around since the 1960s in various forms, e.g., AIX, SCO Unix, SunOS, FreeBSD, OpenBSD, NetBSD, Linux, Mac OS X Unix Multiuser and Multitasking Toolbox philosophy Concise syntax Designed by programmers for programmers 1983 ACM Turning Award (Unix) ACM is the Association for Computing Machinery World’s largest educational and scientific computer society Thompson and Ritchie You can become a student member too www.acm.org The ACM awards the Turing Award every year. It is the “Nobel Prize” of computing Named after british mathematician Alan M. Turing (1912-1954) Alan M. Turing Unix Kernel Includes device drivers for computer hardware devices, e.g., graphics cards, network cards, disks A device driver is a program that allows computer
    [Show full text]
  • The Evolution of File Systems
    The Evolution of File Systems Thomas Rivera, Hitachi Data Systems Craig Harmer, April 2011 SNIA Legal Notice The material contained in this tutorial is copyrighted by the SNIA. Member companies and individuals may use this material in presentations and literature under the following conditions: Any slide or slides used must be reproduced without modification The SNIA must be acknowledged as source of any material used in the body of any document containing material from these presentations. This presentation is a project of the SNIA Education Committee. Neither the Author nor the Presenter is an attorney and nothing in this presentation is intended to be nor should be construed as legal advice or opinion. If you need legal advice or legal opinion please contact an attorney. The information presented herein represents the Author's personal opinion and current understanding of the issues involved. The Author, the Presenter, and the SNIA do not assume any responsibility or liability for damages arising out of any reliance on or use of this information. NO WARRANTIES, EXPRESS OR IMPLIED. USE AT YOUR OWN RISK. The Evolution of File Systems 2 © 2012 Storage Networking Industry Association. All Rights Reserved. 2 Abstract The File Systems Evolution Over time additional file systems appeared focusing on specialized requirements such as: data sharing, remote file access, distributed file access, parallel files access, HPC, archiving, security, etc. Due to the dramatic growth of unstructured data, files as the basic units for data containers are morphing into file objects, providing more semantics and feature- rich capabilities for content processing This presentation will: Categorize and explain the basic principles of currently available file system architectures (e.g.
    [Show full text]