Serverless Network File Systems by Michael Donald Dahlin B.S. (Rice University) 1991 M.S. (University of California at Berkeley) 1993 A dissertation submitted in partial satisfaction of the requirements for the degree of Doctor of Philosophy in Computer Science in the GRADUATE DIVISION of the UNIVERSITY of CALIFORNIA at BERKELEY Committee in charge: Professor David A. Patterson, Chair Professor Thomas E. Anderson Professor Pamela Matson 1995 The dissertation of Michael Donald Dahlin is approved: Chair Date Date Date University of California at Berkeley 1995 Serverless Network File Systems Copyright © 1995 by Michael Donald Dahlin All rights reserved Abstract Serverless Network File Systems by Michael Donald Dahlin Doctor of Philosophy in Computer Science University of California at Berkeley Professor David A. Patterson, Chair This thesis presents the design of a serverless network file system, a file system that dis- tributes its functionality among cooperating, networked machines to eliminate the central file server bottleneck found in current network file systems. In a serverless system, any machine can cache, store, or control any block of data. This location independence provides better performance and scalability than traditional file system architectures. Further, because any machine in the system can assume the responsibilities of a failed component, the serverless design can also provide higher availability. This dissertation details the design of three serverless subsystems, each of which distrib- utes a specific piece of functionality that a traditional system would implement as part of a central server. I describe and evaluate cooperative caching, a way to coordinate client caches to replace the central server’s cache. I evaluate different distributed disk storage architectures and present several improvements on previous log-structured, redundant storage systems. Finally, I present the design of a distributed management architecture that splits the control of the file system among managers while maintaining a seamless single system image. Together, these pieces form a a serverless file system that eliminates central bottlenecks. I describe the integration of these components in the context of xFS, a prototype serverless file system implementation. Initial performance measurements illustrate the promise of the serverless approach for providing scalable file systems. Professor David A. Patterson 1 Table of Contents CHAPTER 1. Introduction .....................................................................................1 1.1. Design Environment and Applicability ...........................................4 1.2. Overview of the Dissertation...........................................................6 CHAPTER 2. Trends in Technology .....................................................................8 2.1. Trends in Technology......................................................................8 2.1.1. Disk Performance........................................................................9 2.1.2. Disk Capacity............................................................................10 2.1.3. Network Latency.......................................................................10 2.1.4. Network Bandwidth..................................................................11 2.1.5. Processor Performance..............................................................12 2.1.6. Memory Bandwidth ..................................................................12 2.1.7. Memory Capacity......................................................................13 2.2. Increasing Demands on File Systems............................................13 2.2.1. Demands From Sequential Applications ..................................14 2.2.2. Demands From Parallel and Distributed Applications .............15 2.2.3. Demands From New Applications............................................16 2.3. Implications For File System Design ............................................17 2.3.1. Cooperative Caching.................................................................17 2.3.2. Distributed Storage ...................................................................18 2.3.3. Distributed Management...........................................................19 2.4. Other Technology Trends..............................................................19 CHAPTER 3. Cooperative Caching .....................................................................21 3.1. Cooperative Caching Algorithms ..................................................23 3.1.1. Direct Client Cooperation.........................................................24 3.1.2. Greedy Forwarding ...................................................................25 3.1.3. Centrally Coordinated Caching.................................................26 3.1.4. Hash Coordinated Caching .......................................................27 3.1.5. Weighted LRU Caching............................................................28 3.1.5.1. Limiting Weighted-LRU Communication Requirements.........31 3.1.6. N-Chance Forwarding...............................................................33 3.1.7. Other Algorithms ......................................................................34 3.2. Simulation Methodology ...............................................................35 3.3. Simulation Results.........................................................................37 3.3.1. Comparison of Algorithms .......................................................38 3.3.2. Detailed Analysis of the Algorithms.........................................43 3.3.2.1. Direct Client Cooperation.........................................................44 3.3.2.2. Greedy Forwarding ...................................................................45 3.3.2.3. Centrally Coordinated and Hash Coordinated Caching............45 3.3.2.4. Weighted LRU..........................................................................47 iii 3.3.2.5. N-Chance Forwarding...............................................................48 3.3.3. Sensitivity to Hardware Parameters..........................................49 3.3.3.1. Client Cache Size......................................................................49 3.3.3.2. Dynamic Cache Sizing..............................................................50 3.3.3.3. Server Cache Size .....................................................................50 3.3.3.4. Network Speed..........................................................................51 3.3.3.5. Future Projections .....................................................................52 3.3.4. Other Workloads.......................................................................54 3.3.4.1. Other Sprite Traces ...................................................................54 3.3.4.2. Berkeley Auspex Workload......................................................56 3.3.4.3. Worst-case Workloads..............................................................59 3.3.5. Summary of Simulation Results ...............................................63 3.4. Related Work.................................................................................65 3.5. Conclusions....................................................................................67 CHAPTER 4. Distributed Disk Storage ...............................................................68 4.1. Log-based Network Striping..........................................................70 4.1.1. RAID.........................................................................................70 4.1.2. LFS............................................................................................71 4.1.3. Zebra .........................................................................................73 4.1.4. Limits to Zebra’s Scalability.....................................................75 4.1.5. Alternatives to Log-based Striping ...........................................75 4.1.5.1. RAID Level 1 Mirroring...........................................................75 4.1.5.2. AutoRAID.................................................................................76 4.1.5.3. Journaling File Systems ............................................................77 4.2. Stripe Groups.................................................................................77 4.2.1. Motivation for Stripe Groups....................................................78 4.2.2. Implementing Stripe Groups.....................................................80 4.3. Cleaning.........................................................................................82 4.3.1. Monitoring Segment Utilization ...............................................83 4.3.1.1. Distributed Bookkeeping ..........................................................83 4.3.1.2. S-Files .......................................................................................84 4.3.2. When to Activate Cleaners .......................................................85 4.3.3. Distributing Cleaning................................................................85 4.3.4. Coordinating Cleaners ..............................................................86 4.4. Cleaning and Cooperative Caching ...............................................86 4.5. Availability ....................................................................................89 4.5.1. Local Storage Server State........................................................89