IEEE TRANSACTIONS ON COMPUTERS, VOL. 39, NO. 4, APRIL 1990 447 Coda: A Highly Available File System for a Distributed Workstation Environment MAHADEV SATYANARAYANAN, MEMBER, IEEE, JAMES J. KISTLER, PUNEET KUMAR, MARIA E. OKASAKI, ELLEN H. SIEGEL, AND DAVID C. STEERE Abstract- Coda is a file system for a large-scale distributed seriously inconvenience many users for time periods ranging computing environment composed of Unix workstations. It pro- from a few minutes to many hours. At least a few such outages vides resiliency to server and network failures through the use of occur somewhere in our system every day. two distinct but complementary mechanisms. One mechanism, server replication ,stores copies of a file at multiple servers. The The Coda File System is a descendant of AFS that is sub- other mechanism, disconnected operation, is a mode of execu- stantially more resilient to failures. The ideal that Coda strives tion in which a caching site temporarily assumes the role of for is constant data availability, allowing a user to continue a replication site. Disconnected operation is particularly use- working regardless of failures elsewhere in the system. Our ful for supporting portable workstations. The design of Coda goal is to provide users with the benefits of a shared data optimizes for availability and performance, and strives to pro- vide the highest degree of consistency attainable in the light repository, but to allow them to rely entirely on local resources of these objectives. Measurements from a prototype show that when that repository is partially or totally inaccessible. the performance cost of providing high availability in Coda is A related goal of Coda is to gracefully integrate the use reasonable. of AFS with portable computers. At present, users manually copy relevant files from AFS, use the machine while isolated Index Terms- Andrew, availability, caching, disconnected op- eration, distributed file system, performance, portable comput- from the network, and manually copy updated files back to ers, scalability, server replication. AFS upon reconnection. These users are effectively perform- ing manual caching of files with write-back on reconnection. If one views the disconnection from AFS as a deliberately- I. INTRODUCTION induced failure, it is clear that a mechanism for supporting LOCATION-transparent distributed file system based on portable machines in isolation is also a mechanism for fault the Unix’ file system model is a valuable mechanism for tolerance. collaborationA between physically dispersed users. This is par- Although this paper focuses on Coda, the problem of data ticularly true in a distributed workstation environment where availability is not specific to it. Growth in scale and complexity the primary activities are education, research, and software inevitably results in more components that can fail, and more development. The Andrew File System (AFS) is a highly hardware and software interactions that can manifest them- successful realization of such a mechanism for a campus-sized selves as failures. Consequently, the large-scale distributed user community Positive experience with AFS has file systems of the future will have to provide continued ac- motivated the recent work on extending it nationwide cess to data in the face of temporary failures if they are to The importance of a shared Unix file system for a distributed remain viable. workstation environment is further confirmed by many other In the rest of this paper, we describe the design and imple- efforts in industry and academia mentation of Coda. We motivate our design choices in Section The work described in this paper arose from our extensive II, present the design in Sections III-V, and elaborate on the experience as implementors and users of AFS over the past key data structures and protocols in Section VI. We describe five years. On the one hand, we were pleased with the func- the status and performance of a Coda prototype in Sections tionality, performance, and ease of administration of AFS. VII and VIII. We conclude with an examination of related At the same time we were concerned with its vulnerability to research in Section IX and a summary of the paper in Section failures of servers and network components. Such failures can X. Manuscript received July 6, 1989; revised November 19, 1989. This work II. DESIGN RATIONALE was supported by National Science Foundation Contract CCR-8657907, De- fense Advanced Research Projects Agency (Order 4976, ContractF33615 Three fundamental factors influenced the design of Coda. 87-C-l@@), the IBM Corporation (Faculty Development Award and Gradu- These are ate Fellowship), and Digital Equipment Corporation (Equipment Grant). The views and conclusions in this document are those of the authors and. do not . our desire to produce a highly scalable system represent the official policies of the timding agencies or Carnegie Mellon . the range of failures we wished to address University. The authors are with the School of Computer Science, Carnegie Mellon . the need to emulate Unix file semantics. University, Pittsburgh,’ PA 152 13. IEEE Log Number 8933887. In the course of our design, we discovered that the constraints Unix is a trademark of AT&T. imposed by these factors often conflicted with each other. The 0018-9340/90/0400-0447 $01 .OO 1990 IEEE 448 IEEE TRANSACTIONS ON COMPUTERS, VOL. 39, NO. APRIL 1990 current design of Coda is therefore a compromise that, in our Clients dynamically map files to servers and cache this judgment, best suits our usage environment. information. It uses token-based authentication and end-to-end encryp- tion integrated with its communication mechanism A. Scalability A scalable distributed system is one that can easily cope Range of Failures with the addition of users and sites. Growth has economic, To achieve our goal of continued client operation in the face performance, and administrative consequences. Our goal was of failures, we had two strategies available to us. The first was to build a system whose growth would incur as little expense, to use replication across servers to render the shared storage performance degradation, and administrative complexity as repository more reliable. The second was tdeliberately-o make each client possible. Since this goal was also the major focus of Coda’s capable of fully autonomous operation if the repository failed. ancestor, AFS, we tried to preserve much of its design. Each of these strategies improves availability, but neither is In AFS, a small set of trusted servers jointly provide a stor- adequate alone. age repository shared by a much larger number of untrusted Enhancing the availability of the shared storage repository clients. To maximize client-server ratio, most load is borne by increases the availability of all shared data. It protects against clients. Only functions essential to integrity or security are per- individual server failures and some network failures. Unfor- formed by servers. Caching is the key to scalability in AFS. tunately, it does not help if all servers fail, or if all of them The operating system on each client intercepts open and close are inaccessible due to a total partition of the client. A spe- file system calls2 and forwards them to a cache-management cial case of the latter is the use of a portable computer when process called Venus. After a file is opened, read and write detached from the network. operations on it bypass Venus. Venus contacts a server only Making each client fully autonomous is infeasible. The disk on a cache miss on open, or on a close after modification. In storage capacity of a client is a small fraction of the total both cases, the file is transferred in its entirety. Cache coher- shared data. This strategy is also inconsistent with our model ence is maintained by a callback mechanism, whereby servers of treating each client’s disk merely as a cache. It represents notify workstations of changes to cached files. Clients dynam- a return to the model of isolated personal computers rather ically determine the location of tiles on servers and cache this than a collection of workstations sharing a file system. The information. advantages of mobility, and the ability of any user to use any The highly dynamic nature of AFS enhances its scalabil- workstation as his own, are lost. Yet temporary autonomy ity. There are very few static bindings that require atomic, seems acceptable for brief periods of time, on the order of systemwide updates for Al-5 to function correctly. A work- minutes or hours, while a user is active at a client. station with a small disk can potentially access any file in AFS In the light of these considerations we decided to use a by name. One can move to any other workstation and effort- combination of the two strategies to cover a broad range of lessly access one’s files from there. Adding a new workstation failures. Coda uses server replication, or the storing of copies merely involves connecting it to the network and assigning of tiles at multiple servers, to provide a shared storage repos- it an address, Workstations can be turned off or physically itory of higher availability than AFS. A client relies on server relocated at any time without fear of inconveniencing other replication as long as it remains in contact with at least one users. Only a small operational staff is required to monitor server. When no server can be contacted, the client resorts and service the relatively few AFS servers. Backup is needed to disconnected operation, a mode of execution in which the only on the servers, since workstation disks are merely used client relies solely on cached data. We regard involuntary dis- as caches. Files can be easily moved between servers during connected operation as a measure of last resort and revert to normal operation without inconveniencing users.