Reducing the Cost of System Administration of a Disk Storage System Built from Commodity Components Satoshi Asami Report No. UCB/CSD-00-1100 May 2000 Computer Science Division (EECS) University of California Berkeley, California 94720 Reducing the Cost of System Administration of a Disk Storage System Built from Commodity Components by Satoshi Asami B.S. (University of Tokyo) 1989 M.S. (University of Tokyo) 1991 A dissertation submitted in partial satisfaction of the requirements for the degree of Doctor of Philosophy in Computer Science in the GRADUATE DIVISION of the UNIVERSITY of CALIFORNIA at BERKELEY Committee in charge: Professor David A. Patterson, Chair Professor Robert Wilensky Professor J. George Shanthikumar 2000 The dissertation of Satoshi Asami is approved: Chair Date Date Date University of California at Berkeley 2000 Reducing the Cost of System Administration of a Disk Storage System Built from Commodity Components Copyright 2000 by Satoshi Asami 1 Abstract Reducing the Cost of System Administration of a Disk Storage System Built from Commodity Components by Satoshi Asami Doctor of Philosophy in Computer Science University of California at Berkeley Professor David A. Patterson, Chair This dissertation explores how to reduce the system administration cost of disk storage systems. There are several reasons why reducing the operator’s burden is the key to success of large storage systems. One is that the cost of system administration usually dominates the budget of storage systems. Another is that an operator error on storage systems can easily have disastrous results. In the field of physiology and psychology, there have been studies that show reducing mental and physical stress on the operator is crucial in preventing human errors. This dissertation describes Tertiary Disk, a large-scale disk array system built from com- modity components, and how we evaluated the feasibility of its design. Instead of incurring the cost of custom hardware, we attempt to solve various problems by design and software. Tertiary Disk is a cluster of storage nodes connected by switched Ethernet. Each storage node is a PC hosting a few dozen SCSI disks, running the FreeBSD operating system. The system is used as a web-based image server for the Zoom Project in cooperation with the Fine Arts Museums of San Francisco. Our system is fully redundant in both hardware and software, and is designed to avoid a single point of failure. There are several approaches to lower the human cost of system administration. One is to make the system as autonomous as possible. I have designed a self-maintenance extension to the operating system to make the system run continuously in the event of failures. There are also several other improvements to the system to make the operator’s job easier. Finally, we will prove the feasibility of our system by evaluating it by simulation. Failure data that has been collected on Tertiary Disk over the course of several years were used to design an event generator. The second program, a simulator, models the system using a directed acyclic graph and computes its availability by solving a connectivity problem. The results have shown that our system performs as expected with the current set of parameters, and also expands nicely into the future. Professor David A. Patterson Dissertation Committee Chair iii Dedication To my parents—I wouldn’t be here without you. iv Contents List of Figures vii List of Tables ix 1 Introduction 1 1.1HaveComputersReallyBecomeCheaper?...................... 1 1.1.1 BuildingLargeStorageSystems....................... 3 1.1.2 SystemAdministrationCost......................... 3 1.1.3 OurApproach................................. 3 1.1.4 Applicability ................................. 4 1.2 Outline of Dissertation . ............................. 4 1.3RelatedWork..................................... 5 1.3.1 DiskStorageSystems............................ 5 1.3.2 FastRecovery................................. 7 1.3.3 Monitoring and Diagnosis . ....................... 8 1.3.4 Self-Management of Storage . ....................... 8 1.4 Research Issues/Contributions ............................ 9 2 Motivation 10 2.1Overview....................................... 10 2.1.1 Backups . .................................. 10 2.1.2 Redundancy .................................. 11 2.2TheImpactofSystemAdministration........................ 12 2.2.1 ComparisonwithHardwareCost....................... 12 2.2.2 TheRiskFactor................................ 13 2.2.3 Availability .................................. 14 2.3OurApproach..................................... 15 2.4Summary....................................... 15 3 The System 16 3.1TheApplication.................................... 16 3.1.1 Problems................................... 16 3.1.2 GRIDPIX ................................... 17 3.1.3 Status..................................... 17 v 3.2 Tertiary Disk Architecture . ............................. 19 3.2.1 Commodity Components ........................... 21 3.2.2 Redundancy .................................. 21 3.2.3 Topology . .................................. 22 3.2.4 DiskInterface................................. 23 3.3SoftwareArchitecture................................. 27 3.3.1 HandlingUserRequests........................... 28 3.3.2 Operating System Support . ....................... 30 3.4Summary....................................... 33 4 The Cost of System Administration 34 4.1 Self-Maintaining System . ............................. 35 4.1.1 Definition of Self-Maintaining System . .................. 35 4.1.2 Requirements................................. 37 4.2ASystemThatIsEasyToMaintain......................... 37 4.2.1 DiskIdentification.............................. 38 4.2.2 Taking Advantage of Redundancy ...................... 38 4.2.3 Modular Upgrading . ............................. 39 4.3Summary....................................... 40 5 Failures 41 5.1CollectingData.................................... 41 5.1.1 SystemLog.................................. 41 5.1.2 HTTPServerLog............................... 43 5.1.3 RepairRecords................................ 43 5.2FailureSummary................................... 44 5.3HardwareErrorDetails................................ 45 5.3.1 PC....................................... 45 5.3.2 Network.................................... 47 5.3.3 TheSCSISubsystem............................. 48 5.4SoftwareErrors.................................... 51 5.4.1 OperatingSystem............................... 51 5.4.2 Application.................................. 51 5.5Summary....................................... 52 6 Validation 53 6.1SimulationSetup................................... 54 6.1.1 TheGenerator................................ 55 6.1.2 TheSimulator................................ 57 6.2AlternativestoSimulation.............................. 63 6.3SimulationResultsandExploration......................... 63 6.3.1 Overview................................... 64 6.3.2 FailureRates................................. 64 6.3.3 OperatorCost................................. 68 6.3.4 RepairInterval................................ 69 vi 6.3.5 Disk Striping ................................. 72 6.3.6 Mirroring . .................................. 74 6.3.7 RAID-5.................................... 75 6.3.8 Double-Ending . ............................. 75 6.3.9 OperatingSystemCrashes.......................... 78 6.3.10NumberofPCsperDisk........................... 80 6.3.11TotalSize................................... 80 6.3.12 99.999% or 99.9999% Availability ...................... 83 6.4Summary....................................... 84 7Conclusion 85 7.1Results......................................... 85 7.2 Contributions . .................................. 85 7.3WhatWeWouldHaveDoneDifferently....................... 86 7.3.1 DiskInterface................................. 86 7.3.2 Disk Purchases . ............................. 86 7.3.3 DiskEnclosures............................... 87 7.3.4 PC Cases . .................................. 87 7.4FutureWork...................................... 87 7.4.1 Completion.................................. 88 7.4.2 DesignParameters.............................. 88 Bibliography 89 vii List of Figures 3.1Three-layerGridPiximage.............................. 18 3.2 GRIDPIX interface.................................. 18 3.3 Tertiary Disk Final Prototype ............................. 20 3.4 Double-Ending of SCSI disks ............................. 21 3.5 Double-Ending with feed-through terminators . .................. 22 3.6 Power connection and mirroring topology on a system with constant termination . 24 3.7 Power connection and mirroring topology on a system without constant termination 25 3.8 Ethernet connection and mirroring topology . .................. 26 3.9Handlinguserrequests................................ 28 3.10Frontendswitchingprotocol............................. 29 5.1Samplesystemlog.................................. 42 5.2 SCSI disk read/write errors . ............................. 49 6.1Asimpleconnectivitygraph............................. 54 6.2Ethernetcables.................................... 55 6.3Samplegeneratoroutput............................... 57 6.4 OR-node ........................................ 58 6.5 AND-node ....................................... 59 6.6 Data, datasets and striped sets ............................ 60 6.7Powersupplies.................................... 61 6.8 Redundant power supplies