THE UNIVERSITY OF CHICAGO FAST AND STABLE DATA AND STORAGE SYSTEMS IN MILLI/MICRO-SECOND ERA A DISSERTATION SUBMITTED TO THE FACULTY OF THE DIVISION OF THE PHYSICAL SCIENCES IN CANDIDACY FOR THE DEGREE OF DOCTOR OF PHILOSOPHY DEPARTMENT OF COMPUTER SCIENCE BY MINGZHE HAO CHICAGO, ILLINOIS DECEMBER 2020 Copyright © 2020 by Mingzhe Hao All Rights Reserved To my family. TABLE OF CONTENTS LISTOFFIGURES...................................... vii LISTOFTABLES ....................................... x ACKNOWLEDGMENTS.................................... xi ABSTRACT........................................... xii 1 INTRODUCTION ..................................... 1 1.1 AnalysisofStoragePerformanceInstability . ............ 2 1.2 BuildingFastandStableStorageSystems . ......... 4 1.2.1 Tiny-TailFlashDevices . .. 4 1.2.2 Millisecond Tail Tolerance with Fast Rejecting SLO-Aware OS Interface . 5 1.2.3 Predictability on Unpredictable Flash Storage with a LightNeural Network 7 1.3 ThesisOrganization. .. .. .. .. .. .. .. .. .... 8 2 BACKGROUNDANDMOTIVATIONS . 10 2.1 PerformanceInstabilityinStorageDevices. ............ 10 2.1.1 Extended Motivation – Read Latency Instability in Production . 12 2.2 Tail-ToleranceMechanisms. ...... 13 2.2.1 ExtendedMotivation–No“TT”inNoSQL . ... 15 2.3 Heuristic-basedandML-basedApproaches . ......... 15 2.3.1 Extended Motivation – Rarity of Research in the “ML for OS”area . 17 3 THE TAILATSTORE: A REVELATIONFROM MILLIONS OF HOURS OF DISK AND SSDDEPLOYMENTS ................................... 19 3.1 Methodology ..................................... 21 3.1.1 RAIDArchitecture .............................. 21 3.1.2 AbouttheDataset............................... 23 3.1.3 Metrics .................................... 24 3.2 Results......................................... 26 3.2.1 SlowdownDistributionsand Characteristics . .......... 27 3.2.2 WorkloadAnalysis .............................. 31 3.2.3 OtherCorrelations ............................. 34 3.2.4 Post-SlowdownAnalysis . .. 37 3.2.5 Summary ................................... 40 3.3 Tail-TolerantRAID ............................... ... 41 3.3.1 Tail-TolerantStrategies. ..... 42 3.3.2 Evaluation .................................. 43 3.4 Discussions ..................................... 44 3.5 RelatedWork ..................................... 46 3.6 Conclusion ...................................... 47 iv 4 MITTOS: SUPPORTING MILLISECOND TAIL TOLERANCE WITH FAST REJECT- INGSLO-AWAREOSINTERFACE. 48 4.1 MITTOSOverview .................................. 49 4.1.1 DeploymentModelandUseCase . 49 4.1.2 UseCase ................................... 50 4.1.3 Goals/Principles.............................. 51 4.1.4 DesignChallenges .............................. 52 4.2 CaseStudies..................................... 53 4.2.1 Disk Noop Scheduler (MITTNOOP)..................... 53 4.2.2 Disk CFQ Scheduler (MITTCFQ) ...................... 54 4.2.3 SSD Management (MITTSSD) ....................... 55 4.2.4 OSCache(MITTCACHE) .......................... 58 4.2.5 ImplementationComplexity . ... 59 4.3 Applications.................................... .. 59 4.4 MillisecondDynamism . .. .. .. .. .. .. .. .. ... 61 4.5 Evaluation...................................... 63 4.5.1 MicrobenchmarkResults. .. 64 4.5.2 MITTCFQResultswithEC2Noise . 65 4.5.3 Tail Amplified by Scale (MITTCFQ) .................... 68 4.5.4 MITTCACHE ResultswithEC2Noise . 70 4.5.5 MITTSSDResultswithEC2Noise . 70 4.5.6 PredictionAccuracy . 71 4.5.7 TailSensitivitytoPredictionError . ........ 73 4.5.8 Otherevaluations.............................. 73 4.6 Discussions ..................................... 77 4.6.1 MITTOSLimitations............................. 77 4.6.2 BeyondtheStorageStack . 78 4.6.3 OtherDiscussions .............................. 79 4.7 RelatedWork ..................................... 80 4.8 Conclusion ...................................... 80 4.9 Appendix:Details................................ ... 81 5 LINNOS: PREDICTABILITY ON UNPREDICTABLE FLASH STORAGE WITH A LIGHT NEURALNETWORK ................................... 83 5.1 Overview ....................................... 85 5.1.1 UsageScenario................................ 85 5.1.2 OverallArchitecture . .. 86 5.1.3 Challenges .................................. 88 5.2 LinnOSDesign .................................... 89 5.2.1 TrainingDataCollection . ... 90 5.2.2 Labeling(withInflectionPoint) . ..... 90 5.2.3 LightNeuralNetworkModel. .. 93 5.2.4 ImprovingAccuracy . .. .. .. .. .. .. .. .. 96 v 5.2.5 ImprovingInferenceTime . .. 98 5.2.6 SummaryofAdvantages . 99 5.2.7 ImplementationComplexity . 100 5.3 Evaluation...................................... 101 5.3.1 Setup .....................................101 5.3.2 InflectionPoint(IP)Stability. .. .. .104 5.3.3 LatencyPredictability . 105 5.3.4 (Low)Inaccuracy............................... 110 5.3.5 Trade-offsBalance . .. .. .. .. .. .. .. .. 111 5.3.6 OtherEvaluations.. .. .. .. .. .. .. .. .. 112 5.4 ConclusionandDiscussions . .115 6 OTHERSTORAGEWORK ................................117 6.1 LeapIO: Efficient and Portable Virtual NVMe Storage on ARMSoCs . .117 6.2 The CASE of FEMU: Cheap, Accurate, Scalable and Extensible Flash Emulator . 121 6.3 Tiny-Tail Flash: Near-Perfect Elimination of Garbage Collection Tail Latencies in NANDSSDs .....................................122 6.4 Fail-Slow at Scale: Evidence of Hardware Performance Faults in Large Production Systems ........................................124 7 CONCLUSIONANDFUTUREWORK. .126 7.1 Conclusion ...................................... 126 7.2 FutureWork...................................... 127 REFERENCES ......................................... 134 vi LIST OF FIGURES 2.1 Latency unpredictability. The figures show CDFs of block-level read latencies, as discussed in Section 2.1.1. For theleft figure, we ran one FIO workload on five different SSD models (the 5 CDF lines). For the right figure, we plot the latencies of 7 block- level traces obtained from 4 read-write servers (colored lines) and 2 read-only servers (bold gray lines) in Azure, Bing, and Cosmos clusters. The x-axis is anonymized for proprietary reasons. The traces are available from Microsoft with NDA. ....... 12 3.1 Stable and slow drives in a RAID group. ...................... 21 3.2 RAID width and dataset duration. .......................... 23 3.3 Conceptual drive slowdown model. ......................... 25 k 3.4 Slowdown (Si) and Tail (T ) distributions (Section 3.2.1-Section 3.2.1). The fig- ures show distributions of disk (top) and SSD (bottom) hourly slowdowns (Si), includ- ing the three longest tails (T 1−3) as defined in Table 3.2. The y-axis range is different in each figure and the x-axis is in log2 scale. We plot two gray vertical lines represent- ing 1.5x and 2x slowdown thresholds. Important slowdown-percentile intersections are listed in Table 3.3. .................................. 27 3.5 Temporal behavior (Section 3.2.1). The figures show (a) the CDF of slowdown intervals (#hours until a slow drive becomes stable) and (b) the CDF of slowdown inter-arrival rates (#hours between two slowdown occurrences). ........... 29 3.6 Slowdown extent (Section 3.2.1). Figure (a) shows the fraction of all drives that have experienced at least one occurrence of X-time slowdown ratio as plotted on the x-axis; the y-axis is in log10 scale. Figure (b) shows the fraction of slow drives that has exhibited at least X slowdown occurrences. .................... 31 3.7 CDF of size and rate imbalance (Section 3.2.2). Figure (a) plots the rate imbalance distribution (RIi) within the population of slow drive hours (Si ≥ 2). A rate imbalance of X implies that the slow drive serves X times more I/Os, as plotted in the x-axis. Reversely, Figure (b) plots the slowdown distribution (Si) within the population of rate-imbalanced drive hours (RIi ≥ 2). Figures (c) and (d) correlate slowdown and size imbalance in the same way as Figures (a) and (b). ................ 32 3.8 Drive age (Section 3.2.3). The figures plot the slowdown distribution across different (a) disk and (b) SSD ages. Each line represents a specific age by year. Each figure legend is sorted from the left-most to right-most lines. ................. 35 3.9 SSD models (Section 3.2.3). The figure plots the slowdown distribution across dif- ferent SSD models and vendors. ............................. 36 3.10 RAID I/O degradation (Section 3.2.4). The figures contrast the distributions of RIO degradation between stable-to-stable and stable-to-tail transitions. .......... 37 3.11 Unplug/replug events (Section 3.2.4-Section 3.2.4). The figures show the relation- ships between slowdown occurrences and unplug/replug events. The top and bottom figures show the distribution of “wait-hour” and “recur-hour” respectively. ...... 39 vii 3.12 ToleRAID evaluation. The figures show the pros and cons of various ToleRAID strategies based on two slowdown distributions: (a) Rare and (b) Periodic. The figures plot the T 1 distribution (i.e., the RAID slowdown). T 1 is essentially based on the longest tail latency among the necessary blocks that each policy needs to wait for. .. 43 4.1 MITTOS Deployment Model (Section 4.1.1). .................... 50 4.2 MITTOS use-case illustration (Section 4.1.2). .................... 52 4.3 Millisecond-level latency dynamism in EC2. The figures are explained in Section 4.4. The 20 lines in Figure (a)-(f) represent the latencies observed in 20 EC2 nodes. 61 4.4 Latency CDFs from microbenchmarks. The figures are explained in Section 4.5.1.