Versatile, Portable, and Efficient File System Profiling A Dissertation Presented by Nikolai Joukov to The Graduate School in Partial fulfillment of the Requirements for the Degree of Doctor of Philosophy in Computer Science Stony Brook University Technical Report FSL-06-05 December 2006 Copyright by Nikolai Joukov 2006 Stony Brook University The Graduate School Nikolai Joukov We, the dissertation committee for the above candidate for the Ph.D. degree, hereby recommend acceptance of this dissertation. Dr. Erez Zadok, Advisor Professor, Computer Science Department, Stony Brook University Dr. Samir Das, Chair of Defense Professor, Computer Science Department, Stony Brook University Dr. Scott Stoller Professor, Computer Science Department, Stony Brook University Dr. Ethan Miller Professor, Computer Science Department, University of California Santa Cruz This dissertation is accepted by the Graduate School Dean of the Graduate School ii Abstract of the Dissertation Versatile, Portable, and Efficient File System Profiling by Nikolai Joukov Doctor of Philosophy in Computer Science Stony Brook University 2006 File systems are complex and their behavior depends on many factors. Source code, if available, does not directly help understand the file system’s behavior, as the behavior depends on actual workloads and external inputs. Runtime profiling is a key technique for understanding the behavior and mutual-influence of modern OS components. Such profiling is useful to prove new concepts, debug problems, and optimize the performance of existing file systems. Unfortunately, existing profiling methods are lacking in impor- tant areas: they do not provide much of the necessary information about the file system’s behavior, they require OS modification and therefore are not portable, or they exact high overheads thus perturbing the profiled file system. We developed a direct, real-time file system profiling method based on the analysis of latency distributions. Our method is versatile: a suitable workload can be used to profile virtually any OS component. Our method is portable because we can intercept operations and measure file system behavior from the user level or from inside the kernel without requiring source code. Our method is efficient: it has small overheads (less than 4% of the CPU time). Moreover, if the source code is available, we can use it to reduce overheads even further. In this dissertation we describe our profiling method, the theory behind it, and the au- tomation of the profile analysis. We demonstrate the usefulness of our method through a series of profiles conducted on Linux, FreeBSD, and Windows, including client/server scenarios. We discovered and investigated a number of interesting interactions, including scheduler behavior, multi-modal I/O distributions, and a previously unknown lock con- tention, which we fixed. We use our profiling method for performance analysis of a com- plex RAID-like fan-out stackable file system called RAIF that we have developed. iii Contents List of Figures vii List of Tables ix Acknowledgments x 1 Introduction 1 1.1 ContradictingProfilingRequirements . ..... 1 1.2 OurApproach................................. 2 1.3 OSprofandFSprof .............................. 3 1.4 ThesisOrganization. .. .. .. .. .. .. .. .. 3 2 Background 4 2.1 KernelCodeProfiling............................. 4 2.1.1 CPUExecutionProfiling . 4 2.1.2 LocksandMemoryProfiling. 5 2.1.3 FileSystemandI/OProfiling . 5 2.2 Latency-BasedProfiling . 5 2.3 FileSystemOperationsInterception . ..... 6 2.3.1 SourceCodeInstrumentation. 6 2.3.2 DynamicCodeInstrumentation . 6 2.3.3 InterceptionfromtheUser-Mode . 6 2.3.4 LayeredInterception . 6 3 Profiling Method 7 3.1 ProfileCollectionandAnalysis . ... 9 3.1.1 ProfilesPreprocessing . 9 3.1.2 PriorKnowledgeBasedAnalysis . 11 3.1.3 DifferentialProfileAnalysis . 12 3.1.4 LayeredProfiling.. .. .. .. .. .. .. .. 12 3.1.5 ProfilesSampling.. .. .. .. .. .. .. .. 12 3.1.6 DirectProfilesandValuesCorrelation . ... 14 3.2 ProfilesAnalysisAutomation. .. 14 3.2.1 IndividualProfilesComparison . 14 3.2.2 CompleteProfilesComparison. 16 iv 3.3 Multi-ProcessProfiles. 17 3.3.1 ForciblePreemptionEffects . 17 3.3.2 WaitTimesatHighCPULoads . 21 3.4 Multi-CPUProfiles .............................. 26 3.4.1 TimeSynchronization . 26 3.4.2 SharedDataStructures . 27 3.4.3 ProfileAnalysis............................ 28 3.5 ProfilinginVirtualEnvironments. .... 32 3.6 MethodSummary............................... 35 4 File System Instrumentation (FoSgen) 36 4.1 FiST...................................... 38 4.2 VFSOperationInterception. .. 38 4.3 FiSTSupportbyFoSgen ........................... 41 4.4 FSprof.fist................................... 43 4.5 FoSgenSteps ................................. 45 5 Implementation 47 5.1 The aggregate statsLibrary.......................... 47 5.2 POSIXUser-LevelProfilers . 47 5.3 WindowsUser-levelProfilers . .. 48 5.4 LinuxandFreeBSDFileSystemLevelProfilers . ..... 48 5.4.1 FoSgenandFSprofFiSTextensions . 48 5.5 WindowsFileSystemLevelProfilers. ... 49 5.6 DriverLevelProfilers . .. .. .. .. .. .. .. .. 49 5.7 ProfileAnalysisAutomation . 50 5.7.1 IndividualProfilesComparison . 50 5.7.2 CombinedProfileComparisonMethods . 51 5.8 RepresentingResults . .. .. .. .. .. .. .. .. 52 5.9 Portability................................... 53 6 Evaluation 54 6.1 MemoryUsageandCaches. 54 6.2 CPUTimeOverheads............................. 55 6.3 ProfileAnalysisAutomation . 56 6.4 FoSgen .................................... 60 7 Example File System Profiles 61 7.1 AnalyzingDiskSeeks............................. 61 7.2 AnalyzingFileSystemReadPatterns. .... 63 7.2.1 Ext2.................................. 63 7.2.2 NTFSandExt3............................ 65 7.3 HandlingAccessTimeUpdates. 67 7.3.1 NTFS................................. 67 7.3.2 ReiserfsandProfilesSampling. 67 v 7.3.3 Ext3.................................. 73 7.4 AnalyzingNetworkFileSystems. .. 74 7.5 InfluenceofStackableFileSystems . ... 77 8 Using FSprof without Buckets 81 8.1 WorkloadCharacterization . .. 81 8.2 QualityofReplayingAnalysis . .. 84 9 CaseStudy: RAIF 86 9.1 RedundantArrayofIndependentFilesystems . ...... 86 9.2 BenchmarkingwithPostmark. 89 9.2.1 RAIF0 ................................ 92 9.2.2 RAIF1 ................................ 92 9.2.3 RAIF4 ................................ 92 9.2.4 RAIF5 ................................ 92 9.3 RAIF5Profiling................................ 93 9.4 PriorandRelatedWork............................ 97 9.4.1 Fan-OutStackableFileSystems . 97 9.4.2 Block-LevelRAID . .. .. .. .. .. .. .. 97 9.4.3 File-System–LevelRAID . 97 9.4.4 StorageUtilization . 98 9.4.5 LoadBalancing............................ 98 9.4.6 RAIDSurvivability. 98 10 Conclusions 99 10.1 FutureWork..................................100 Bibliography 101 A FSprof FiST Extension 109 vi List of Figures 3.1 Aprofileof CreateThread operation on Windows XP, concurrently is- suedbytwoprocesses. ............................ 8 3.2 Complete profile of Linux 2.4.24 Ext2 under the grep -r workload. 10 3.3 Tri-modal Profile of the file read inode operation on a Linux 2.4.24 Ext2. 11 3.4 Our infrastructure allows profiling at the user, file system, driver, and net- work levels. Possible profiler locations are shown using the shadedboxes. 13 3.5 Three-dimensional profile of the Ext2 lookup operation under the kernel buildworkload. ................................ 15 3.6 Three-dimensional profile of the Ext2 lookup operation under the kernel build workload. Buckets contain their expected total latency. ........ 15 3.7 Forcible preemption probability as a function of the bucketnumber. 19 3.8 Profilesofa read operation on a preemptive and non-preemptive kernels. 20 3.9 Idle loop profiles with high resolution (r =2)................. 22 3.10 Profile of two processes that read files with varying amount of CPU activity betweenreads.................................. 23 3.11 Profile of an I/O-active process concurrently run with and without another CPU-onlyprocess. .............................. 25 3.12 TSCregistersynchronizationonSMPsystems. ....... 26 3.13 The probability of two concurrent writes to the same bucket on SMP systems. 29 3.14 Two processes updating the same set of buckets on a dual-CPU Linux system. 29 3.15 clone systemcallprofilesonLinuxinUPandSMPmodes. 30 3.16 clone systemcallprofilesonFreeBSD inUPandSMPmodes. 31 3.17 IdleloopprofilescapturedinVMware. ..... 34 4.1 The Base0fs stackable file system mounted overExt2. ........ 37 4.2 FoSgenscriptoperation. 37 4.3 Linux Ext2 directory operations and FreeBSD NFS vnode operations. 39 4.4 WindowsXPfilterdriverfilesystemoperations. ....... 40 4.5 Original Ext2 directory operations vector (top) and its FoSgen-transformed versionwiththewrapperfunctions(bottom).. ..... 42 4.6 A minimal latency profiling FiST extension for FoSgen. ......... 44 4.7 An original and generated writepage operation of the Ext2 file system. 46 6.1 Postmark configuration that we used for FSprof benchmarking........ 55 6.2 Profiledfunctioncomponents. .. 56 6.3 TOTOPS test results compared with manual profiles analysis. .. 57 vii 6.4 TOTLAT test results compared with manual profiles analysis. .. 57 6.5 CHISQUARE test resultscompared withmanualprofiles analysis. 58 6.6 EARTHMOVER test results compared with manual profiles analysis. 58 6.7 GROUPOPS test results compared with manual profiles analysis. 59 6.8 GROUPLAT test results compared with manual profiles analysis. 59 7.1 The llseek operationunderrandomreads.. 62 7.2 Profiles of Linux 2.6.11 Ext2 readdir (top) and readpage (bottom) operations captured for