Measurement-Based Modeling of Distributed Systems Meßbasierte Modellierung verteilter Systeme Der Technischen Fakultät der Universität Erlangen-Nürnberg zur Erlangung des Grades DOKTOR-INGENIEUR vorgelegt von Kai-Steffen Jens Hielscher Erlangen - 2008 Als Dissertation genehmigt von der Technischen Fakultät der Universität Erlangen-Nürnberg Tag der Einreichung: 12. März 2008 Tag der Promotion: 21. April 2008 Dekan: Prof. Dr.-Ing. habil. Johannes Huber Berichterstatter: Prof. Dr.-Ing. Reinhard German Prof. Dr.-Ing. Wolfgang Schröder-Preikschat Contents List of Figures7 List of Tables9 Abstract 11 Zusammenfassung 15 1 Introduction 19 2 Related Work 23 ó.Õ Measurements.............................. óì ó.ó Time Synchronization.......................... ó¦ ó.ì Input Modeling.............................. ó¢ ó.¦ Performance Evaluation of Web Servers............... ó¢ 3 The Web Cluster Laboratory 27 ì.Õ e Linux Virtual Server System.................... ó ì.ó Hardware Setup............................. ìþ 4 Measurement Concepts 33 ¦.Õ Computer Clocks............................ ì ¦.ó Clock Errors............................... ìÉ ¦.ó.Õ Classication in the Frequency Domain........... ¦ä ¦.ó.ó Classication in the Time Domain.............. ¦ ¦.ì Reference Clocks............................. ¢Õ ¦.ì.Õ NTP............................... ¢ó ¦.ì.ó Time Sources.......................... ¢¦ ¦.ì.ì e PPS API.......................... ¢¢ ì Contents 5 Dedicated Measurement Infrastructure 59 ¢.Õ PPS Pulse Latency............................ äì ¢.Õ.Õ Echo Feedback......................... ä¢ ¢.ó Oine Synchronization......................... äÉ ¢.ì Instrumentation............................. ßì ¢.ì.Õ IP Stack Instrumentation................... ßì ¢.ì.ó Web Server Instrumentation................. ßß ¢.ì.ì Load Generator Instrumentation............... ßß ¢.ì.¦ Application Server Instrumentation............. ß ¢.ì.¢ Summary Performance Data................. Õ ¢.¦ Analysis of the Traces.......................... ó ¢.¢ Example Measurement Results..................... ì 6 Advanced Input Modeling 91 ä.Õ Traces and Empirical Distributions.................. Éó ä.ó Outlier Values.............................. ɦ ä.ì Autocorrelation............................. É¢ ä.¦ Standard eoretical Distributions.................. Éß ä.¢ Multimodal Distributions........................ ÕþÕ ä.ä Multimodal Distributions with Phases................ Õþó ä.ß Bézier Distributions........................... Õþ¢ ä. A New Model for Autocorrelated Data................ Õþß 7 Simulation Model 117 ß.Õ Model Structure............................. ÕÕ ß.ó TCP.................................... ÕÕÉ ß.ó.Õ RFC ßÉì............................. Õóþ ß.ó.ó RFC ÕÕóó............................. Õóì ß.ó.ì RFC Õìóì............................. Õó¦ ß.ó.¦ RFC ó¢Õ............................. Õó¢ ß.ó.¢ RFC óÉ............................ Õóä ß.ì Client................................... Õó ß.ì.Õ Application........................... Õó ß.ì.ó TCP............................... ÕóÉ ß.ì.ì Processor............................ ÕóÉ ß.¦ Network Channels............................ Õìþ ß.¢ Load Balancer.............................. Õìó ¦ Contents ß.ä Servers.................................. Õìó ß.ä.Õ Processes............................ Õìì ß.ä.ó System Processes........................ Õì¦ ß.ä.ì Processor............................ Õì¢ ß.ß Utility Classes and Execution Control................. Õìä ß. Experiments............................... Õìß 8 Conclusions and Future Work 143 Bibliography 147 ¢ Contents ä List of Figures ì.Õ Distributed Web Server Architecture................. óß ì.ó Load Balancing via NAT........................ óÉ ¦.Õ Hardware Monitoring.......................... ì¢ ¦.ó Soware Monitoring.......................... ìä ¦.ì Hybrid Monitoring........................... ìß ¦.¦ Latencies for Reading the Time.................... ìÉ ¦.¢ Frequency Changes with Temperature................ ¦Õ ¦.ä Frequency Variation........................... ¦ó ¦.ß Frequency Distribution......................... ¦ì ¦. Phase Errors............................... ¦¦ ¦.É UDP Delays................................ ¦¢ ¦.Õþ Power-Law Spectral Densities..................... ¦ß ¦.ÕÕ Allan Deviation............................. ¢þ ¦.Õó NTP Time Transfer........................... ¢ó ¦.Õì NTP Architecture............................ ¢¦ ¦.Õ¦ NTP and the PPS API.......................... ¢ä ¢.Õ Detail of UDP Delays.......................... äÕ ¢.ó Synchronization System......................... äó ¢.ì Interrupt Latencies............................ ä¢ ¢.¦ External Clock.............................. ää ¢.¢ Time Deviation............................. ä ¢.ä Oine Synchronization......................... ßþ ¢.ß IP Stack Instrumentation........................ ߢ ¢. Application Server Instrumentation Architecture.......... þ ¢.É Illustration of Delays in the Object System.............. ä ¢.Õþ Trace Plot of Measured Delays..................... ß ¢.ÕÕ Trace Plots of Individual Delays.................... ß List of Figures ¢.Õó Delay Components for Requests.................... É ¢.Õì Summary Statistics the Delays..................... Éþ ä.Õ Histograms of Observed Delays.................... Éì ä.ó Correlation Plots (lag ≤ ¢þþ)...................... Éä ä.ì Correlation Plots (lag ≤ ¦þ)....................... É ä.¦ Trace Plots Sorted by Real Server...................ÉÉ ä.¢ Distribution Comparison for Delay óó................ ÕþÕ ä.ä Distribution Comparison for Delay ì................. Õþì ä.ß State Chart for Phase Transitions................... Õþ¦ ä. Distribution Comparison for Delay ÕÉ................ Õþ¢ ä.É Screenshot of PRIME.......................... Õþä ä.Õþ Distribution Comparison for Delay Õ................ Õþ ä.ÕÕ Histogram Ho of the Deltas for Delay ¢................ ÕþÉ ä.Õó Trace Plot of Delay ¢........................... ÕÕþ ä.Õì Delta over the Values of Delay ¢.................... ÕÕÕ ä.Õ¦ ìD Histogram of Delta ¢........................ ÕÕó ä.Õ¢ Weighting Areas............................. ÕÕì ä.Õä Weighting Factors............................ ÕÕ¦ ä.Õß Original and Weighted Histogram for Delta ¢............ ÕÕ¦ ä.Õ Distribution Comparison for Delay ¢................. ÕÕ¢ ß.Õ Conceptual Model............................ ÕÕÉ ß.ó TCP.................................... Õóþ ß.ì Model of a TCP Segment........................ ÕóÕ ß.¦ Central TCP State Chart receive_packet ........... Õóó ß.¢ Structure of the Client Object..................... Õó ß.ä Conceptual Model of the Network Channels............. ÕìÕ ß.ß Server Model and Embedded Objects................. Õìì ß. Graphical Comparison of the Results................. Õ¦þ List of Tables ¦.Õ Slope Characteristics.......................... ¢Õ ¢.Õ Quantile Summary for Delays in Microseconds........... ¢ ä.Õ Fitted Standard eoretical Distributions.............. Õþþ ä.ó Fitted Multimodal Distributions.................... Õþó ä.ì Fitted Multimodal Distributions with Phases............ Õþ¦ ß.Õ Core Simulation Parameters...................... ÕìÉ ß.ó Quantile Comparisons in Milliseconds................ Õ¦Õ ß.ì CPU Load Comparison......................... Õ¦ó É List of Tables Õþ Abstract Nowadays, distributed systems are ubiquitous. Since the delays during processing in such systems are oen essential, many research projects deal with performance analyses of these systems. Most of them treat the systems from an abstract point of view and oarse-grained models are built. ese do not include results of detailed measurement studies of real systems. e goal of this work is to demonstrate a methodology that allows to create precise models of distributed systems that are parametrized, calibrated and validated from ne-grained measurements of a laboratory setup. e approach is exemplied on a cluster-based web server system. e resulting model contains many details that inuence the behavior and performance of the system like one-way delays or system activity caused by the hardware. Since network aspects play a central role in distributed systems, it is important to be able to capture the timing characteristics of packet delays in the network exactly. erefore, a modular system has been developed for the Linux operating system that allows to record sent and received TCP segments and to generate timestamps for the sending and receiving actions. For that purpose, the netlter framework has been extended to insert packet headers and corresponding timestamps into a ring buer in kernel space. e timestamps in the resulting event trace are generated using the clock of the object system that is observed. To calculate one- way delays for packets, it is necessary to synchronize the clocks of the nodes, because the timestamps for the sending and the receiving event are taken from dierent clocks. is can be achieved with standard solutions like the use of NTP during the measurement. An alternative that is more suitable in many situations is the use of a dedicated oine synchronization. is method is an own development based on an algorithm classically used for online synchronization of computer clocks. e method allows to use the cycle counter of the processor (TSC) of the object system for timestamping. Due to this feature, a context switch for obtaining kernel clock timestamps can be avoided. erefore, the latencies for ÕÕ Abstract reading the clocks are minimized. To implement this solution, the PPS output of a GPS receiver is connected to the nodes of the object system that need to be synchronized. PPS