Time Surveying: Clock Synchronization over Packet Networks by Gregory D. Troxel S.B., Massachusetts Institute of Technology (1987) S.M., E.E., Massachusetts Institute of Technology (1990) Submitted to the Department of Electrical Engineering and Computer Science in partial fulfillment of the requirements for the degrees of Doctor of Philosophy in Electrical Engineering and Computer Science at the Massachusetts Institute of Technology May, 1994 (O)1994 Massachusetts Institute of Technology. All rights reserved. Signature of Author In ,. I.. I Department of Electrical Engineering and Computer Science I/n /1 /' .~?7 h May 12, 1994 Certified by IJ/ Ad --- - I - ::, David Clark 2 Thesis Supervisor Accepted by LIBRARIES 2 Time Surveying: Clock Synchronization over Packet Networks by Gregory D. Troxel Submitted to the Department of Electrical Engineering and Computer Science on May 12, 1994 in partial fulfillment of the requirements for the degree of Doctor of Philosophy in Electrical Engineering and Computer Science Abstract This thesis applies the methods and mindset of a high-accuracy land surveyor to the problem of clock synchronization over packet networks. It presents a novel technique for analyzing synchronization performance, termed the running sum of adjustments, or rsadj, synchronization measurement technique. The new technique does not require the use of an accurate local time reference. It presents a novel synchronization scheme, based on the rsadj technique, termed integrated timekeeping. The thesis discusses what hardware support is necessary to support precision timekeeping, and argues that only very simple mechanisms are needed. The rsadj synchronization measurement technique completely decouples the effects of the synchronization algorithm from the underlying behavior of the computer's oscillator, the network and remote clocks. The measurement technique uses the oscillator of the computer as an important measurement tool; it simultaneously characterizes the oscillator and uses it to analyze the behavior of remote clocks. It estimates oscillator parameters, and computes how the clock should have been steered. The original data and what the behavior would have been had the clock been optimally steered are examined. The thesis describes an implementation of the technique, and examines actual synchronization behavior. The thesis examines the behavior of an important and widely deployed synchronization mechanism, the Network Time Protocol (NTP). It shows that NTP does not fully exploit the stability of the computer's oscillator under some conditions, particularly when the oscillator is very stable (a few parts in 108) and significant numbers of packets encounter 10 ms or more of additional delay. The thesis suggests incremental changes to improve NTP. The thesis presents a novel technique for synchronization, termed integrated timekeeping, which periodically estimates parameters of the computer's oscillator and sets the system clock based on the estimated parameters. The thesis presents data showing the behavior of clocks synchronized by NTP and by the integrated timekeeping scheme, obtained by simu- lation and by an implementation. Given realistic network conditions and a local oscillator stable to a few parts in 108, the new scheme achieves timekeeping stability of approximately an order of magnitude better than NTP. Thesis Supervisor: David Clark Title: Principal Research Scientist 3 4 Acknowledgments I would like to thank David Mills for his contagious devotion to accurate time, and for his authorship of the Network Time Protocol, without which this work would almost certainly not have occurred. I have also appreciated his encouragement over the years to try to actually synchronize clocks. I appreciate very much his loan of a WWVB receiver to my research group, and his continuing efforts to provide a high-quality implementation of NTP to the Internet community. I would like to thank Ty Sealy, Tom Greene, Charles Robinson, William Ang, and Kevin Garceau of the MIT Laboratory for Computer Science for their assistance in installing experimental apparatus for this research, securing permission to install equipment in space not controlled by the Laboratory, and helping with the inevitable problems that occur with such installations. I would like to thank the staff of BTE Engineering for their acceptance of my equipment in their lunchroom. I would like to thank Bill Chiarchiaro for the loan of one of the oscillators used in this research. John Paul Braud helped with the derivation of formulas used in the very early days of this work. Karen Palmer and other current and former residents of Ashdown House helped along the way in ways that might seem small but were important and much appreciated. Many members of the Advanced Network Architecture and Telemedia, Networks, and Systems research groups of the MIT Laboratory for Computer Science were helpful during this research. I would like to thank Mike Ciholas for the many hours of helpful discussions about hardware requirements for synchronization, and his work in the design and con- struction of the prototype GDT-TIME hardware timekeeping module. I had many useful discussions with Tim Shepard, who was always willing to listen to me attempt to explain my schemes, and provided helpful comments on drafts of this document. He also helped greatly in taking care of experimental apparatus at times when it needed attention and I was not at the lab. His authorship of the plotting program used in this work is greatly appreciated, as are contributions to that program made by Andrew Heybey. Vanu Bose read drafts of several chapters, and provided many helpful comments. I appreciate the ty- pographical assistance of Mark Reinhold, who rescued me from having incompatible fonts in the figures. I appreciate the assistance of Kathi-Jo Madera and Lisa Taylor during the several years over which this research was conducted. I would like to thank my thesis supervisor, David Clark, for encouraging me to pursue this research even though it was not directly relevant to the research agenda of his research group. Our many discussions helped greatly in refining the ideas presented in this document from their original muddled form into ones that can be stated clearly and precisely. I would like to thank Tom Herring, who served as a reader for this thesis, for his helpful comments on drafts of the document. As a student in his class on the Global Positioning System I became interested in surveying, and this led to a greater understanding of the problem at hand. I would like to thank Barbara Liskov for her extremely helpful comments, particularly on the presentation of material in this document, and for having shown that synchronizing clocks well almost all of the time is very useful. I would like to thank my parents for their encouragement to attend graduate school, their support over many years, and for instilling in me a desire to learn. My sisters, Andrea 5 and Jocelyn, provided statistical advice and encouragement. I would like to thank my wife, Mary Ellen, for her support, love, and companionship. Her constant encouragement made writing the document much more pleasant than it otherwise would have been, and her expert proofreading of final drafts helped greatly. This research was supported by the Advanced Research Projects Agency under contract number DABT63-92-C-0002. 6 Contents 1 Introduction 11 1.1 Overview. 11 1.2 What is clock synchronization, and why is it impiortant? .......... 13 1.3 The Network Time Protocol ............ ... .... ... ... ... 14 1.3.1 Offset measurements. ................ .. 15 1.3.2 Information about time quality ...... ... ... .. ..... 16 1.3.3 Selection of low-delay observations .... ......... 16 1.3.4 Clock selection and combining ...... .. ......... 16 1.3.5 Manipulation of the system clock ..... .. ..... ..... ... 17 1.3.6 Performance of NTP. ..... ......... 18 1.4 Time Surveying ................... .............. 18 1.4.1 The nature of surveying .......... .............. 18 1.4.2 Why is Time Surveying different? ..... .............. 20 1.4.3 Tasks of the Time Surveyor ........ .............. 22 1.4.4 Approaches to time synchronization ... .............. 23 1.4.5 Trusted references. .............. 23 1.5 Related work. .............. 24 1.5.1 Improvements to NTP by David Mills . ... .. ... ... .. 25 1.5.2 The Digital Time Synchronization Service ................ 25 1.5.3 The Smart Clock scheme of NIST .... ................. 25........ 1.6 Roadmap ...................... .............. ..26 2 The rsadj Synchronization Measurement Technique 29 2.1 The Basic rsadj Technique ................. ........... .30 2.1.1 Motivation. ........... .30 2.1.2 Keeping track of adjustments to the system clock -- rsadj ..... 30 2.1.3 Uncorrected observations . ........... .32 2.1.4 Modeling of the local oscillator ........... ........... .33 2.1.5 Use of hindsight estimates as an analysis tool ... ........... .34 2.2 Definitions for the rsadj technique ............. ........... .36 2.2.1 True time and other kinds of time ......... ........... .36 2.2.2 Different varieties of rsadj . .. ......... ..37 2.2.3 Different varieties of offsets ............. ......... ..37 2.3 What does time as a function of time mean? ....... ......... ..38 2.4 Recorded data and derived data ............... ......... ..39 2.5 Basic computation of predicted time ............ ......... ..39 2.5.1 Calculatepredrsadj rather than ptime . .............. ..40 2.5.2 Minimizepoffset rather than ptime - ttime . ........