QUT Digital Repository: http://eprints.qut.edu.au/ Tian, Guo-Song and Tian, Yu-Chu and Fidge, Colin J. (2008) High-precision relative clock synchronization using time stamp counters. In: 13th IEEE International Conference on Engineering of Complex Computer Systems, 31 March-4 April 2008, Beflast, UK. © Copyright 2008 IEEE Personal use of this material is permitted. However, permission to reprint/republish this material for advertising or promotional purposes or for creating new collective works for resale or redistribution to servers or lists, or to reuse any copyrighted component of this work in other works must be obtained from the IEEE. 13th IEEE International Conference on Engineering of Complex Computer Systems High-Precision Relative Clock Synchronization Using Time Stamp Counters Guo-Song Tian, Yu-Chu Tian and Colin Fidge Faculty of Information Technology, Queensland University of Technology, Brisbane, QLD 4001, Australia Abstract quirements for relative clock synchronization in control sys- tems are of the order of millisecond or even sub-millisecond In this paper we show how to use a computer proces- precision. For instance, real-time controllers typically as- sor’s Time Stamp Counter register to provide a precise and sume that important events occur at predetermined times, stable time reference, via a high-precision relative clock and the system’s calculations might be wrong if sensors are synchronization protocol. Existing clock synchronization not sampled at the assumed rate or if signals are not sent to techniques, such as the Network Time Protocol, were de- actuators at the expected times. signed for wide-area networks with large propagation de- High-precision clock synchronization can be achieved lays, but the millisecond-scale precision they offer is too either by hardware or software clocks. The clocks may coarse for local-area applications such as instrument mon- be Time-Of-Day (TOD) clocks using an oscillator that is itoring systems, high-quality digital audio systems and sen- consistent with Coordinated Universal Time or a software sor networks. Our new clock synchronization technique clock using other absolute time sources such as Internet does not require specialized hardware but instead uses the time servers or a satellite radio clock. In this paper, how- Time Stamp Counter already available in the widely-used ever, our clock synchronization method is relative only, so Intel Pentium processor. Experimental results show that does not rely on an external high-precision hardware timer. we can achieve a synchronization precision in the order of Generally, the precision of network clock synchroniza- 10 microseconds in a small-scale local area network using tion depends on two aspects: determining the one-way TSC registers, which is much higher than can be achieved transmission delay of timestamp messages and the preci- by using a computer processor’s Time-Of-Day clock. sion of a local clock. The well-known four-timestamp offset-calculation mechanism of the Network Time Proto- col (NTP) is widely used by synchronization protocols to 1 Introduction improve the accuracy of estimated one-way transmission delays. Networks using NTP-like synchronization mech- With the emergence of new time-critical distributed com- anisms typically provide precision of the order of one mil- puting paradigms, such as distributed control converters, lisecond [12]. In our work, the adopted timestamp mecha- human-robot physical interaction and ad hoc networks com- nism is derived from NTP. posed of large numbers of disposable sensors, clock syn- One of keys to achieving high-precision relative clock chronization is again becoming a critical issue in distributed synchronization is the use of high accuracy local clocks computing environments [1, 3, 6, 9, 14, 15]. In a typical because the goal of the synchronization is to correct local business application network, the only synchronization re- clocks accurately before they drift out of acceptable range. quirement is the need to keep clocks in a Wide-Area Net- Unfortunately, Time-Of-Day clocks rely on potentially un- work (WAN) loosely synchronized with an absolute Time- stable crystal oscillators [9]. Compared with an oscillator, Of-Day reference. By contrast, distributed control systems however, we note that a computer processor’s Time Stamp require strong, relative clock consistency in a Local-Area Counter (TSC) register has several advantages for achiev- Network (LAN) [4]. ing high precision relative clock synchronization. The TSC Relative clock synchronization refers to the mechanisms register counts processor cycles and is normally used for and protocols used to maintain mutually-consistent clocks measuring short time intervals. Its time resolution is much in a coordinated network of computers. That is, relative higher than an oscillator’s. For example, for a processor clock synchronization focuses on the time difference be- running at a frequency of 800MHz, its TSC register can pro- tween clocks, not the offset of each clock against absolute vide a time resolution of 1.25 nanoseconds. Also, the time time (sometimes called ‘wall clock time’). The accuracy re- required to read from a TSC register is far less than that of 978-0-7695-3139-7/08 $25.00 © 2008 IEEE 69 DOI 10.1109/ICECCS.2008.39 reading from the computer’s TOD clock. Finally, a com- puter’s Central Processing Unit (CPU) frequency is usually Table 1. Test configuration more stable than that of the oscillator used to drive the TOD Hardware Intel Pentium III 800MHz clock. 256M/512M SDRAM Memory In this paper we present a high-precision relative clock Operating systems Fedora 3/Linux kernel 2.6.12.1 synchronization protocol with a master-slave synchroniza- Network adapter Network adapter 3COM 3c59x tion structure. The protocol is intended to support high- Fast Ethernet Card precision real-time calculations in Local-Area Networks, Ethernet switch Baystack 303 Ethernet Switch rather than the usual low-precision timestamping mecha- nism supported in Internet applications. Since we are in- processors contain a timer that operates at the clock cycle terested in intra-network clock consistency, rather synchro- level. The timer is a special register that gets incremented nizing clocks against an absolute time standard, the pro- every single clock cycle. Although there is no uniform, tocol does not require a high-precision time server whose platform-independent interface by which programmers can precision has to be guaranteed by additional hardware make use of these counters, it is easy to create a program in- clocks [3, 12]. The protocol is suitable for use with net- terface for any specific machine with just a small amount of works of processors that provide access to a TSC register. assembly code. The Time Stamp Counter is a cycle counter Thanks to the stability of the TSC register, the precision used in the P6 microarchitecture (the PentiumPro and its achieved by our protocol can be much greater than that of successors). It is a 64-bit unsigned number, which counts synchronization methods using Time-Of-Day functions, as the number of cycles since the CPU was powered up or last we demonstrate by experiments performed under a simple reset. Although the CPU chip’s frequency is affected by ex- synchronization structure. ternal factors, such as age and operating temperature, higher CPU speeds allow finer time resolution of this counter. 2 TOD clock and TSC clock characterization The current “time” of a TSC clock tt, relative to the CPU’s last start-up or reset, can be calculated easily from This section introduces the characteristics of both Time- the value of the TSC register T and the measured CPU fre- Of-Day and Time Stamp Counter clocks, and compares the quency fm as follows. performance of timing measurement using a TOD clock and T tt = a TSC-based clock. Also the effects of clock skew are mea- f (1) sured using TSC-based clocks. m Maintaining a TOD clock requires the computer to gen- The value of the TSC register can be read by an ‘rdtsc’ in- erate a clock-tick signal at a predetermined rate. The source struction. Unfortunately in non-real-time operating system, of a computer’s system timer is a basic uncompensated it is hard to measure the CPU’s frequency exactly without quartz crystal oscillator whose frequency is 14.31818 MHz. an external high-precision timer, because of effects such as Dividing the basic frequency by 12 gives 1.19318 MHz context switching, caching and branch predication. For ex- (thus the period is 0.8381 microseconds), which is the clock ample, the precision achieved by a coarse CPU frequency frequency used by the computer’s system timers [2]. measurement may be as much as 1.0% [2]. Therefore, a The precision of a TOD clock is affected by not only (relative) TSC-based clock tt and an (absolute) TOD clock t the resolution of the computer’s system timer but also the may run at a different speeds. periodic timer interrupt. The Time-Of-Day function in The precision of a software clock is determined by at general-purpose computers is commonly implemented us- least two factors: the time cost of reading the hardware ing clock readings from an uncompensated quartz crystal clock and the hardware clock’s intrinsic frequency error. oscillator and counter, which delivers a pulse train with The time required to read a hardware clock can be affected a period ranging from 10 to 1 milliseconds. Each pulse by factors such as the scheduling characteristics of the oper- causes a timer interrupt, which increments a software logi- ating system and the CPU’s computational capabilities. The cal clock variable by a fixed value scaled in microseconds intrinsic frequency error of the hardware clock further de- or nanoseconds. Therefore, the precision of a TOD clock teriorates the precision of the derived software clock, and may be greater than 0.8381 microseconds, even as much accumulated errors cause clock drift.