Clock Synchronization Clock Synchronization Part 2, Chapter 5 Roger Wattenhofer ETH Zurich – Distributed Computing – www.disco.ethz.ch 5/1 5/2 Overview TexPoint fonts used in EMF. Motivation Read the TexPoint manual before you delete this box.: AAAA A • Logical Time (“happened-before”) • Motivation • Determine the order of events in a distributed system • Real World Clock Sources, Hardware and Applications • Synchronize resources • Clock Synchronization in Distributed Systems • Theory of Clock Synchronization • Physical Time • Protocol: PulseSync • Timestamp events (email, sensor data, file access times etc.) • Synchronize audio and video streams • Measure signal propagation delays (Localization) • Wireless (TDMA, duty cycling) • Digital control systems (ESP, airplane autopilot etc.) 5/3 5/4 Properties of Clock Synchronization Algorithms World Time (UTC) • External vs. internal synchronization • Atomic Clock – External sync: Nodes synchronize with an external clock source (UTC) – UTC: Coordinated Universal Time – Internal sync: Nodes synchronize to a common time – SI definition 1s := 9192631770 oscillation cycles of the caesium-133 atom – to a leader, to an averaged time, ... – Clocks excite these atoms to oscillate and count the cycles – Almost no drift (about 1s in 10 Million years) • One-shot vs. continuous synchronization – Getting smaller and more energy efficient! – Periodic synchronization required to compensate clock drift • Online vs. offline time information – Offline: Can reconstruct time of an event when needed • Global vs. local synchronization (explained later) • Accuracy vs. convergence time, Byzantine nodes, … 5/5 5/6 Atomic Clocks vs. Length of a Day Access to UTC • Radio Clock Signal – Clock signal from a reference source (atomic clock) is transmitted over a long wave radio signal – DCF77 station near Frankfurt, Germany transmits at 77.5 kHz with a transmission range of up to 2000 km – Accuracy limited by the propagation delay of the signal, Frankfurt-Zurich is about 1ms – Special antenna/receiver hardware required 5/7 5/8 What is UTC, really? Comparing (and Averaging) • International Atomic Time (TAI) – About 200 atomic clocks – About 50 national laboratories 푑퐴 푑퐵 – Reduce clock skew by comparing and averaging – UTC = TAI + UTC leap seconds (irregular rotation of earth) • GPS – USNO Time Station A Station B – USNO vs. TAI difference is a few nanoseconds 푡Δ퐴 = 푡퐴 − (푡푆푉+푑퐴) 푡Δ퐵 = 푡퐵 − (푡푆푉+푑퐵) 푡Δ = 푡Δ퐵 − 푡Δ퐴 = 푡퐵 − 푡푆푉 + 푑퐵 − 푡퐴 + 푡푆푉 + 푑퐴 = 푡퐵 − 푡퐴 + 푑퐴 − 푑퐵 5/9 5/10 Global Positioning System (GPS) GPS Localization 풔 • Satellites continuously transmit own position and time code Assuming that time of ퟐ 풔ퟏ • Line of sight between satellite and receiver required GPS satellites is correctly 푡 + 푡12 − 푡2 synchronized… 풔ퟒ • Special antenna/receiver hardware required 풔ퟑ • Time of flight of GPS signals varies between 64 and 89ms 푡 − 푡1 푡 + 푡 − t • Positioning in space and time! 14 4 푡 + 푡13 − 푡3 • What is more accurate, 풑 GPS or Radio Clock Signal? 푡 푡12 푡13 푡14 푡1 푡4 푡2 푡3 푡푆푉 푟1 푟2 푟3 푟4 푡푅 5/11 5/12 GPS Localization Keeping GPS Satellites synchronized 풔ퟐ 풔 ퟏ 풔ퟏ − 풑 = 푡 − 푡1 푐 푡 + 푡12 − 푡2 풔ퟐ − 풑 = 푡 + 푡 − 푡 풔ퟒ 푐 12 2 풔ퟑ 풔ퟑ − 풑 푡 − 푡1 = 푡 + 푡13 − 푡3 푐 푡 + 푡14 − t4 ⋮ 푡 + 푡13 − 푡3 ⋮ 풔풏 − 풑 = 푡 + 푡 − 푡 푐 1푛 푛 풑 푐 = speed of light Find least squares solution in 푡 and 풑 5/13 5/14 Alternative (Silly) Clock Sources Clock Devices in Computers • AC power lines • Real Time Clock (IBM PC) – Use the magnetic field radiating from electric AC power lines • Battery backed up – AC power line oscillations are extremely stable • 32.768 kHz oscillator + Counter (drift about 10 ppm, ppm = parts per million) • Get value via interrupt system – Power efficient, consumes only 58 μW – Single communication round required to correct phase offset after initialization • HPET (High Precision Event Timer) • Oscillator: 10 Mhz … 100 Mhz • Sunlight • Up to 10 ns resolution! – Using a light sensor to measure the length of a day • Schedule threads – Offline algorithm for reconstructing global • Smooth media playback timestamps by correlating annual solar patterns (no communication required) • Usually inside Southbridge 5/15 5/16 Clock Drift Clock Synchronization in Computer Networks • Clock drift: random deviation from the nominal rate dependent on power supply, • Network Time Protocol (NTP) temperature, etc. • Clock sync via Internet/Network (UDP) rate • Publicly available NTP Servers (UTC) 1+² 1 • You can also run your own server! ² 1- t • E.g. TinyNodes have a maximum drift of 30-50 ppm (parts per million) This is a drift of up to 50μs per second or 0.18s per hour • Packet delay is estimated to reduce clock skew 5/17 5/18 Propagation Delay Estimation (NTP) Messages Experience Jitter in the Delay • Measuring the Round-Trip Time (RTT) . Problem: Jitter in the message delay Time accor- 푡2 ding to B 푡3 B Various sources of errors (deterministic and non-deterministic) Request Answer from A from B A 0-100 ms 0-500 ms 1-10 ms Time accor- 푡1 ding to A 푡4 SendCmd Access Transmission • Propagation delay 훿 and clock skew Θ can be calculated Reception Callback 0-100 ms 푡4 − 푡1 − (푡3 − 푡2) t 훿 = 2 . Solution: Timestamping packets at the MAC layer 푡2 − (푡1 + 훿) − (푡4 − (푡3 + 훿)) 푡2 − 푡1 + (푡3 − 푡4) → Jitter in the message delay is reduced to a few clock ticks Θ = = 2 2 5/19 5/20 Jitter Measurements Clock Synchronization in Computer Networks (PTP) • Different radio chips use different paradigms • Precision Time Protocol (PTP) is very similar to NTP – Left is a CC1000 radio chip which generates an interrupt with each byte. – Right is a CC2420 radio chip that generates a single interrupt for the packet • Commodity network adapters/routers/switches can assist in time sync by after the start frame delimiter is received. timestamping PTP packets at the MAC layer • Packet delay is only estimated on request • In wireless networks propagation • Synchronization through one packet from server to clients! can be ignored (<1¹s for 300m). • Some newer hardware (1G Intel cards, 82580) can timestamp any packet • Still there is quite some variance at the MAC layer in transmission delay because of latencies in interrupt handling • Achieving skew of about 1 microsecond (picture right). 5/21 5/22 Hardware Clock Distribution Clock Synchronization Tricks in Wireless Networks 푡2 • Synchronous digital circuits require all components to act in sync A • Reference Broadcast Synchronization (RBS) Θ Synchronizing atomic clocks S 20 20 • Sender synchronizes set of clocks B 15 20 10 20 t1 12 20 0 1 • Time-sync Protocol for Sensor Networks (TPSN) 푡2 B 9 20 푡 Network Time Protocol 3 15 20 1 2 • Estimating round trip time to sync more accurately 푡 1 A 푡 4 2 2 • The bigger the clock skew, the longer the clock period 0 • The clock signal that governs this rhythm needs to be distributed to all • Flooding Time Synchronization Protocol components such that skew and wire length is minimized (FTSP)Precision Time Protocol 5 4 • Optimize routing, insert buffers (also to improve signal) • Timestamp packets at the MAC Layer to improve 1 accuracy 7 6 2 3 5/23 5/24 Best tree for tree-based clock synchronization? Clock Synchronization Tricks (GTSP) • Finding a good tree for clock synchronization is a tough problem . Synchronize with all neighboring – Spanning tree with small (maximum or average) stretch. nodes • Broadcast periodic time beacons, • Example: Grid network, with n = m2 nodes. e.g., every 30 s • No reference node necessary 0 1 2 3 4 5 • No matter what tree you use, the maximum 4 6 7 8 9 10 . How to synchronize clocks without 5 stretch of the spanning tree will always be having a leader? at least m (just try on the grid). 1 11 12 13 14 15 • Follow the node with the 7 fastest/slowest clock? 6 • In general, finding the minimum max 16 17 18 19 20 • Idea: Go to the average clock 2 3 stretch spanning tree is a hard problem, value/rate of all neighbors (including however approximation algorithms exist. 21 22 23 24 25 node itself) 5/25 5/26 Variants of Clock Synchronization Algorithms FTSP vs. GTSP: Global Skew Tree-like Algorithms Distributed Algorithms • Network synchronization error (global skew) e.g. FTSP e.g. GTSP – Pair-wise synchronization error between any two nodes in the network FTSP (avg: 7.7 μs) GTSP (avg: 14.0 μs) Bad local All nodes consistently average skew errors to all neigbhors 5/27 5/28 FTSP vs. GTSP: Local Skew Global vs. Local Time Synchronization • Neighbor Synchronization error (local skew) • Common time is essential for many applications: – Pair-wise synchronization error between neighboring nodes – Assigning a timestamp to a globally sensed event (e.g. earthquake) • Synchronization error between two direct neighbors: – Precise event localization (e.g. shooter detection, multiplayer games) FTSP (avg: 15.0 μs) GTSP (avg: 2.8 μs) – TDMA-based MAC layer in wireless networks – Coordination of wake-up and sleeping times (energy efficiency) 5/29 5/30 Theory of Clock Synchronization Time Must Behave! • Given a communication network • Time (logical clocks) should not be allowed to stand still or jump 1. Each node equipped with hardware clock with drift 2. Message delays with jitter worst-case (but constant) • Let’s be more careful (and ambitious): • Logical clocks should always move forward • Goal: Synchronize Clocks (“Logical Clocks”) • Sometimes faster, sometimes slower is OK. • Both global and local synchronization! • But there should be a minimum and a maximum speed. • As close to correct time as possible! 5/31 5/32 Lteis(tti)(t:=) max (L i (t)) Formal Model Synchronization Algorithms: An Example (“Amax”) ¯ • Hardware clock Hv(t) = s[0,t] hv(¿) d¿ Clock drift ² is typically small, e.g.