Clock Synchronization Clock Synchronization Part 2, Chapter 5

Roger Wattenhofer

ETH Zurich – Distributed Computing – www.disco.ethz.ch 5/1 5/2

Overview TexPoint fonts used in EMF. Motivation Read the TexPoint manual before you delete this box.: AAAA A • Logical Time (“happened-before”) • Motivation • Determine the order of events in a distributed system • Real World Clock Sources, Hardware and Applications • Synchronize resources • Clock Synchronization in Distributed Systems • Theory of Clock Synchronization • Physical Time • Protocol: PulseSync • Timestamp events (email, sensor data, file access times etc.) • Synchronize audio and video streams • Measure signal propagation delays (Localization)

• Wireless (TDMA, duty cycling) • Digital control systems (ESP, airplane autopilot etc.)

5/3 5/4 Properties of Clock Synchronization Algorithms World Time (UTC)

• External vs. internal synchronization • Atomic Clock – External sync: Nodes synchronize with an external clock source (UTC) – UTC: Coordinated Universal Time – Internal sync: Nodes synchronize to a common time – SI definition 1s := 9192631770 oscillation cycles of the caesium-133 atom – to a leader, to an averaged time, ... – Clocks excite these atoms to oscillate and count the cycles – Almost no drift (about 1s in 10 Million years) • One-shot vs. continuous synchronization – Getting smaller and more energy efficient! – Periodic synchronization required to compensate clock drift

• Online vs. offline time information – Offline: Can reconstruct time of an event when needed

• Global vs. local synchronization (explained later)

• Accuracy vs. convergence time, Byzantine nodes, …

5/5 5/6

Atomic Clocks vs. Length of a Day Access to UTC

• Radio Clock Signal – Clock signal from a reference source (atomic clock) is transmitted over a long wave radio signal – DCF77 station near Frankfurt, Germany transmits at 77.5 kHz with a transmission range of up to 2000 km – Accuracy limited by the propagation delay of the signal, Frankfurt-Zurich is about 1ms – Special antenna/receiver hardware required

5/7 5/8 What is UTC, really? Comparing (and Averaging)

• International Atomic Time (TAI) – About 200 atomic clocks – About 50 national laboratories 푑퐴 푑퐵 – Reduce clock skew by comparing and averaging – UTC = TAI + UTC leap seconds (irregular rotation of earth)

• GPS – USNO Time Station A Station B – USNO vs. TAI difference is a few nanoseconds 푡Δ퐴 = 푡퐴 − (푡푆푉+푑퐴) 푡Δ퐵 = 푡퐵 − (푡푆푉+푑퐵)

푡Δ = 푡Δ퐵 − 푡Δ퐴 = 푡퐵 − 푡푆푉 + 푑퐵 − 푡퐴 + 푡푆푉 + 푑퐴 = 푡퐵 − 푡퐴 + 푑퐴 − 푑퐵

5/9 5/10

Global Positioning System (GPS) GPS Localization

풔 • Satellites continuously transmit own position and time code Assuming that time of ퟐ 풔ퟏ • Line of sight between satellite and receiver required GPS satellites is correctly 푡 + 푡12 − 푡2 synchronized… 풔ퟒ • Special antenna/receiver hardware required 풔ퟑ • Time of flight of GPS signals varies between 64 and 89ms 푡 − 푡1 푡 + 푡 − t • Positioning in space and time! 14 4 푡 + 푡13 − 푡3

• What is more accurate, 풑 GPS or Radio Clock Signal?

푡

푡12 푡13 푡14

푡1 푡4 푡2 푡3 푡푆푉 푟1 푟2 푟3 푟4 푡푅

5/11 5/12 GPS Localization Keeping GPS Satellites synchronized

풔 ퟐ 풔ퟏ 풔ퟏ − 풑 = 푡 − 푡1 푐 푡 + 푡12 − 푡2 풔ퟐ − 풑 = 푡 + 푡 − 푡 풔ퟒ 푐 12 2 풔ퟑ 풔ퟑ − 풑 푡 − 푡1 = 푡 + 푡13 − 푡3 푐 푡 + 푡14 − t4 ⋮ 푡 + 푡13 − 푡3 ⋮ 풔풏 − 풑 = 푡 + 푡 − 푡 푐 1푛 푛 풑

푐 = speed of light

Find least squares solution in 푡 and 풑

5/13 5/14

Alternative (Silly) Clock Sources Clock Devices in Computers

• AC power lines • Real Time Clock (IBM PC) – Use the magnetic field radiating from electric AC power lines • Battery backed up – AC power line oscillations are extremely stable • 32.768 kHz oscillator + Counter (drift about 10 ppm, ppm = parts per million) • Get value via interrupt system – Power efficient, consumes only 58 μW – Single communication round required to correct phase offset after initialization • HPET (High Precision Event Timer)

• Oscillator: 10 Mhz … 100 Mhz • Sunlight • Up to 10 ns resolution! – Using a light sensor to measure the length of a day • Schedule threads – Offline algorithm for reconstructing global • Smooth media playback timestamps by correlating annual solar patterns (no communication required) • Usually inside Southbridge

5/15 5/16 Clock Drift Clock Synchronization in Computer Networks

• Clock drift: random deviation from the nominal rate dependent on power supply, • Network Time Protocol (NTP) temperature, etc. • Clock sync via Internet/Network (UDP) rate • Publicly available NTP Servers (UTC) 1+² 1 • You can also run your own server! ² 1- t

• E.g. TinyNodes have a maximum drift of 30-50 ppm (parts per million)

This is a drift of up to 50μs per second or 0.18s per hour

• Packet delay is estimated to reduce clock skew

5/17 5/18

Propagation Delay Estimation (NTP) Messages Experience Jitter in the Delay

• Measuring the Round-Trip Time (RTT) . Problem: Jitter in the message delay Time accor- 푡2 ding to B 푡3 B Various sources of errors (deterministic and non-deterministic) Request Answer from A from B A 0-100 ms 0-500 ms 1-10 ms Time accor- 푡1 ding to A 푡4 SendCmd Access Transmission

• Propagation delay 훿 and clock skew Θ can be calculated Reception Callback 0-100 ms 푡4 − 푡1 − (푡3 − 푡2) t 훿 = 2 . Solution: Timestamping packets at the MAC layer 푡2 − (푡1 + 훿) − (푡4 − (푡3 + 훿)) 푡2 − 푡1 + (푡3 − 푡4) Θ = = → Jitter in the message delay is reduced to a few clock ticks 2 2

5/19 5/20 Jitter Measurements Clock Synchronization in Computer Networks (PTP)

• Different radio chips use different paradigms • Precision Time Protocol (PTP) is very similar to NTP – Left is a CC1000 radio chip which generates an interrupt with each byte. – Right is a CC2420 radio chip that generates a single interrupt for the packet • Commodity network adapters/routers/switches can assist in time sync by after the start frame delimiter is received. timestamping PTP packets at the MAC layer

• Packet delay is only estimated on request

• In wireless networks propagation • Synchronization through one packet from server to clients! can be ignored (<1¹s for 300m). • Some newer hardware (1G Intel cards, 82580) can timestamp any packet • Still there is quite some variance at the MAC layer in transmission delay because of latencies in interrupt handling • Achieving skew of about 1 microsecond (picture right).

5/21 5/22

Hardware Clock Distribution Clock Synchronization Tricks in Wireless Networks

푡2

• Synchronous digital circuits require all components to act in sync A • Reference Broadcast Synchronization (RBS)  Θ Synchronizing atomic clocks S 20 20 • Sender synchronizes set of clocks B 15 20 t1 10 20

12 20 0

1 • Time-sync Protocol for Sensor Networks (TPSN)  푡2 B 9 20 푡 Network Time Protocol 3 15 20 1 2 • Estimating round trip time to sync more accurately 푡 1 A 푡 4 2 2

• The bigger the clock skew, the longer the clock period 0 • The clock signal that governs this rhythm needs to be distributed to all • Flooding Time Synchronization Protocol components such that skew and wire length is minimized (FTSP)Precision Time Protocol 5 4 • Optimize routing, insert buffers (also to improve signal) • Timestamp packets at the MAC Layer to improve 1 accuracy 7 6 2 3

5/23 5/24

Best tree for tree-based clock synchronization? Clock Synchronization Tricks (GTSP)

• Finding a good tree for clock synchronization is a tough problem . Synchronize with all neighboring – Spanning tree with small (maximum or average) stretch. nodes • Broadcast periodic time beacons, • Example: Grid network, with n = m2 nodes. e.g., every 30 s • No reference node necessary 0 1 2 3 4 5

• No matter what tree you use, the maximum 4 6 7 8 9 10 . How to synchronize clocks without 5 stretch of the spanning tree will always be having a leader? at least m (just try on the grid). 1 11 12 13 14 15 • Follow the node with the 7 fastest/slowest clock? 6 • In general, finding the minimum max 16 17 18 19 20 • Idea: Go to the average clock 2 3 stretch spanning tree is a hard problem, value/rate of all neighbors (including however approximation algorithms exist. 21 22 23 24 25 node itself)

5/25 5/26

Variants of Clock Synchronization Algorithms FTSP vs. GTSP: Global Skew

Tree-like Algorithms Distributed Algorithms • Network synchronization error (global skew) e.g. FTSP e.g. GTSP – Pair-wise synchronization error between any two nodes in the network

FTSP (avg: 7.7 μs) GTSP (avg: 14.0 μs)

Bad local All nodes consistently average skew errors to all neigbhors

5/27 5/28 FTSP vs. GTSP: Local Skew Global vs. Local Time Synchronization

• Neighbor Synchronization error (local skew) • Common time is essential for many applications: – Pair-wise synchronization error between neighboring nodes – Assigning a timestamp to a globally sensed event (e.g. earthquake)

• Synchronization error between two direct neighbors: – Precise event localization (e.g. shooter detection, multiplayer games)

FTSP (avg: 15.0 μs) GTSP (avg: 2.8 μs) – TDMA-based MAC layer in wireless networks

– Coordination of wake-up and sleeping times (energy efficiency)

5/29 5/30

Theory of Clock Synchronization Time Must Behave!

• Given a communication network • Time (logical clocks) should not be allowed to stand still or jump 1. Each node equipped with hardware clock with drift 2. Message delays with jitter worst-case (but constant)

• Let’s be more careful (and ambitious): • Logical clocks should always move forward • Goal: Synchronize Clocks (“Logical Clocks”) • Sometimes faster, sometimes slower is OK. • Both global and local synchronization! • But there should be a minimum and a maximum speed. • As close to correct time as possible!

5/31 5/32 Lteis(tti)(t:=) max (L i (t))

Formal Model Synchronization Algorithms: An Example (“Amax”)

¯ • Hardware clock Hv(t) = s[0,t] hv(¿) d¿ Clock drift ² is typically small, e.g. • Question: How to update the logical clock Allow = 1 -4 with clock rate hv(t) 2 [1-²,1+²] ² ¼10 for a cheap quartz oscillator based on the messages from the neighbors? • Idea: Minimizing the skew to the fastest neighbor Logical clocks with rate less than 1 – Set the clock to the maximum clock value received from any neighbor • Logical clock Lv(∙) which increases at rate at least 1 and at most ¯ behave differently (“synchronizer”) (if larger than local clock value) – forward new values immediately Neglect fixed share of delay, • Optimum global skew of about D • Message delays 2 [0,1] normalize jitter • Poor local property

– First all messages take 1 time unit… • Employ a synchronization algorithm Time is 152 – …then we have a fast message! to update the logical clock according Hv New time is D+x to hardware clock and Fastest New time is D+x skew D! Hardware messages from Clock Time is D+x Time is D+x Time is D+x neighbors Time is 140 Time is 150 …

Clock value: Old clock value: Old clock value: Old clock value: D+x D+x-1 x+1 x Lv?

5/33 5/34

Synchronization Algorithms: Amax’ Local Skew: Overview of Results

• The problem of Amax is that the clock is always increased to the maximum Everybody‘s expectation, value five years ago („solved“) • Idea: Allow a constant slack γ between the maximum neighbor clock value Blocking All natural algorithms algorithm [Locher et al., DISC 2006] and the own clock value Lower bound of logD / loglogD [Fan & Lynch, PODC 2004] • Amax’ The algorithm sets the local clock value Li(t) to 퐿푖 푡 ≔ max⁡(퐿푖 푡 , max푗∈푁 퐿푗 푡 − 훾)⁡ 푖 1 logD √D D …

→ Worst-case clock skew between two neighboring nodes is still Θ(D)

independent of the choice of γ! Dynamic Networks! Kappa algorithm [Kuhn et al., SPAA 2009] • How can we do better? [Lenzen et al., FOCS 2008] – Adjust logical clock speeds to catch up with fastest node (i.e. no jump)? Tight lower bound [Lenzen et al., PODC 2009] – Idea: Take the clock of all neighbors into account by choosing the average value?

5/35 5/36 Enforcing Clock Skew Local Skew: Lower Bound

푙0 ℎ푣 = 1 퐿푣(푡)=x ℎ푣 = 1 + 휖 퐿푣 푡 = 푥 + 2 u 2 3 4 5 6 7 Higher v 푙0 = 퐷 clock 2 3 4 5 6 7 rates

퐿 (푡) 2 3 4 5 6 7 2 3 4 5 6 7 ℎ푤 = 1 푤 ℎ푤 = 1 퐿푤(푡)

푙0 푙0 • Add 2 skew in 2휖⁡time, messing with clock rates and messages 2 3 4 5 6 7 2 3 4 5 6 7 • Afterwards: Continue execution for 푙0 ⁡⁡time (all ℎ = 1) 4(훽−1) 푥 푙0 푙0 • Messages between two neighboring nodes may be fast in one direction  Skew reduces by at most 4  at least 4 skew remains and slow in the other, or vice versa. 휖 푙1  Consider a subpath of length 푙1 = 푙0 ⋅ 2 훽−1 ⁡with at least 4 skew

 Add 푙1 skew in 푙1 = 푙0 time  at least 3 ⋅ 푙 ⁡skew in subpath • A constant skew between neighbors may be „hidden“. 2 2휖 4(훽−1) 4 1 • Repeat this trick (+½,-¼,+½,-¼,…) log2(β−1) D times 휖 • In a path, the global skew may be in the order of D/2. Theorem: (logβ−1 D) skew between neighbors 휖 5/37 5/38

Local Skew: Upper Bound Back to Practice: Synchronizing Nodes

. Sending periodic beacon messages to synchronize nodes • Surprisingly, up to small constants, the (log D) lower bound can be (¯-1)/² matched with clock rates 2 [1,¯] (tough part, not included) • We get the following picture [Lenzen et al., PODC 2009]: Beacon interval B

max rate ¯ 1+² 1+£(²) 1+√² 100 130 2 large t 0 reference clock

local skew 1 £(log D) £(log1/² D) £(log1/² D) £(log1/² D) t=100 t=130

We can have both ... because too large t 1 smooth and accurate clock rates will amplify clocks! the clock drift ². J J jitter jitter

• In practice, we usually have 1/² ¼ 104 > D. In other words, our initial intuition of a constant local skew was not entirely wrong! 

5/39 5/40 How accurately can we synchronize two nodes? Clock Skew between two Nodes

. Message delay jitter affects clock synchronization quality . Lower Bound on the clock skew between two neighbors

^ ^ Error in the rate estimation: y r y r 0 r 0 r  Jitter in the message delay  Beacon interval  Number of beacons k ^ ^ r y(x) = ^r∙x + ∆y r

clock offset relative clock rate Synchronization error: (estimated)

x x ∆y J J ∆y J J 1 1

Beacon interval B Beacon interval B

5/41 5/42

Multi-hop Clock Synchronization Linear Regression (e.g. FTSP)

. FTSP uses linear regression to compensate for clock drift . Nodes forward their current estimate of the reference clock Jitter is amplified before it is sent to the next hop Each synchronization beacon is affected by a random jitter J

0 1 2 3 4 ... d 0 y r J1 J2 J3 J4 J5 Jd Example for k=2

^r synchronization error . Sum of the jitter grows with the square-root of the distance

stddev(J1 + J2 + J3 + J4 + J5 + ... Jd) = √d×stddev(J) y(x) = ^r∙x + ∆y Single-hop: Multi-hop: clock offset relative clock rate ∆y (estimated) x J J 1

Beacon interval B

5/43 5/44 The PulseSync Protocol The PulseSync Protocol (2)

• Send fast synchronization pulses through the network • Remove self-amplification of synchronization error  Speed-up the initialization phase  Fast flooding cannot completely eliminate amplification  Faster adaptation to changes in temperature or network topology

Beacon time B 0 y r

Example for k=2 0 1 ^r synchronization error FTSP 2 Expected time 3 = D·B/2 4 t y(x) = ^r∙x + ∆y Beacon time B clock offset 0 relative clock rate 1 ∆y (estimated) x PulseSync 2 J J Expected time 3 1 The green line is calculated using k measurement points that are = D·tpulse 4 t Beacon interval B tpulse statistically independent of the red line.

5/45 5/46

FTSP vs. PulseSync FTSP vs. PulseSync

• Global Clock Skew • Maximum synchronization error between any two nodes • Sychnronization Error vs. distance from root node

FTSP PulseSync FTSP PulseSync

Synchronization Error FTSP PulseSync

Average (t>2000s) 23.96 µs 4.44 µs Maximum (t>2000s) 249 µs 38 µs

5/47 5/48

Credits

• Approximation algorithms for minimum max stretch spanning tree, e.g. That’s all! Emek and Peleg, 2004. Questions & Comments? • More credits to come

Roger Wattenhofer

5/49 ETH Zurich – Distributed Computing – www.disco.ethz.ch 5/50

TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.: AAAA A A

A