WHITE PAPER Delivering Sub-Microsecond Accurate Time to Linux Applications Around the World Time Where It’s Needed .......................1 Accurate Time Where It’s Needed of delay, making adjusting for the delay Time Offsets, Delays And Delay While the science of timekeeping has more difficult. In fact, these delays are the Variations…No Way Around Them! .......1 fascinated many over the years, what primary source of time transfer error. In The Four Sources Of Sub-Microsecond counts most today is delivering accurate time synchronization we rely on accurately Time Error .............................................2 time to where it’s needed. If a person needs measuring timing packet transit delays to Network Path Delay Related Errors ....2 the time they may look at their phone to adjust the clocks to compute the correct time. Linux Stack Delay Related Errors ......3 Linux Clock Call Delay Errors ............3 see an accurate time that is distributed As the need for more accurate time at Clock Instability Errors ......................4 through the mobile network. If an a program increases, the impact that Solving The Sources Of Time application program running on a computer Synchronization Error ...........................5 needs the time it may execute a command variable packet delays have on the ability PTP Hardware/Software that accesses the system clock. That to correct a clock increases as well. To Combinations .......................................5 application is a consumer of the time and make matters worse, the quality of a clock Why Hardware Based PTP Slaves not a clock. Ultimately, it is the accuracy also effects how well you can synchronize Deliver The Best Timing .......................5 and precision of the time delivered to the the time. Understanding the problems and Solving Network Path Delay application program that matters most. how to compensate for them greatly helps Related Errors .......................................5 The more accurate and precise the time in delivering accurate and precise time More Packets For Better Filtering ...5 requirement is, the harder it is to deliver it. to clocks and ultimately to an application OCXO Oscillator Provides A Stable program. Time Base .......................................5 This whitepaper explores the challenges in delivering sub-microsecond accurate time Hardware Time Stamping Improves The difference between the times on to a Linux program, and it will present IEEE Time Accuracy ................................6 two clocks is known as the offset. In 1588 Precise Time Protocol (PTP) solutions Validating Synchronization Over The timekeeping we strive to keep the offset Network ..........................................6 and test data showing how to overcome below a particular value so that we can Push Time To Memory To Improve those challenges. Speed And Accuracy ...........................6 assign an accuracy value to the time on Verifying High Availability DMA Time Time Offsets, Delays and Delay the clock. Generally in synchronization Writes .............................................7 Variations…No Way Around Them! schemes one clock is more accurate than Verifying DMA Time Stamp Our objective is to deliver accurate and the other so the offset is tracked relative Accuracy .........................................7 precise time over a packet based network to the more accurate clock. Best Possible Time To Utc At to a computer, then on to the application The Application Layer ....................8 The process of setting one clock to another programs that need it. Timing packets is a matter of exchanging timing packets, Best Possible Time Around The on a network can be delayed, thereby World ..............................................9 computing the offset between a master impairing the ability to accurately transfer Conclusion, PTP Clock Cards Deliver clock and a slave clock, and making an time from one clock to another. There are The Best Possible Time Accuracy .........9 adjustment to the slave clock (called factors that cause the delays change, so clock steering when done periodically). there will also be variations in the amount Page 1 of 9 WHITE PAPER Delivering Sub-Microsecond Accurate Time to Linux Applications Around the World The time the packets take to travel from the master clock to the slave is called delay, Master Slave Time Time or path delay. Figure 1 shows the PTP packet exchanges and Equation 1 is the Timestamps fundamental offset equation the slave will known by slave use to compute the time offset and correct the time. t1 Sync Slave Time Offset= (t2-t1 )+(t3-t4) 2 t2 t1, t2 Equation 1: Determination of clock error at the slave derived from PTP message departure and arrival timestamps. The Four Sources of Sub-Microsecond t3 t1, t2, t3 Time Error Delay_Req There are four main sources of error in delivering time to an application program over a network from a master clock. t4 Asymmetric time transfer delays between the master clock and the program on the slave computer requesting the time Delay_Resp accounts for three of them. The fourth is clock instability at the slave computer t1, t2, t3, t4 (assuming the master is the more stable time source). With sub-microsecond accurate time required at an application program on a slave computer, oscillator drift and even a slight temperature change . Figure 1: Sequence diagram showing the timing event packet exchange between a PTP master clock caused by the slave computer fan can . throw the time off by several microseconds. and a PTP slave clock. Network Path Delay Related Errors In packet based networks, timing packets are exchanged between the master clock and the slave clock for the purpose of computing the time offset at the slave. If the packet exchanges were instantaneous there would be no delay and the offset Master Slave could be computed perfectly. If the packet Sync Messages Sync Messages exchange delay on the master-to-slave Network Sent on Time Delay Arrive with Varible Delays path and slave-to-master paths were identical; the offset could be easily and precisely computed since the delays would cancel each other mathematically. The Time Offset Between Clocks case where path delay is the same both ways between master and slave is called symmetric delay, and time transfer over . Figure 2: Timing messages sent from the master to the slave (and vice versa) can experience packet networks most often assumes . variable delays caused by switches in the network and application delays leading to timing errors . symmetric delay. at the slave clock. Page 2 of 9 WHITE PAPER Delivering Sub-Microsecond Accurate Time to Linux Applications Around the World Unfortunately, path delay is different Switch between master-to-slave and slave- to-master, and this difference is called Data Traffic asymmetric delay. Furthermore, not only is the delay different along the two paths, but the delay difference can also vary with each packet sent. As illustrated in Timing Traffic Figure 2, Asymmetric path delays are the main problem in accurate and precise . Figure 3: Packet queuing inside the switch where data and timing packets must be forwarded time synchronization using packet based . to the same port can introduce Packet Delay Variation (PDV) leading to timing errors. techniques over a network. In a simplified network, asymmetric delay adds a time offset error equal to one half the difference in delay times. For example, receive ports, in which case there will be Linux Stack Delay Related Errors if it took one second for the timing packet to a delay in forwarding the packets. The As timing packets exit the network they go from the master to the slave and three time the packet is delayed is called the enter the Linux stack of either the master seconds on the reverse path, the offset residence time. See Figure 3. or slave computer. While a computer may computed by the slave would be incorrect be very fast at delivering timing packets by one second. Apply this to computer Due to the variable queuing delays inside quickly to the clock steering program, any hardware that is time stamping packets the switch the timing packets exchanged delivery time variation in microseconds being exchanged every second between a between the master and slave are delayed directly adds microseconds of time master and a slave with the master having and the length of the delay is variable. synchronization error. This is because it 50 nanosecond timestamp accuracy. (That This variable delay is called Packet Delay likely introduces an asymmetric path delay. is, a PTP grandmaster with accuracy to Variation (PDV). PDV refers to the arrival Note that delays delivering a received UTC or Coordinated Universal Time—such time jitter for timing packets that are sent packet to the timing program are just as a master clock with a GPS reference.) from the master or slave to the other as critical as delays in sending timing These timing packets transit a LAN but have variable arrival times at the packets out of the computer. Differences containing switches (or worse, routers) that destination. In the world of microsecond in these stack transit times directly may add asymmetric path delay in the 10s and sub-microsecond time synchronization affect how accurately and precisely the of microseconds, which result in similar over packet networks, PDV makes it very time can be corrected, particularly in size time offset computation errors. difficult to accurately compute precise computers targeting sub-microsecond offsets and adjust the slave clock. synchronization. In other words, the Queuing and delays inside network Linux stack is another place that delay Network time protocols, such as NTP and switches are generally the culprit in packet asymmetry can occur. delays. Switches are often store-and- PTP can compensate for packet delay, forward devices. Packets received on one but only if the delay is the same in each Linux Clock Call Delay Errors port are stored temporarily while the device direction. This is why the error is referred Recall that the fundamental objective is figures out which forwarding port(s) to send to as delay asymmetry error.