Precision Time Protocol for Spectroscope Synchronization Armin Weiss, Tobias Kammacher Adviser: Prof. Hans-Joachim Gelke Tuesday 17th February, 2015 Abstract This project was conducted by the Institue of Embedded Systems (InES) at Zurich Uni- versity of Applied Sciences (ZHAW) in collaboration with Zurich Instruments AG (ZI) and the Commission for Technology and Innovation. The aim was to implement a Ethernet based time synchronization between multiple Etzel, which are advanced digital lock-in amplifiers from ZI, and validate and verify the performance. The requested time synchronization accu- racy is in the order of 10 ns. PTP was determined to be the most suitable time synchronization protocol because of the precision, the independence of other devices and the availability of an in-house PTP stack implementation of the protocol. The feasibility of a time synchronization solution using PTP was proven by evaluating multiple simulations and implementing the available stack on two Apalis evaluation hardware boards. A concept was devised for integrating the time syn- chronization functionality into existing Etzel devices and implemented with the help of ZI. Testing was conducted on the underlying implementation on Apalis boards as well as on the final system with multiple Etzels in order to verify the performance requirements. During the implementation of the PTP stack, adjustments had to be made to the Ethernet controller driver. The PTP stack was adapted to work on a Linux with 3.1 kernel and the integrated interfaces for the PTP clock as well as the hardware timestamping. Hardware limitations (routing of pins) led to two variants for pin assignment. The accuracy of the time synchronization is dependent on environmental conditions and configuration of the PTP ap- plication. Time synchronization measurements between two Etzels show a standard deviation between the timestamp registers under best measurement conditions of 12.1 ns and a mean value of -1.7 ns. The performance of the system was found to satisfy the needs of Zurich Instruments under the condition that the Ethernet connection is realized in a direct (Peer-to-Peer) manner or via a PTP-capable switch. Contents 1 Introduction 1 1.1 Overview . .1 1.2 Goal . .2 1.2.1 Implementation . .3 1.2.2 Testing . .3 2 Implementation Concept 4 2.1 Time Synchronization Protocols . .4 2.1.1 Network Time Protocol (NTP) and Simple Network Time Protocol (SNTP) . .4 2.1.2 Precision Time Protocol (PTP) . .5 2.2 PTP Stack . .8 2.2.1 Choice of PTP Implementation . .8 2.2.2 Proprietary PTP Stack of Institute of Embedded Systems (InES) . .9 2.3 Integration into Etzel . 10 2.3.1 Basic Idea . 11 2.3.2 Synchronization of the Field Programmable Gate Array (FPGA) . 12 2.3.3 FPGA Implementation Concept . 14 2.3.4 Simulation . 18 2.3.5 System Modeling . 28 3 Implementation 35 3.1 Project overview . 35 3.2 PTP Stack Implementation on Apalis T30 . 35 3.2.1 Overview . 35 3.2.2 PTP Application . 36 3.2.3 Activation of the HW Timestamping . 37 3.2.4 Timestamping of PTP Frames . 39 3.2.5 Generation of PPS . 41 3.2.6 PTP Clock Drift Control . 43 3.2.7 PTP Clock Offset Correction . 44 3.2.8 Problems during the Implementation . 44 3.2.9 Precision of the PTP Synchronization . 47 3.3 Integration into Etzel . 48 3.3.1 FPGA Interface . 48 3.3.2 PPS Activation . 49 3.3.3 FPGA PPS Generation . 49 3.3.4 ZI Server Integration Problem . 49 4 Measurement Concept 51 4.1 Performance Key Figures . 51 4.1.1 Base PTP Implementation (Unit) . 52 4.1.2 Integrated Measurement Setup (System) . 52 4.2 Requirements . 53 4.3 Implementation Testpoints . 53 4.3.1 Pulse-Per-Second (for Unit Tests) . 53 4.3.2 Pulse-Per-Second (for System Tests) . 55 4.3.3 Timestamp of Measurement Event (for System Tests) . 55 5 Measurement Conditions 57 5.1 Network Topologies . 57 5.2 Software Settings . 58 5.3 Oscillator Stability . 59 5.4 Environmental Conditions . 60 6 Measurement Plan 62 6.1 Schedule . 62 6.1.1 Unit Tests and Performance Measurement . 63 6.1.2 System Tests Etzel . 64 7 Results 66 7.1 Unit Tests . 66 7.1.1 U0 - Time-to-Lock . 66 7.1.2 U1 - Time Offset (PPS) . 68 7.1.3 U2 - Time Offset (PPS) Long-Term . 69 7.1.4 U3 - Bandwidth for Synchronization Packets vs. Accuracy . 70 7.1.5 U4 - Network Load vs. Accuracy . 72 7.1.6 U5 - CPU Load vs. Accuracy . 73 7.1.7 U6 - Temperature vs. Accuracy . 74 7.1.8 U7 - Cable Length vs. Accuracy . 76 7.1.9 U8 - More than Two Devices (with and without a PTP-Capable Switch) 77 7.2 System Tests . 79 7.2.1 S0 - Etzel Time Synchronization . 79 7.2.2 S1 - Verification of Time Increments . 83 7.2.3 S2 - Synchronized Measurement Events . 84 8 Conclusion 91 8.1 Implementation . 91 8.1.1 Achieved Goals . 91 8.1.2 Improvements on the Implementation . 91 8.1.3 Further Development on the Implementation . 92 8.2 Testing . 92 8.2.1 Achieved Goals . 92 8.2.2 Necessary Basic Conditions . 93 8.2.3 Discovered Limitations . 94 8.2.4 Future Measurements . 94 8.2.5 Further Development Opportunities . 94 Appendix Implementation 95 A.1 Configuration of the Hardware Timestamping . 96 A.2 FPGA Registers . 97 Appendix Testing 99 B.1 Measurement Plan for Unit Tests . 99 B.1.1 U0 - Time-to-Lock . 100 B.1.2 U1 - Time Offset (PPS) . 103 B.1.3 U2 - Time Offset (PPS) long-term . 104 B.1.4 U3 - Bandwidth for Synchronization Packets vs. Accuracy . 105 B.1.5 U4 - Network Load vs. Accuracy . 107 B.1.6 U5 - CPU Load vs. Accuracy . 109 B.1.7 U6 - Temperature vs. Accuracy . 110 B.1.8 U7 - Cable Length vs. Accuracy . 111 B.1.9 U8 - More Than Two Devices (With and Without a PTP-Capable Switch)112 B.2 Measurement Plan for System Tests . 114 B.2.1 S0 - Etzel Time Synchronization . 116 B.2.2 S1 - Verification of Time Increments . 119 B.2.3 S2 - Synchronized Measurement Events . 120 B.2.4 S3 - Long-Term Consistency of Time Increments . 125 B.2.5 S4 - Long-Term Evaluation of Synchronized Measurement Events . 126 B.3 Configuration of Measurement Devices . 127 B.3.1 Oscilloscope . 127 B.3.2 Climatic Chamber . 128 B.4 Hardware Specifications . 128 B.4.1 Etzel FPGA Crystal Oscillator . 128 B.4.2 Ethernet Controller Crystal Oscillator . 128 B.4.3 Non PTP-Capable Ethernet Switch . 129 B.4.4 PTP-Capable Ethernet Switch . 129 B.5 Software Tools . 129 Bibliography 130 List of Tables 132 List of Figures 134 Glossary ADC Analog Digital Converter API Application Programming Interface BC Boundary Clock BMC Best Master Clock BSP Board Support Package CDF Cumulative Distribution Function CPU Central Processing Unit CTI Commission for Technology and Innovation DUT Device Under Test E2E End-to-End FPGA Field Programmable Gate Array GPIO General Purpose Input Output GPS Global Positioning System HF High Frequency IC Integrated Circuit InES Institute of Embedded Systems IP Internet Protocol L4T Linux for Tegra LED Light Emitting Diode LAN Local Area Network LXDE Lightweight X11 Desktop Environment MAC Media Access Control (layer) nc NetCat NMEA National Marine Electronics Association NTP Network Time Protocol OS Operating System P2P Peer-to-Peer PCB Printed Circuit Board PCHIP Piecewise Cubic Interpolating Polynomial PCIe Peripheral Component Interconnect Express PHY Physical (layer) PI Proportional-Integral PLL Phase Locked Loop ppm parts per million PPS Pulse per Second PTP Precision Time Protocol PTPD Precision Time Protocol Daemon pv Pipe Viewer RNDIS Remote Network Driver Interface Specification RMSE Root Mean Square Error SDP Software Defined Pin SNR Signal-to-Noise-Ratio SNTP Simple Network Time Protocol TAI Temps Atomique International TC Transparent Clock TCP Transmission Control Protocol TSU Time Stamp Unit TC Transparent Clock UDP User Datagram Protocol UTC Universal Time, Coordinated VHDL Very High Speed Integrated Hardware Description Language WR White Rabbit ZHAW Zurich University of Applied Sciences ZI Zurich Instruments AG Chapter 1 Introduction This chapter provides an overview of the project and states the required goals. 1.1 Overview Current generation measurement and testing equipment most commonly provides the ability to connect to a Local Area Network (LAN) for remote control and data retrieval purposes..