On Enhanced Clock Synchronization Performance Through Dedicated Ethernet Hardware Support
Total Page：16
File Type：pdf, Size：1020Kb
Die approbierte Originalversion dieser Dissertation ist an der Hauptbibliothek der Technischen Universität Wien aufgestellt (http://www.ub.tuwien.ac.at). The approved original version of this thesis is available at the main library of the Vienna University of Technology (http://www.ub.tuwien.ac.at/englweb/). Dissertation On Enhanced Clock Synchronization Performance Through Dedicated Ethernet Hardware Support ausgefuhrt¨ zum Zwecke der Erlangung des akademischen Grades eines Doktors der technischen Wissenschaften unter der Leitung von Ao. Univ. Prof. Dipl.-Ing. Dr. Wolfgang Kastner Institut fur¨ Rechnergestutzte¨ Automation, Arbeitsbereich Automatisierungssysteme und Prof. Dr. Francisco Vasques Departement of Mechanical Engineering, University of Porto, Portugal eingereicht an der Technischen Universit¨at Wien Fakult¨at fur¨ Informatik durchgefuhrt¨ am Institut fur¨ Integrierte Sensorsysteme der Osterreichische¨ Akademie der Wissenschaften unter der Leitung von Dipl.-Ing. Dr. Thilo Sauter von Dipl.-Ing. Patrick Loschmidt Wienergasse 114-116/2/5 2380 Perchtoldsdorf Matr.-Nr. 9526652 Wien, im Dezember 2010 Patrick Loschmidt Contents 1 Introduction8 1.1 The Necessity of Clock Synchronization in Distributed Systems......8 1.2 Basic Terms................................... 12 1.3 Clock Synchronization in Packet-Oriented Networks............ 14 1.3.1 Transmission Delay Boundaries.................... 15 1.3.2 Packet Relaying Effects........................ 16 1.4 Hardware Support for High-Precision Clock Synchronization........ 17 1.4.1 Required Elements........................... 18 1.4.2 Transfer of Timestamps and the OSI Reference Model....... 21 1.5 Contribution of the Thesis........................... 23 2 Motivation of the Problem 25 2.1 Remote Metering using Power Line...................... 25 2.1.1 Access Network............................. 26 2.1.2 Power Line Network.......................... 27 2.2 Wireless Position Determination....................... 28 2.2.1 Method for Localisation........................ 29 2.2.2 Implementation Issues......................... 31 2.3 Control System for Particle Accelerators................... 33 2.3.1 White Rabbit Communication Infrastructure............ 34 3 State of the Art and Related Work 41 3.1 Synchronous Network Approaches...................... 42 3.1.1 Synchronous Ethernet......................... 43 3.1.2 Delay Compensation Techniques................... 45 3.2 Protocols.................................... 46 3.2.1 Network Time Protocol (NTP).................... 47 3.2.2 IEEE 1588............................... 51 3.3 Real-Time Networks.............................. 56 3.3.1 PROFINET............................... 56 3.3.2 TTEthernet............................... 57 3.3.3 EtherCAT................................ 59 3.3.4 Ethernet POWERLINK........................ 60 3.3.5 SERCOS-III.............................. 61 3.3.6 LAN eXtensions for Instrumentation (LXI)............. 62 Contents 3/185 3.4 Hardware Elements............................... 63 3.4.1 National Instruments Synchronization Modules........... 63 3.4.2 National Semiconductor PHY..................... 64 3.4.3 Intel Gigabit Ethernet Controller................... 64 3.5 Implications................................... 65 4 System Characterisation 66 4.1 System Immanent Jitter Sources....................... 66 4.2 Oscillators.................................... 71 4.2.1 Frequency Domain Characterisation................. 72 4.2.2 Time Domain Characterisation.................... 75 4.3 Ethernet Transmission Standards....................... 76 4.3.1 10 Mbit/s Ethernet (10 Base-T).................... 77 4.3.2 100 Mbit/s Ethernet (100 Base-T)................... 78 4.3.3 1 Gbit/s Ethernet (1000 Base-T)................... 79 4.4 Prerequisites for Hardware Support...................... 81 4.4.1 Media Dependency and Event Access................ 81 4.4.2 Timestamp Transportation Issues................... 82 4.4.3 System Clock Structure........................ 85 5 Evaluation Architecture for High-Performance Synchronization 86 5.1 Applying System Requirements........................ 86 5.1.1 Clock Resolution............................ 87 5.1.2 Performance.............................. 88 5.1.3 Software / Hardware Partitioning................... 89 5.2 Basic Prototype and Proof of Concept.................... 91 5.2.1 Timestamp FIFO............................ 94 5.2.2 Accuracy Counters and Amortisation................ 95 5.2.3 Event Timestamp Registers...................... 96 5.2.4 Trigger Functionality.......................... 96 5.2.5 AHB Register Interface........................ 97 5.3 Event-Based, High-Accuracy Cell....................... 97 5.3.1 Prescaler................................ 98 5.3.2 Adder-Based Clock Enhancements.................. 98 5.3.3 Tick-Based Architecture........................ 100 5.4 Media Independent Interface Scanner..................... 101 5.4.1 Functionality.............................. 101 5.4.2 Precision................................ 103 5.4.3 One-Step Clock............................. 109 6 Synchronization System 111 6.1 Hardware.................................... 111 6.1.1 Network Interface Card........................ 111 6.1.2 Time-Aware Switch.......................... 114 Contents 4/185 6.1.3 Support in Embedded Processors................... 118 6.2 Software..................................... 120 6.2.1 Driver and Software Structure.................... 121 6.2.2 NTP Daemon Reference Clock Driver................ 129 6.2.3 PTP Stack............................... 131 6.2.4 GPS Stack............................... 136 7 Validation 138 7.1 Hardware Measurements............................ 138 7.1.1 Applicability of Phase Estimation to Enhance Precision...... 138 7.1.2 Synchronization Performance..................... 141 7.2 Simulation.................................... 144 7.2.1 Oscillator Model Concept....................... 146 7.2.2 Discrete Event Simulation (DES) Model Description........ 150 7.2.3 DES Model Implementation and Evaluation............. 151 8 Outlook and Conclusion 154 8.1 Layer 2 Implementations............................ 155 8.2 Embedded Frequency Carrier for Wired Links................ 156 8.3 Wireless Links................................. 160 8.4 Final Conclusions................................ 162 Acronyms 165 Abstract The evolvement of Ethernet in factory automation and test & measurement applications generated the need for high-accuracy clock synchronization. While factory communication can normally fulfil application requirements based on an average accuracy in the range of microseconds, test & measurement and several other applications (e. g. wireless position determination) seek for ultimate performance, meaning nanoseconds at most. This new demand, together with the issue of simplified node configuration, prepared the ground for a new protocol allowing for standardised synchronization in new dimensions. The Precision Time Protocol standardised as IEEE 1588 addresses the topic of net- worked measurement and control systems by giving a set of rules for running high accuracy clock synchronization in a self-configuring topology. Due to the fact that not only the protocol itself is defined, but also the message timestamp point for several network technologies, it allows the implementation with hardware support. The latter enables the possibility to gain timestamps without jitter introduced by varying processing time in (software implemented) higher network layers. The main goal of the present work is to analyse and propose a way to get from common PTP accuracy, which is about 100 ns, to systems being able to deliver sub-nanosecond performance. Although state-of-the-art devices for hardware timestamping PTP messages use the interface between the physical and data link layer, jitter sources remain, which can hinder high accuracy. The aimed hardware support for highly accurate synchronization over Ethernet is as well targeted at the mentioned interface. This is argued by the fact that most network interface implementations use hardware blocks at this level and software routines for the higher functionalities. If the { for network synchronization indispensable { timestamps are drawn there, the effects of timely uncertainties of the software stack can be removed. Nevertheless, it is possible to operate at bit level. An extensive analysis of existing disturbance factors on the synchronization accuracy and their (statistical) behaviour allows to design appropriate architectures and counter measures to further reduce the effects on the precision of drawn timestamps at this level. The examinations are partly done by measuring and modelling the disturbances and completed by mathematical expressions built from the knowledge of the type of the internal jitter source. In order to show a complete picture of all relevant factors, not only the physical influence, but also structural issues are covered. These include the interaction of software with the time-keeping hardware, the analysis of external application requirements and the actual implementation of clock synchronizing hardware cells. To round network controlled systems off, the analysis describes design issues for synchronization systems and their relation to the actual hardware implementation. Zusammenfassung Der verst¨arkte Einsatz von Ethernet in der Industrieautomatisierung und die zuneh- mende Verbreitung in Test- und Meßsystemen bewirkte den Bedarf an hochgenauer Uhrensynchronisation auf diesem Medium. W¨ahrend