Synchronization Performance of the Precision Time Protocol in Industrial

IEEE TRANSACTIONS ON INSTRUMENTATION AND MEASUREMENT, VOL. 58, NO. 6, JUNE 2009 1849 Synchronization Performance of the Precision Time Protocol in Industrial Automation Networks Ruxandra Lupas Scheiterer, Chongning Na, Student Member, IEEE, Dragan Obradovic, Member, IEEE, and Günter Steindl

Abstract—This paper analyzes the factors that affect the syn- Boundary clocks adjust their own clock to the master clock chronization performance in peer-to-peer precision time protocol and then serve as masters for the next network segment. The (PTP). We first study the influence of frequency drift in the authors of [8] introduced the transparent clock (TC) concept, absence of jitter and compare the gravity of the master drift in which intermediate bridges are treated as network compo- with that of the slave drift. Then, we study the influence of jitter under the assumption of constant frequencies and the effect of nents with known delay. By doing this, the synchronization averaging. The analytic formulas provide a theoretical ground for at the time client is not dependent on the control loop design understanding the simulation results, some of which are presented, in the intermediate bridges. The TC concept has been adopted as well as the guidelines for choosing both system and control in the new version of IEEE 1588 published in 2007 (http: parameters. //ieee1588.nist.gov/). The IEEE 1588-2008, also known as Index Terms—Frequency drift, IEEE 1588, jitter, master, pre- 1588 Version 2, was approved by the IEEE on March 27, 2008. cision time protocol (PTP) Version 2, rate-compensation factor The current state of the art is to guarantee a synchronization (RCF), slave, synchronization error, synchronization precision, precision of 1 μs for topologies with 30 consecutive slaves. transparent clock (TC). To expand this limit, it is important to study the factors that influence the quality of the synchronization process and to I. INTRODUCTION work on minimizing their effect. Industrial environments are THERNET [1], due to its cheap cabling and infrastruc- such that unpredictable and independent temperature changes E ture costs, high bandwidth, efficient switching technology, at each node are commonly encountered, causing short-term and better interoperability, has been adopted in various areas frequency drifts, unless precluded by expensive temperature- to provide the basic networking solution. Many Ethernet- compensated (TCXO) or oven-controlled (OCXO) crystal based applications require the networked clocks to be precisely oscillators. Factors that affect the synchronization quality synchronized. Typical examples include the synchronization achievable by PTP include the stability of oscillators, the res- of base stations for handover or interference cancellation [2] olution of time stamping the message, the frequency of sending in telecommunication networks, distribution of audio/video synchronization messages, and the propagation delay variation streams over Ethernet-based networks [3], and motion control caused by the jitter in the intermediate elements. Previous work in industrial Ethernet [4]. Standard network time protocol on the subject includes [10], which derives the synchronization (NTP) [5], [6] synchronization over Ethernet provides a time errors due to jitters and master frequency drift; [11] and [12], accuracy at the millisecond level, which is enough for processes where improved algorithms are proposed; [13], where the line that are not time critical. However, in many applications, delay estimation error due to clock frequency drift is obtained; base station synchronization or motion control, where only and [14], which derives the synchronization error due to slave submicrosecond-level synchronization errors are allowed (the frequency drift. so-called isochronous mode), a more accurate solution is This paper summarizes the work in [10]–[14], adding ana- needed. The precision time protocol (PTP), which was deliv- lytic results on the effect of averaging, comparative simulation ered by the IEEE 1588 standard [7] and published in 2002, con- results, and guidelines for choosing both system and control stitutes a promising Ethernet synchronization protocol in which parameters when applying PTP. messages carrying precise timing information are propagated in This paper is organized as follows. Section II introduces the network to synchronize the slave clocks to a master clock. the 1588 peer-to-peer system model and describes the PTP. An analysis of the influence of frequency drifts is presented in Section III and of jitters in Section IV. Consequences are Manuscript received March 18, 2008; revised September 26, 2008. discussed in Section V and strengthened by simulation results First published April 14, 2009; current version published May 13, 2009. in Section VI. The Associate Editor coordinating the review process for this paper was Dr. Georg Gaderer. R. L. Scheiterer, C. Na, and D. Obradovic are with Corporate Technology, Information and Communications, Siemens AG, 81730 Munich, II. DESCRIPTION OF PTP WITH TCS Germany (e-mail: [email protected]; na.chongning.ext@ siemens.com; [email protected]). Since the standard does not specify the details, this section G. Steindl is with Automation & Drives, Industrial Automation Systems, introduces the system model and notation. Fig. 1 shows a Siemens AG, 92220 Amberg, Germany (e-mail: guenter.steindl@ system with N +1 cascaded elements connected in a line siemens.com). topology. The PTP has a master/slave structure. Color versions of one or more of the figures in this paper are available online at http://ieeexplore.ieee.org. The time server, called grand(master), provides the reference Digital Object Identifier 10.1109/TIM.2009.2013655 time to the other elements, called slaves, via time-aware bridges

Authorized licensed use limited to: T U MUENCHEN. Downloaded on September 3, 2009 at 03:44 from IEEE Xplore. Restrictions apply. 1850 IEEE TRANSACTIONS ON INSTRUMENTATION AND MEASUREMENT, VOL. 58, NO. 6, JUNE 2009

Fig. 2. PTP with TCs.

ˆ i Fig. 1. System model. (a) Network topology. (b) System parameters. master counter value Mn−1 augmented by its local delay, i.e., the sum of its (skewed) line and bridge delays translated to (TCs). The master sends Sync messages every T seconds, sync master time by multiplication with RCFn: which carry the counter state of the master clock M i, stamped Mˆ i = Mˆ i + Sˆ (LDi )+Sˆ (BDi ) · RCF i at sending, and are propagated along the network. Quantities, n n−1 n n n n n (2) ˆ i i certain or not, linked with the Sync message transmitted by M0 = M = M(ti). the master at time t are labeled by superscript i.Iff is the i This equation represents the “support” for an extrapolation clock frequency, a delay D is measured by the local clock process that advances the controlled local clock until the arrival as the counter increase D · f; or as an integral thereof, if the of new information. The last ingredient necessary for the update frequency drifts. The line delay LDi is the propagation time n of (2) is the estimation of the line delay to the predecessor, between the nth slave and its uplink element, and is estimated shown on the right in Fig. 2; j indexes the line delay com- by using the so-called “line delay estimation process” of (6). putation. This process uses four time stamps: with periodicity Each time a time stamp is read, a jitter of known distribution is R, node n (the requestor) sends a request message to node incurred. The Sync message is forwarded after a bridge delay − j − BDi , which is recorded at each slave as the difference of the n 1 and records its time of departure Sn,req_out. Node n 1 n (the responder) reports the two time stamps of receiving the times stamped at reception and forwarding. A time labeled by j request message and transmitting the reply S − and Sn (respectively, M) means “measured in the local time of n 1,req_in j − j slave n (respectively, master time)”; a hat on a symbol means Sn−1,resp_out.Theresponder delay of node n 1 is RDn−1 ˆ i “estimate.” For example, Sn(BDn) means “the bridge delay in absolute time and is, in local time that affects the ith Sync message at slave n, estimated in terms j j − j i Sn−1,respD := Sn−1,resp_out Sn−1,req_in. (3) of the own local free-running clock.” At slave n, we deﬁne LBn as the sum of line delay plus bridge delay of Sync message j Node n records the time S of receiving the desired i,n i − i−1 n,resp_in i, and δLB = LBn LBn as the difference between the LB reply after a requestor delay in node n time of values that affected Sync messages i and i − 1. j j − j The time intervals measured by two different clocks will Sn,reqD := Sn,resp_in Sn,req_out. (4) be called “skewed”. To be able to add or subtract them from each other, they have to be converted to the same time basis. To be able to subtract the skewed time intervals of (3) and To this end, each slave determines its frequency offset to the (4), each element maintains an “RCF peer” estimate, i.e., the master. The rate compensation factor (RCF, which is also the frequency ratio estimate to its predecessor, that is estimated via “rate ratio” [9], [12]) is deﬁned as the frequency ratio of two j − j−1 j Sn,req_out Sn,req_out clocks. We use RCFX/Y to denote the frequency ratio between RCF = . (5) Sn/Sn−1 j j−1 S − − S − X and Y , i.e., ideally RCFX/Y = fX /fY . The rate offset n 1,req_in n 1,req_in to the master RCF can be estimated via the master counter n Then, the line delay can be estimated as estimates Mˆ in two Sync messages whose transmission times j − j · are separated by an interval T (equal to Tsync, or a multiple S S − RCFS /S − Sˆ LDj = n,reqD n 1,respD n n 1 . (6) thereof) and the local counter values S at arrival of these n n 2 messages at slave n, i.e., Usually, several successive line delay estimates are averaged. ˆ i − ˆ i−1 Mn−1 Mn−1 This concludes the description of the synchronization process, RCFn ≡ RCFM/S = . (1) n i i−1 whose accuracy is studied in this paper. In our analytic work, S − Sn n we examine each effect in isolation. However, all are simulta- Fig. 2 illustrates the pillars of time synchronization: timing neously present in the subsequent simulations. We assume zero propagation, and line delay estimation. Upon reception of Sync delay skew, i.e., direction-independent cable run time. This is message i,slaven hands an estimate of the master time at the a mild idealization since the IEC 61784-5-3 mandates stringent time of reception to the control loop of its controlled clock. In requirements for the delay skew, e.g., for PROFINET, it may addition, he acts as TC by passing on the received (estimated) not exceed 20 ns/100 m.

Authorized licensed use limited to: T U MUENCHEN. Downloaded on September 3, 2009 at 03:44 from IEEE Xplore. Restrictions apply. SCHEITERER et al.: SYNCHRONIZATION PERFORMANCE OF THE PTP IN INDUSTRIAL AUTOMATION NETWORKS 1851

i III. ANALYSIS OF ERROR PROPAGATION—PART 1: THE where Ln is the propagation time of the ith Sync message to i EFFECT OF CLOCK DRIFT IN THE ABSENCE OF JITTER slave n (which is called latency) with L0 =0, and n−1 n A. Scenario Description i i i ≈ i i ≥ Ln = LBk + LDn LDk + BDk ,n1 (14) To estimate the master counter value from (2), estimates of k=1 k=1 the line delay and RCF value need to be available, and the i and An is the “age” of the line delay computation valid for Sync RCF and the master counter estimation processes are tightly Message i, i.e., the middle of the last line delay computation intertwined. The clock frequency drifts affect these estimates interval until the arrival of the current Sync message, upper and inﬂuence the synchronization performance. In this section, bounded by the requestor interval R introduced earlier, with we quantify these effects and compare the two scenarios where − (denoting δi,n = Li − Li 1) the master frequency stays constant, while the slave frequencies L n n i,n drift, and vice versa. Tsync + δL , if no new line Ai − Ai−1 = The frequency of all the elements is constant until t0. Then, n n delay estimate (15) i,n − the drifting elements’ frequencies linearly increase. The short- Tsync + δL R, else. term linear frequency drifts are typically temperature induced, as shown by the corresponding characteristic curves. For other C. Error in RCF and Master-Counter Estimation Due to situations, our analysis is a local ﬁrst-order approximation. Master Frequency Drift for Nondrifting Slaves The frequency of an element k with constant frequency follows: For the master, (8) and (11) apply, whereas for the slaves, (7) and (9) apply. The slaves’ line delay estimates of (13) become ⎧ fk(ti)=fk(ti−1)=fk (7) j(i) R+RD ⎨⎪ · j(i) − j(i) · M · f1 LD1 RDM 4 ΔM whereas the frequency of element n whose frequency drifts Sˆ LDj(i) ≈ · j(i) − j(i),1 n n ⎪ = f1 LD1 εLD ,n=1 with slope Δn follows: ⎩ j(i) fn · LDn , else. fn(ti)=fn(ti−1)+Δn · (ti − ti−1),ti >ti−1 >t0 (8) (16)

The RCF value computed by Slave1 from the Sync message i.e., fn(ti)/fn(ti−1)=1+Δn · (ti − ti−1)/fn(ti−1). The counter value increase of each element over the time transmitted by the master at time ti is interval (t − ,t ) is the integral over the element’s frequency. M(t ) − M(t − ) i 1 i RCF i = i i 1 1 i i−1 For the constant-frequency element, this results in S (t + LD ) − S t − + LD 1 i 1 1 i 1 1 ti+ti−1 · − − · − fM (ti ti−1) Ck(ti) Ck(ti−1)=fk (ti ti−1) (9) = 2 T + LDi − LDi−1 · f 1 1 1 whereas for drifting element n, it is calculated as T fM ti − T = 2 · . (17) f i,1 ti 1 T + δLD − · Cn(ti) Cn(ti−1)= fn(t) dt Without jitter, LDi = const and δi,n =0, hence: n LD t i−1 f t − T i M i 2 Δn 2 RCF1 = . (18) = fn(ti−1)·(ti −ti−1)+ ·(ti −ti−1) . (10) f 2 1 Slave1 forwards the received Sync message to Slave2 at time Due to the assumed linearity of the frequency change, we can t + LBi and replaces the received master counter content by rewrite (10) as the product of the frequency in the middle of the i 1 its estimate at the time of transmission using (2), i.e., time interval times the length of the interval t +LBi ˆ i 1 ˆ j(i) ˆ i · i M =M(ti)+ S1 LD1 +S1 BD1 RCF1 . (19) ti + ti−1 S1out Cn(ti) − Cn(ti−1)=fn · (ti − ti−1). (11) 2 Using (16), (12), and (18), we can rewrite this as T For example, the bridge delay is approximately t +LBi fM ti − Mˆ i 1 = M(t )+ f ·LDj(i) −εj,1 +f ·BDi · 2 S out i 1 1 LD 1 1 1 f1 Sˆ BDi = f (mid Bridge Delay) · BDi . (12) n n n n f t − T i · − T − j,1 · M i 2 = M(ti)+LB1 fM ti εLD 2 f1 T + LDj(i) − LDi · f t − . (20) B. Effect of Frequency Drift on the Accuracy of the Line Delay 1 1 M i 2 Estimate in the Absence of Jitter If there is no jitter, LDj(i) = LDi , and the last term vanishes. In [13], the line delay estimate of slave n is derived to be 1 1 Assuming that the master frequency continues to increase Sˆ LDj(i) ≈ f t + Li − Ai · LDj(i) with the same slope, the true value of the master counter at this n n n i n n n time is [from (10)] j(i) j(i) R + RDn−1 t +LBi i ΔM i2 + RD · · (Δ − Δ − ) (13) M| i 1 = M(t )+f (t ) · LB + · LB . (21) n−1 4 n n 1 i M i 1 2 1

Authorized licensed use limited to: T U MUENCHEN. Downloaded on September 3, 2009 at 03:44 from IEEE Xplore. Restrictions apply. 1852 IEEE TRANSACTIONS ON INSTRUMENTATION AND MEASUREMENT, VOL. 58, NO. 6, JUNE 2009

Comparing (21) and (20), the error at this time in the TC Hence, the error at this time in the TC transmission of transmission of Slave1 is [using (8)] Slave2 is

i i t +LB +LB t +LB T i 1 2 − ˆ i 1 − − · i M − Mˆ M M = fM (ti) fM ti LB1 S2out S1out 2 T T i i f t − = fM (ti−1) − fM ti − · LB + LB ΔM · i2 j,1 · M i 2 2 1 2 + LB1 + εLD 2 f1 i i−1 − T Δ 2 LB ·LB fM ti − T + M · LBi +LBi −Δ · 2 1 +εj,1 · 2 Δ 2 fM ti 1 2 M i,1 LD M · · i i j,1 · 2 2 1+δ /T f1 = T LB1 + LB1 + εLD . BD 2 f1 ΔM · · i i i i 2 (22) = T LB1 +LB2 + LB1 +LB2 2 i−1 T LBi · LB fM ti − The RCF value computed by Slave2 from the Sync message − · 2 1 j,1 · 2 ΔM i,1 + εLD . (26) f1 transmitted by the master at time ti, which now contains 1+δBD/T Slave1’s master counter estimate, is: Using the latency defined in (14), the RCF and error expres- i i−1 Mˆ S out (ti +LB )−Mˆ S out ti−1 +LB sions generalize as follows to slave N (inductive proof without RCF i = 1 1 1 1 2 i i − i−1 i−1 error in line delay in [10]): S2 (ti +LB1 +LD2) S2 ti−1 +LB1 +LD2 − ⎧ Li 1 T i,1 i 1 T N−1 ⎪ M(ti) − M(ti−1)+fM ti − · δ RCF ≈ · f t − +Δ · (27) ⎨⎪ 2 LB N M i M i,N−1 fN 2 1+δ /T T T i−1 L + f t − − f t − − · LB M i 2 M i 1 2 1 N−1 − ⎪ i i · i 1 ⎪ T T ti+L ΔM 2 LBn+1 Ln ⎩ − f t − − f t − − ˆ N · · i i − · j(i 1),1 · M ( i 1 2 ) − j(i),1 · M ( i 2 ) M M = T LN +LN ΔM + ε ε SN out i,n LD f1 LD f1 2 1+δ /T = n=1 L · i − i−1 f2 ti + LB1 ti−1 + LB1 T fM ti − + εj,1 · 2 T i,1 i−1 LD (28) f t − · T + δ +Δ · T · LB f1 ≈ 1 · M i 2 LB M 1 f i,1 2 T + δLB or, in an equivalent form, which makes clearer the dependence 1 T Δ · LBi−1 on the difference in latency values · − M 1 = fM ti + i,1 . (23) f2 2 1+δ /T N LB t +Li Δ 2 − ˆ i N M · i · i M M = LN T + LBn We omit the last term in step 3 because its magnitude is SN out 2 n=1 negligible in comparison with the other terms. Slave passes on N−1 i,n 2 LBi · δ · 1+Li /T f t − T the received Sync message to Slave at time t + LBi + LBi · n+1 L n j,1 · M i 2 3 i 1 2 +ΔM i,n + εLD . (29) 1+δ /T f1 and replaces the received master counter content by its estimate n=1 L of the master counter value at the time of transmission. Using (16), (12), and (23), (2) becomes The first term in the second line of (29) is a zero mean term that is present due to the different Sync message propagation times. t +LBi +LBi t +LBi ˆ i 1 2 ˆ i 1 ˆj(i) î · i The generalization for SlaveN’s master counter estimate is M = M + S2 (LD)+S2(BD) RCF2 S2out S1out t +Li T ˆ i N i t +LBi M ≈ M(ti)+fM ti − · L i 1 j(i) i i S out N = Mˆ + f2 · LD + f2 · BD · RCF N 2 S out 2 2 2 1 N−1 i · i−1 f t − T LBn+1 Ln j,1 M i 2 t +LBi +Δ · − ε · . ˆ i 1 i i · · i M i,n LD (30) = M + LD2 + BD2 f2 RCF2 1+δ /T f1 S1out n=1 L f t − T i · − T − j,1 · M i 2 From this, we see that, regardless of the subsequent master = M(ti)+LB1 fM ti εLD 2 f1 behavior, the slaves’ estimates grow with the master frequency · i−1 gradient in the most recent two Sync messages. To see what i T ΔM LB1 + LB · fM ti − + 2 2 i,1 error is added in each element, it is useful to write the foregoing 1+δLB /T formulas in incremental form. The incremental master estimate T i i = M(ti)+fM ti − · LB + LB follows directly from (2), using (16), (12), and (27), and the 2 1 2 same reasoning as in (24), i.e., i−1 i − T ΔM ·LB ·LB fM ti t +Li t +Li 1 2 − j,1 · 2 ˆ i N ˆ i N−1 i i · · i + i,1 εLD . (24) M = M + LDN + BDN fN RCFN f1 SN out SN−1out 1+δLB /T i · i−1 t +Li T LB L − = Mˆ i N−1 +LBi ·f t − +Δ · N N 1 . For steps 2 to 3, we have used the fact that in the absence of N M i M i,N−1 SN−1out 2 1+δ /T j(i) i L jitter the line delay is constant; therefore, LD2 = LD2.From (31) (10), the true value of the master counter at this time is

i i Whereas the incremental true master counter is [from (10)] |ti+LB1+LB2 M = M(ti)+fM (ti−1) i t +Li ΔM 2 |ti+LN | i N−1 i · i · i Δ 2 M = M +fM ti +LN−1 LBN + LBN · LBi + LBi + M · LBi + LBi . (25) 2 1 2 2 1 2 (32)

Fig. 4. Synchronization error at slave2 after a change in the master frequency drift. Fig. 3. Synchronization error at slave2. so that the incremental master counter estimate error is The incremental estimation error in this case will be

i i i t +L t +L Δ 2 ti+L ti+L − Δ 2 − ˆ i N − ˆ i N−1 M · i − ˆ N − ˆ N 1 M · i M M = M M + LBN M M =(M M) − + LBN SN out SN−1out 2 N N 1 2 T i − − i T + fM ti + LN−1 fM ti + f t + L − f t − 2 M i N−1 M i 2 i i−1 · − i LBN LN−1 i i 1 · − · LB · L − LBN ΔM i,N−1 i N N 1 · LB − ΔM · 1+δL /T N i,N−1 1+δL /T t +Li − ˆ i N−1 · i = M M +ΔM LBN i i S − out t +L LB N 1 − ˆ i N−1 · i · i N − = M M +ΔM LBN LN−1 + i i 1 SN out T + LB L − 2 · i N − N 1 LN−1 + − 2 1+δi,N 1/T i−1 L T L − · i · − N 1 t +Li + δM LBN i,N−1 − ˆ i N−1 · i 2 1+δ /T = M M +ΔM LBN L SN−1out − t +Li i i,N 1 · i − ˆ i N−1 − · i T +LBN δL (1 + LN−1/T ) = M M +(ΔM ΔM ) LBN · + . (33) SN−1out 2 i,N−1 1+δL /T LBi · Li + N +Δ · LBi This shows the new error term added in each element if N−1 2 M N the master drifts. The first term does not change sign, and the − i i,N 1 · i second is zero mean due to the variation in line and bridge T +LB δL (1+LN−1/T ) · N + . (36) delays. If these were constant in time, then the error would 2 i,N−1 1+ δL /T increase by i Fig. 4 shows in gray this error at the second slave for t +Li t +L Δ 2 (M −Mˆ )| i N −(M −Mˆ )| i N−1 = M · T · LBi +LBi constant line and bridge delays, where the last summand in (36) SN out SN−1out 2 N N (34) vanishes. Figs. 3 and 4 show that the slaves partially follow the frequency change of the master. As the calculation of RCF for all elements after slave1.Slave1 also incurs the error in line uses two consecutive Sync messages, slave elements learn the delay estimation caused by the frequency drift in the master trend of the frequency change of the master from the counters clock [last term in (29)]. Fig. 3 shows the error term of (34) delivered in these two Sync messages. If the frequency drift for the second slave and illustrates the incremental propagation changes its direction after the departure of a Sync message, down the line as long as the master drift persists. The areas of then the propagation of this Sync message does not contain any the gray rectangles correspond to the first error summand, and information about the new frequency drift, and the slaves will the area of the triangles to the second error summand. still follow the old frequency drift. At that moment, the (old) The worst-case scenario is that the drift changes its direction line delay estimation does not reflect the new frequency drift right after the transmission of the ith Sync message. In this case, either. SlaveN’s master counter estimate follows (30). However, if ΔM is the slope of the new frequency change, then the true counter value of the master at the corresponding time is D. Error in RCF and Master-Counter Estimation Due to Slave Frequency Drift for Nondrifting Master t +Li t +Li i i ΔM i2 M| i N = M| i N−1 +f t + L · LB + · LB . M i N−1 N 2 N For the nondrifting master, (7) and (9) apply, whereas for the (35) drifting slaves, (8) and (11) apply. Summarizing from [14], the

Authorized licensed use limited to: T U MUENCHEN. Downloaded on September 3, 2009 at 03:44 from IEEE Xplore. Restrictions apply. 1854 IEEE TRANSACTIONS ON INSTRUMENTATION AND MEASUREMENT, VOL. 58, NO. 6, JUNE 2009

RCF and the master counter estimation error for slave N are constant frequencies the formula obtained in [13] for the line delay estimate of slave n [cf. (13)], we obtain f RCF i ∼ M N = (37) j T +δi,N RD i − L j 1 n−1 j,n fN ti + L Sˆ (LD ) ≈ f · CD + · I(J − )+I(J )+ · δ N 2 n n n n 2 n 1 n R LD t +Li t +Li − ˆ i N − ˆ i N−1 · i − j(i) (39) M M = M M + fM LDN LDN SN out SN−1out j,n j(i) where δ is zero mean. Since for equal elements J − = f ·Δ − RD − f ·Δ LD n 1 + M N 1 · N 1 · R+RDj(i) − M N f (t ) 4 N−1 f (t ) Jn = J, averaging results in N i N i i,N ˆ j ≈ · T + δ avg Sn LDn fn (CDn + J) (40) · LDj(i) · L − Ai + RDj(i) N 2 N N−1 i.e., averaging the line delay estimates removes the mean of the R + RDj(i) line delay jitter from the master counter estimation error. Using · N−1 i · i i,N i i + BDN BDN + T + δL /2 . the fact that for equal elements LDn = CDn + ILD(J) 4 avg Sˆ LDj ≈ f · LDi − Ii (J)+J¯ (38) n n n n LD · i − i ˘ = fn LDn ILD(J) . (41) This error has four components. The 1st term is the error handed down by the predecessor. The 2nd term is due to the estimated Thus, the total line and bridge estimate jitter in (2) is line delay being different from the actual incurred line delay, avg Sˆ LDj + Sˆ (BDi ) when there are jitters. The 3rd term is the own error, even if n n n n the own frequency is constant due to the error in line delay ≈ · i − i ˘ i − i fn LDn ILD(J) + fn BDn IBD(J) computation if the predecessor is drifting. Finally, the 4th is the = f · LBi − Ii (J˘) − Ii (J) own error due to the own frequency drift, which also includes n n LD BD a clear line delay computation error. We see that each drifting ≡ · i − i fn LBn Jn . (42) slave leads to an additive error of identical structure, which will be passed on unchanged down the line. An additional smaller Ji ≡ Ii (J˘)+Ii (J) error, due to the error in the line delay estimation, is contributed We have denoted n LD BD . In addition, the master counter estimate needs to be rounded to his successor. For every drifting slave, these error terms get to an integer before it can be passed on. This is captured in added both in the drifting slave and its successor and are passed the rounding error ρi,n, which is a zero-mean random variable down the line together with the previous accumulated error. Mˆ with |ρi,n|≤1/2. Summarizing from [10], the RCF estimate of Mˆ Slave is IV. ANALYSIS OF ERROR PROPAGATION—PART 2: THE N INFLUENCE OF JITTER AT CONSTANT CLOCK SPEEDS i fM · − i,N RCFN = 1 εRCF (43) Without frequency drift, rounding and receive/transmit stamp fN jitter are the main sources of synchronization error. Both effects and the TC transmission error of SlaveN is accumulate along the line, compromising the synchronization precision of distant elements. The resulting error propagation N i,n t +Li ρ is not additive because jitter also affects the RCF estimation. M − Mˆ i N = f · LBi − Ji · εi,n + Ji − Mˆ S out M n n RCF n N fM Since in (2), the line and bridge delay estimates enter via their n=1 sum, we have in [10] “virtually” allocated the whole jitter to (44) the line delay. Here, we drop this simplification. Let the sum of the random variables “receive and transmit stamp jitter” in rec trans where the error in the RCF computation of the Nth slave is element n be Jn = Jn + Jn , with mean J n. The jitter that given by the recursive formula (45), shown at the bottom of the ˆ i i,1 i,1 i,1 affects an element’s estimate of its bridge delay Sn(BDn) is page, and ε = δ /(T + δ ) is the particularization of i ˘ RCF LD LD IBD(Jn), which is an instantiation of Jn, and let I(Jn) be a (45) to N =1. In (44), the first term is recursively defined and sample of Jn − J n. All are in time, not counters. The line delay reflects the errors in RCF computation. The other two additive of a message and the RCF estimate of element n are affected terms reflect the direct impacts of jitter and rounding error. They trans rec by samples of Jn = Jn−1 + Jn . To simplify, we consider all are present even if the RCF computation is perfect. ˆ i elements to be equal, i.e., Jn = Jn = J. Then, Sn(BDn)= The benefit of RCF estimate averaging becomes obvious · i − i fn (BDn IBD(J)). The (constant) cable delay to element in this scenario. Let two Sync messages be used for RCF i n is CDn, with LDn = CDn + I(Jn). Adapting to jitter and computation, and let ΔtM and ΔtS be their interdeparture and

i,n i−1,n − ρ −ρ − − N ˘i − ˘i−1 N 1 Mˆ Mˆ N 1 i − i · i,n − i−1 − i−1 · i 1,n Jn Jn + f + LBn Jn εRCF LBn Jn εRCF i,N n=1 n=1 M n=1 ε = − (45) RCF N 1 i,n N ˘i − ˘i−1 T + n=1 δBD + n=1 Jn Jn

interarrival times measured in absolute time. Then, ΔtS = TABLE I SIMULATION SETTINGS ΔtM + δLD, and the corresponding counter increases are ΔC = f · Δt and ΔC = f · Δt . Hence M M M S S S ΔC f δ RCF = M = M · 1 − LD . (46) ΔCS fS ΔtS Since the second term is zero mean, averaging will result in the desired convergence of RCF to the quotient of the clock frequencies. The following section discusses the consequences of the obtained analytical results.

V. D ISCUSSION AND RECOMMENDATIONS

We have examined the error at the “points of support” of can conclude that a costly master brings a disproportionately the synchronization process, i.e., the times when an element large synchronization benefit as compared with allocating the receives new information via a Sync message. In between, the corresponding cost fraction to improving each slave. estimate of the master counter is updated according to i i+1,n Mˆ | i =Mˆ | i +S (τ) · RCF ,τ∈ 0,T +δ . n ti+Ln+τ n ti+Ln n n sync L VI. SIMULATION RESULTS (47) We have developed a MATLAB simulation tool to test and The quality of any synchronization result in between these time analyze the synchronization performance of IEEE 1588 in a instants is critically dependent on the correctness of the support line with cascaded bridges. We have used this tool to simulate estimates and on the stability of the master quartz during Sync PTP in PROFINET [15]. The model parameters, which are message interarrival intervals (holdover time). summarized in Table I, are given by the Siemens Automation To derive recommendations for parameter choice, a system- and Drive Department. In the simulation performed for this identification process has to determine whether the momentary paper, we investigate the effect of linear frequency drifts of limiting factor is due to frequency drift or to jitters. 3 ppm/s. Since we are not investigating quartzes but are investi- If the limiting factor is frequency drift, equations (28) and gating synchronization, we have herein combined the effects of (29) indicate how to achieve a possible reduction of the time several possible perturbation causes, such as aging, temperature synchronization error, namely by drift, voltage fluctuations (electromagnetic compatibility), etc. 1) minimizing the influence of outside disturbances (e.g., The frequency change starts at 20 s, increases with maximal temperature change) on the oscillator (e.g., insulation); slope in the next 20 s, and then stays constant again. For these 2) choosing oscillators that are not sensitive to disturbances; simulations, we have used bridge delays of 5–15 ms, which 3) decreasing T (inter-transmission time of the two Sync have been state-of-the-art until recently, and which clearly messages used for RCF calculation); however, this illustrate the obtained results. Decreased bridge delays of conflicts with the goal to minimize the influence of 5–125 μs lead to significant performance gains. jitter; 4) shortening the latency (line and bridge delays); 5) regulating the forwarding of Sync messages so that the A. Simulation Scenario 1: Master Drifting bridge delays are approximately constant over time, re- In this section, we simulate the case where the master is moving the zero mean error term; drifting while the slaves’ frequencies are constant. 6) shortening the response delay and interval between delay We first look at the error in line delay estimation, shown in messages to reduce the error in line delay estimation; Fig. 5. The first slave estimates the line delay based on the time 7) developing new synchronization algorithms that take into stamped in itself and in the master, whose frequency is drifting consideration the effect of frequency change. A possible during [20, 40] s. As shown, error is introduced in the line delay solution was presented in [11]. estimation. This can be observed in subplot 1. Subplot 2 shows If the limiting factor is jitter, then (44) and (45) indicate the line delay estimation error at slave 4. Since the estimation of how to achieve a reduction of the synchronization error, line delay at slave 4 only depends on the frequencies of slaves 4 namely by and 3, which do not drift, the estimate is correct. Subplot 1 8) increasing T [see 3)] to decrease the RCF error; however, also shows that by shortening either the response delay or the this is detrimental if the master frequency is changing over interval between delay messages, the magnitude of the line time; delay estimation error is reduced. 9) decreasing bridge delay (and cable delay if possible); The errors in the master counter estimation are shown in 10) reducing jitter by improved hardware; Fig. 6. We see that the frequency change introduces a bias to 11) averaging over a number of consecutive RCF estimates to the estimated master time (the error is not zero mean). The reduce the error in the recursive additive term. magnitude of this bias increases along the line. Our analytic On the other hand, the comparison of master drifting versus result in (33) also shows that each slave adds its own contri- slaves drifting [see (33) and (38)] shows that the single-slave bution to this bias. When the frequency change stops, this bias frequency drift is very benign compared with the master fre- disappears. The noise on top of the bias is present due to the quency drift, which is only matched by all slaves drifting. We variation of the line delays and bridge delays.

Authorized licensed use limited to: T U MUENCHEN. Downloaded on September 3, 2009 at 03:44 from IEEE Xplore. Restrictions apply. 1856 IEEE TRANSACTIONS ON INSTRUMENTATION AND MEASUREMENT, VOL. 58, NO. 6, JUNE 2009

Fig. 7. Sync error at slaves 1, 2, 32, and 49 when master is drifting with the interval of RCF calculation halved to 100 ms.

Fig. 5. Line delay estimation error when the master is drifting.

Fig. 8. Sync error at slaves 1, 2, 32, and 49 when slave 1 is drifting.

Fig. 6. Sync error at slaves 1, 2, 32, and 49 when the master is drifting.

As discussed in Section V, several measures can be taken to reduce the effect of outside disturbances. We halve the interval of RCF calculation and run the simulation again. The results, which are shown in Fig. 7, have clearly improved.

B. Simulation Scenario 1: Different Slaves Drifting In this section, we simulate the case when drifting takes place at different slaves while the master frequency is constant and display the synchronization error in the following figures: Fig. 8 for the case that only the first slave is drifting; Fig. 9 if exactly the first two slaves are identically drifting; and Fig. 10 if all the slaves are identically drifting. Fig. 9. Sync error at slaves 1, 2, 32, and 49 when slaves 1 and 2 are drifting. Comparing Figs. 8–10 with Fig. 6, we observe that, if only one element is drifting, the error at the end of the line is by a VII. FUTURE WORK factor of 10 larger if this element is the master and not a slave. The same relationship holds for the master versus two slaves Our next steps will be to further investigate short-term fre- drifting. Only if all the slaves exhibit identical nonzero frequency deviations, e.g., due to vibration and shocks, as well as quency drift do they match the effect of “master only drifting.” work toward more precise statements concerning the influence

Ruxandra Lupas Scheiterer received the Vordiplom degree in electrical engineering from the Technical University of Munich, Germany, in 1983 and the M.S. degree in electrical engineering and computer science and the Ph.D. degree in information sciences and systems from Princeton University, NJ, in 1985 and 1989, respectively. Since 1989, she has been with Corporate Tech- nology, Siemens AG, Munich, Germany, with oc- casional maternity arrangements. She is currently a Senior Research Scientist with the Information and Communications Department. Her interest in math was sparked as a 12-year-old in Romania by her grandmother, a mathematician. She won the German National High-School Math Competition in 1979, 1980, and 1981. Her R&D interests include optimization and learning for intelligent systems and the interworking of telecommunication, control, and decision making.

Fig. 10. Sync error at slaves 1, 2, 32, and 49 when all the slaves are drifting. Chongning Na (S’08) received the B.E. degree in mechanical engineering in 2002 from Tsinghua of quantization errors on synchronization accuracy. A first University, Beijing, China, and the M.S. degree report of this work has recently been published in [16]. (with an academic award) in electrical engineering in 2005 from the Technical University of Munich, Germany, where he is currently working toward the REFERENCES Ph.D. degree. His Ph.D. work is funded by Siemens AG, Corporate Technology, Information, and [1] IEEE Standard for LAN/MAN CSMA/CD Access, IEEE 802.3, 1997. Communications. [2] J. Nieminen, “Synchronization of next generation wireless communica- In 2002, he was with Microsoft Research Asia, tion systems,” M.S. thesis, Helsinki Univ. Technol., Finland, 2007. Beijing. Since Summer 2004 he has been with [3] IEEE Standard for Local and Metropolitan Area Networks—Timing and Siemens AG, Munich, and since Spring 2005 with Corporate Technology, Synchronization for Time-Sensitive Applications in Bridged Local Area Information, and Communications. His research interests are in information Networks, IEEE 802.1AS, 2007. theory, probabilistic inference, machine learning and their application in [4] B. Chen, Y. P. Chen, J. M. Xie, Z. D. Zhou, and J. M. Sa, “Control telecommunications, control systems, and signal processing. methodologies in networked motion control systems,” in Proc. Int. Conf. Mach. Learn. Cybern., Guangzhou, China, 2005, pp. 1088–1093. [5] D. L. Mills, “Internet time synchronization: The network time protocol,” Network Working Group Request for Comments: 1129, Oct. 1989. [6] D. L. Mills, “Precision synchronization of computer network clocks,” ACM SIGCOMM Comput. Commun. Rev., vol. 24, no. 2, pp. 28–43, Apr. 1994. [Online]. Available: citeseer.nj.nec.com/mills94precision.html Dragan Obradovic (M’86) received the B.E. and [7] IEEE Standard for a Precision Clock Synchronization Protocol for Net- M.E. degrees in mechanical engineering from the worked Measurement and Control Systems, ANSI/IEEE Std. 1588-2002. University of Belgrade, Yugoslavia, in 1980 and [8] J. Jasperneite, K. Shehab, and K. Weber, “Enhancements to the 1985, respectively, and the Ph.D. degree in mechan- time synchronization standard IEEE-1588 for a system of cascaded ical engineering from the Massachusetts Institute of bridges,” in Proc. IEEE Int. Workshop Factory Commun. Syst., 2004, Technology, Cambridge, in 1990. pp. 239–244. Since 1992, he has been with Corporate Tech- [9] IEEE P1588TM D2.2 Draft Standard for a Precision Clock Synchroniza- nology, Siemens AG, Munich, Germany, where he tion Protocol for Networked Measurement and Control Systems, 2007. is currently a Senior Research Scientist with the [10] C. Na, D. Obradovic, R. L. Scheiterer, G. Steindl, and F. J. Goetz, “Syn- Information and Communications Department. He is chronization performance of the precision time protocol,” in Proc. IEEE also an Industrial Lecturer with the Munich Univer- Int. Symp. Precision Clock Synchronization Meas., Control Commun., sity of Technology, where he has been teaching since 2001 with the inter- Vienna, Austria, 2007, pp. 25–32. national Masters of Science in Communication Engineering program within [11] C. Na, D. Obradovic, R. L. Scheiterer, G. Steindl, and F. J. Goetz, the Electrical Engineering Department. His research involves the development “Enhancement of the precision time protocol in automation networks and application of machine learning methods for system control and signal with a line topology,” in Proc. IFAC World Congr., Seoul, Korea, 2008. processing. He has published dozens of journal and international conference [12] D. Obradovic, R. L. Scheiterer, C. Na, G. Steindl, and F. J. Goetz, “Clock papers and the book An Information-Theoretic Approach to Neural Computing synchronization in industrial automation networks: Comparison of differ- (New York, Springer, 1996) and is author of several patents. ent synchronization methods,” in Proc. 5th Int. Conf. Informat. Control, Autom. Robot., Funchal, Portugal, 2008. [13] R. L. Scheiterer, D. Obradovic, C. Na, G. Steindl, and F. J. Goetz, “Syn- chronization performance of the precision time protocol: Effect of clock frequency drift on the line delay computation,” in Proc. WFCS,Dresden, Germany, 2008, pp. 243–246. Günter Steindl received the Master degree in elec- [14] R. L. Scheiterer, C. Na, D. Obradovic, G. Steindl, and F. J. Goetz, “Syn- trical engineering from Fachhochschule Regensburg, chronization performance of the precision time protocol in the face of Ratisbona, Germany, in 1990. slave clock frequency drift,” in Proc. IEEE CASE Conf., Washington DC, In 1990, he joined the Industry Sector, Automation 2008, pp. 554–559. Systems, Research, and Development Department, [15] J. Jasperneite and J. Feld, “PROFINET: An integration platform for Siemens AG, Amberg, Germany, where he devel- heterogeneous industrial communication systems,” in Proc. 10th IEEE oped the communication protocol PROFIBUS DP. ETFA, 2005, pp. 9–12. In 2001, he began the design of the architecture [16] R. L. Scheiterer, C. Na, D. Obradovic, G. Steindl, and F. J. Goetz, of the Ethernet communication protocol PROFINET “1 μs-conform line length of the transparent clock mechanism defined IO. In 2002, he started work on the industrial stan- by the precision time protocol (PTP Version2),” in Proc. IEEE Int. Symp. dardization of PROFINET IO by the Profibus User Precision Clock Synchronization Meas., Control Commun., Ann Arbor, Organisation for Siemens AG and, in 2005, the international standardization of MI, 2008, pp. 92–97. the IEC SC 65.

Authorized licensed use limited to: T U MUENCHEN. Downloaded on September 3, 2009 at 03:44 from IEEE Xplore. Restrictions apply.