1

The White Rabbit Synchronization Protocol for Synchrophasor Networks Asja Derviskadiˇ c,´ Member, IEEE, Reza Razzaghi, Member, IEEE, Quentin Walger, Student Member, IEEE, Mario Paolone, Senior Member, IEEE

Abstract—Within the context of time dissemination techniques to the sky) [5]. As suggested in [3], until timing challenges for power systems applications, the paper discusses the use of have been resolved and time dissemination reliability assured, the White Rabbit (WR) protocol for synchrophasor networks. PMUs cannot be used for mission-critical operations, such Specifically, the paper presents a Phasor Measurement Unit (PMU) integrating the WR technology and its experimental as protection or automated control. To improve timing re- validation with a focus on the synchrophasor phase estimation dundancy and reliability, given the potential vulnerability of in steady state conditions, by using a PMU calibrator generating GPS, synchrophasor applications should use multiple timing the reference signals. We further compare the accuracy of the sources, for instance deployable over the legacy power sys- developed PMU with other state-of-the-art time synchronization tem telecom infrastructure. As a matter of fact, any PMU- technologies for PMUs, i.e., Global Positioning System (GPS) and Precision Time Protocol (PTP), demonstrating applicability based monitoring or control application relies on a telecom of WR for PMU sensing networks. infrastructure to stream PMU data and the same physical layer may be used for time dissemination purposes. In this context, Index Terms—Phasor Measurement Unit (PMU), synchropha- sor network, time synchronization, White Rabbit the paper investigates alternative or complement solutions to the GPS, with particular focus on cases when the sky is not accessible and the Ethernet-based telecom infrastructure I.INTRODUCTION is already available (e.g., urban areas). Among the possible Synchrophasor technology is the leading edge of timing use alternatives, the paper presents the White Rabbit (WR) Time for power systems as Phasor Measurement Units (PMUs) can- Protocol [6]–[8], and compares its performance with respect not be adopted for mission-critical or automated actions, unless to the GPS and the Precision Time Protocol (PTP) [9]. coupled with an appropriate time dissemination technique. The PTP was introduced by the IEEE Std. 1588 in order to IEEE Std. C37.118.1 requires a maximum uncertainty in the provide a hardware-level time accuracy using a standard Local synchrophasor time stamp of 1 µs [1]. Indeed, in order to Area Network (LAN) connection (e.g., Ethernet) [9], and properly phase-align and report synchrophasors measured by is characterized by an accuracy of 1 µs. As an evolution PMUs located in geographically-distant substations, the net- of PTP, the WR time synchronization protocol is a low- work nodes have to share a common, accurate and reliable time latency, time-deterministic Ethernet-based time dissemination reference. Poor time-synchronization causes inaccurate phasor technique, developed for distributed sensing systems [6]. The estimations (particularly relevant for phase estimations) that, if project was initiated at CERN (European Organization for Nu- undetected by the overlying applications (e.g., state estimators clear Research) to develop an ultra-precise timing system for bad data processes [2]), may cause incorrect interpretations CERN accelerator complex. The WR is based on the standards of the grid conditions and inappropriate actions [3]. This is Ethernet (IEEE 802.3) [10] and Synchronous Ethernet (SyncE) particularly critical for distribution networks since they require [11]. It enables the synchronization of thousands of devices an increased level of PMU accuracy [4]. connected in a network spanning several kilometers through The time synchronization of PMUs typically relies on the already existing communication networks. The accuracy, is Global Positioning System (GPS) as it represents a good meant to achieve the sub-nanosecond, assuming only fiber trade-off between performance and cost. GPS provides an interconnections and dedicated telecom switches. Moreover, accuracy in the order of ±50 ns when coupled with modern the protocol features a reliable and deterministic data delivery. GPS-receivers. However, the GPS is vulnerable to timing- Within this context, the scope of this paper is twofold. First, attacks and is not always physically accessible (consider for we present a PMU (also called WR-PMU) integrating the instance the case of underground substation without an access WR protocol as time dissemination technology. We describe all technological aspects of the timing architecture and its Asja Derviskadiˇ c,´ Quentin Walger, Mario Paolone are with the Ecole´ integration in an embedded device. Second, we assess the Polytechnique Fed´ erale´ de Lausanne EPFL, CH-1015, Lausanne, Switzerland performance of the developed WR-PMU. We experimentally (e-mail: asja.derviskadic@epfl.ch). validate the phase estimation stability over the short, medium Reza Razzaghi is with Monash University, Melbourne, Australia (e-mail: [email protected]). and long term, by means of reference signals generated by This project is carried out within the frame of the Swiss Centre for a PMU calibrator [12]. A preliminary analysis has been Competence in Energy Research on the Future Swiss Electrical Infrastructure presented in [13]. In the current manuscript, we present an (SCCER-FURIES) with the financial support of the Swiss Innovation Agency (Innosuisse - SCCER program). Also, the work has received funding from the enhanced version of the WR-PMU that integrates an internal Qatar Environment and Energy Research Institute (QEERI). regulated by a PI controller. Furthermore, we compare 2 the performance of the developed WR-PMU with respect to accuracy requirements, at least two orders of magnitude lower other state-of-the-art synchronization technologies for PMUs, than those met by transmission PMUs (TVE lower than 0.01%) i.e., GPS and PTP. To the best of the Authors knowledge, this [4]. Therefore, the uncertainty contribution coming from the is the first paper presenting a WR-synchronized PMU. timing unit should be reduced the order of tens of ns [3]. The paper is structured as follows. Section II describes state- In the following, two time dissemination technologies that of-the-art time dissemination techniques for PMU applications. are currently being used for PMU applications are described: Section III illustrates the operating principles of the WR (i) satellite and (ii) network-based synchronization systems, protocol as well as its applicability to synchrophasor networks. making reference to their functional features and performance. Section IV describes the implementation details of the three We discuss their applicability to synchrophaosor technology developed PMUs. Section V assesses their performance. Sec- and their vulnerability to timing-attacks [3]. tion VI concludes the paper with final remarks. A. Satellite-based Time Synchronization Systems II.TIME SYNCHRONIZATION TECHNIQUESFOR PMUS The operation principle of satellite systems is based on the Time synchronization is a key factor in any PMU-based time measurement of synchronizing signals between satellites monitoring systems [3]. The IEEE Std. C37.118.1 [1] defines and terrestrial receivers. The satellites are equipped with the phase of a synchrophasor as the instantaneous phase angle atomic , daily monitored and controlled to be highly relative to a cosine function at the nominal power system fre- synchronized and traceable to the UTC time. The receivers quency, synchronized to Coordinated Universal Time (UTC). are equipped with an internal clock, and are able to determine In that sense, any uncertainty in the time synchronization ∆t the actual UTC time by collecting and processing messages linearly translates in a phase uncertainty ∆ϕ, depending on from several satellites. GPS receivers are often used as pri- the instantaneous frequency f of the signal: mary absolute timing source for most of time dissemination ∆ϕ = 2πf∆t + εalg + εacq (1) techniques. As known, PMU applications generally rely on the GPS that where ε and ε account for two additional uncertainty alg acq provides an accuracy in the order of ± 100 ns when coupled sources, i.e., the phase error introduced by the adopted syn- with commercial GPS-receivers (e.g., [14]), although modern chrophasor estimation algorithm and the produced units can nowadays reach accuracy lower than ±50 ns. In by the acquisition process (including the measurement chain such scenario, a dedicated GPS receiver must be installed at from the sensor to the PMU analog input), respectively. Since every PMU location, and the same applies to Phasor Data these errors come from independent devices, we assume these Concentrators (PDC) in case time-stamping functionalities are two contributions to be statistically independent and uncorre- implemented at data collection. lated, and we focus mainly on the synchronization uncertainty To correctly lock satellites, the GPS receiver requires a (see Section V for more details). The same standard further clear view of the sky. Indeed, being in an enclosed space requires that synchrophasor measurements are reported by such as a high rise urban environment, reduces the number PMUs at a specific reporting rate, with the first frame within of tracked satellites and determines signal reflections and the second at the UTC-second rollover. wakening, resulting in a degradation of the time information The IEEE Std. C37.118.1 [1] suggests a maximum uncer- accuracy [15]. tainty in the synchrophasor time stamp of 1 µs. This value Regarding security, the GPS signals can be easily spoofed is indirectly determined by the requirement for a maximum resulting into complex and potentially dangerous time attacks Total Vector Error (TVE) of 1 %. The TVE is defined [16]. Among different types of attacks, GPS spoofing is the as the Euclidean distance between the true and estimated most malicious and difficult to detect [5]. It is achieved by synchrophasors, normalized with respect to the amplitude of superimposing a fake signal with a higher signal-to-noise ratio, the true synchrophasor. As such, it is a performance indicator which would enable an attacker to manipulate the GPS clock. that accounts for a component due to the measurement of am- With particular reference to the GPS-based PMUs, a spoofing plitude and a component due to the measurement of phase. Let attack can cause the GPS receiver of a PMU to compute an us suppose that the contribution of the amplitude error to the erroneous clock offset, resulting in an erroneous time stamp TVE is negligible and therefore the TVE is only influenced by calculation, which introduces an error in the PMU’s phase the phase error. Simple trigonometry will lead to the fact that, measurement. The failure to deliver data to concentrators and regardless of the angle being measured, a phase uncertainty applications within acceptable latency periods causes data gaps of 0.01 rad will itself cause a 1% TVE. If we consider the that could corrupt early warning information about dynamic synchronous grid of Continental Europe characterized by a grid conditions. nominal frequency of 50 Hz, according to (1), this corresponds to an error of ± 31 µs, when time is the only source of error. A reliable time source should be characterized by an uncertainty B. Packed-Switching Synchronization Messaging Protocols at least 10 better, giving some allowance for sources of Typically, synchrophasor networks use the Ethernet network error other than synchronization, leading to the recommended protocol as physical layer to transfer data. The protocol, intro- time uncertainty of 1 µs. duced by the IEEE Std. 802.3, represents a well-established However, it is well-established that PMUs operating in and very high-performance solution, that is capable of sup- distribution networks are expected to meet more stringent porting the high-throughput of synchrophasor data streams 3

[10]. The Ethernet protocol also integrates various standards UTC time Reference PPS UTC and PPS provided WR master only at start-up that enable the time synchronization of the network nodes clock 10 MHz with different levels of accuracy. In other words, the same telecom infrastructure used for seamless data transfer could WR switch WR switch be exploited for disseminating the time information.

The (NTP) has been proposed to WR switch WR switch WR node synchronize the clocks of a distributed system over the Internet [17]. However, the average accuracy provided by the NTP is WR node WR node in the range of few milliseconds that does not fulfill the PMU requirements. Fig. 1. The WR network architecture. The Precision Time Protocol (PTP) was introduced by the IEEE Std. 1588 in order to provide time accuracies beyond those attainable using NTP, thanks to a technique called III.THE WHITE RABBIT TIME SYNCHRONIZATION hardware time-stamping [9]. The most recent PTP version 2 PROTOCOL (PTPv2) provides 1 µs accuracy, measured as the deviation of Recently, the WR protocol, also known as PTP version each node with respect to the UTC. 3 (PTPv3), has been developed and used at CERN to align The core element of the PTP is the exchange of time- the clocks of their accelerator complex [6]–[8]. The protocol tagged messages in a peer-to-peer link between master and enables the synchronization of thousands of devices connected slave clocks, used to calculate the link delay between the two in a network spanning several kilometers through already clocks. Specifically, at time t1 the master node sends a Sync existing Ethernet-based networks. The accuracy, measured as message, that is received at time t2 by the slave. Similarly, at the deviation of each node with respect to the UTC, achieves time t3, the slave node sends a message, received at time t4 the sub-nanosecond, assuming only fiber interconnections. by the master. Knowing these four time-stamps, the one-way Moreover, the protocol features a reliable and deterministic delay between the two clocks can be estimated as: data delivery. The project is open source [23]. These features make the WR an appropriate time syn- δ = (t2 − t1 + t4 − t3) /2 (2) chronization protocol for smart grids applications [24]–[26]. Indeed, the accuracy on 1 ns exceeds the one of synchrophasor The slave node can account for this offset when adjusting its needs. Also, the superior determinism with respect to PTP is clock time with respect to the one of its master clock. good for reliability and mission-critical applications. This tech- The PTP assumes that all network nodes are equipped with nology represents an appropriate alternative or complement to PTP-aware routers or switches, implementing the so-called the GPS with particular focus on the cases when (i) the sky is hardware-assisted time-stamping, a technique to measure and not accessible (e.g., urban areas), (ii) the telecommunication compensate for the time spent by messages in queuing at their infrastructure is already available, and (iii) the typical length own ports. between two PMUs is less than 10 km (e.g., sub transmission The first limitation of the PTP is that it assumes that the one- or power distribution networks). The main limitation of the way delay is exactly half of the two-way delay, which, due to WR technology for synchrophasor networks arises when the link asymmetry is true only as long as the cable is very short. electrical grid is not equipped with fiber-optic cables. Indeed, The second limitation is that the final PTP accuracy is limited from an economical perspective, refurbishing the feeder with by the precision and resolution of the master and slave clocks fiber links may result in large installation costs that could to measure the time when sending or receiving messages, hinder the cost-benefit analysis related to the WR solution. typically of 100 ppm. The third limitation is that these clocks It is worth mentioning that recent studies have demonstrated are typically free-running oscillators, without any guarantee the stability of the WR protocol over fiber links up to 950 of synchronism between oscillators at different nodes. This km [27], [28]. Further studies have addressed the problem of results in uncontrolled time drift between masters and slaves. temperature-related hardware delays [29]. The higher the exchange rate of PTP messages, the lower the time drift, the higher the bandwidth needed for PTP-related traffic. A. The White Rabbit Network Architecture The security of PTP (as well as WR) against cyber-attacks Figure 1 shows the layout of a typical WR network, that is studied in [18] by using a so-called delay-box that intro- is composed of WR nodes and WR switches, interconnected duces a malicious offset of a few microseconds in the slave by fiber links1. Data-wise it is a standard Ethernet switched clock. Nevertheless, the attack can be counteracted by using network, i.e., there is no hierarchy: any node can talk to any redundant and disjoint communication paths or using the GPS other node in the network. Regarding time synchronization, as a redundant time source. there is a hierarchy, that goes from the top, namely from the An extended profile for the use of PTP in power system WR master, down to other WR switches and consequently applications is specified in [19] and, for instance, used in nodes. The WR switch, key element of any WR network, is [20]. With a specific reference to PMUs, the protocol has been 1Although fiber is the preferred physical layer for WR technology, copper integrated into synchrophasor networks to distribute the time (1000BaseT) can be also used in small portions of the network with less- [21], [22]. stringent timing requirements [30]. 4 similar to a standard Ethernet switch, but it is also able to clock recovered by the data link to sample the incoming data. precisely distribute the WR master clock over the network Then, it uses an embedded PLL-based oscillator, locked to thanks to a technique called precise phase measurement [31]. the recovered clock, for transmission. This procedures ensures The uppermost switch in the hierarchy, also called grand- high level jitter elimination. Since it acts on the physical layer, master, receives the absolute clock from an NTP source (e.g., its accuracy is independent of data transmission (packet delay the NTP daemon running on a computer), together with the or traffic load). The technology has been proven to be able to pulse-per-second (PPS) and the 10 MHz from an external transfer very accurate timing over long distances [7], [11]. reference (e.g., a GPS receiver or a Cesium clock). At start-up, 3) Precise Phase Measurement: The accumulation of phase the WR switch uses the NTP and the PPS to determine the noise degrades the performance of network-based synchro- absolute UTC time. Then, it calculates the time using only the nization protocols. To this end, every WR switch is equipped 10 MHz signal. After the switch has completed the rebooting with a phase measurement module based on phase/frequency routine, i.e., few minutes after powering it on, the NTP service detectors that periodically measures the phase difference be- and the PPS are not needed anymore and the grand-master tween the recovered clock and the master clock [33]. The switch could be potentially disconnected from these sources. calculated phase difference is transmitted to a slave node for The accuracy of the round-trip time measurement is mostly further compensation of the round-trip link delay with sub- determined by the accuracy of the 10 MHz source. The grand- nanosecond accuracy. master switch then distributes the time information to further WR nodes via intermediate WR switches. It is worth pointing IV. INTEGRATIONSCHEMESOFTIMEREFERENCESINTOA out that the subsequent switches do not have to be connected DEDICATED PMU to a 10 MHz source. In the power system context, it is reasonable to expect that In order to compare the performance of the time syn- the grand-master WR switch is located in a safe location, such chronization techniques under investigation, we develop three as the control room of the network operator. To guarantee PMUs based on the same synchrophasor estimation algorithm reliability, the grand-master switch as well as the 10MHz and the same hardware. The only difference among the three sources should be powered via an uninterruptible power supply is the adopted technique to synchronize to the absolute time (UPS). It is also reasonable to expect that the grand-master reference: the so-called GPS-PMU is based on the GPS time switch is rebooted only if needed, few times over the lifetime dissemination technique, and is further described in Section of the synchrophasor network, for instance at the same time IV-C, the PTP-PMU is based on PTPv2 and is described in of rebooting the central phasor data concentrator (PDC) or Section IV-D, whereas the WR-PMU is based on the WR updating PMUs firmware. protocol and its implementation details are given in Section Finally, the security of WR against delay-attacks is studied IV-E. The main features of the three devices are very similar in [18], and countermeasures for this type of attack are to those of the PMU described in [14]: any difference or proposed. similarity is illustrated in this section, with a focus on all implementation details that condition time accuracy. To limit any discrepancy introduced by the synchrophasor B. The White Rabbit Synchronization Scheme estimation process, the three PMUs are based on the same The WR is based on existing standards, namely Ethernet synchrophasor estimation algorithm of [14], an enhanced (IEEE 802.3) [10], Synchronous Ethernet (SyncE) [11], IEEE version of the interpolated Discrete Fourier Transform (DFT), 1588 (PTPv2) [9] and adopts a technique called Precise hereafter called e-IpDFT, that compensates for the effects of Phase Measurement. The combination of these technologies, spectral leakage coming from the negative image of the tone further described in this section, enables to achieve the sub- under analysis. Such PMU was developed at the Distributed nanosecond accuracy [32]. Electrical Systems Laboratory (DESL) or EPFL. Further de- 1) PTPv2: The same process described in Section II-B tails are provided in Section IV-A. holds for calculating the one-way transmission delays. How- As in [14], the hardware platform of the three devices ever, in a WR network, PTP messages are managed not only is based on the National Instruments compactRIO (cRIO) by the grand master clock, but also by the WR switches. system, an embedded industrial controller with a real-time This method prevents PTP messages to be exchanged between processor, a user-programmable Field Programmable Gate long links from the master to a far side slave, reducing the Array (FPGA) and reconfigurable IO modules [34]. It is unavoidable jitter introduced by each switch. Also, the number worth to point out that in the designed architecture, the of messages between master and slaves is reduced, reducing three main processes, i.e., (i) PMU time synchronization, (ii) the PTP-related throughput and allowing more bandwidth for signal acquisition and (iii) synchrophasor estimation, run at mission-critical data exchange. the FPGA level. Indeed, FPGAs provide hardware-timed speed 2) SynchE: Typical PTP implementations use free-running and reliability, that are two essential features for PMUs. oscillators in each node, resulting in growing time drifts The sampling of the voltage and current waveforms is real- between master and slaves. This is solved by the SynchE ized by means of two parallel 24-bits delta–sigma converters, protocol, a technique to transfers the frequency over the module NI 9225 and 9227 respectively, characterized by a Ethernet physical layer, in order to lock all the network nodes sampling rate Fs of 50 kHz and an input range of 300 VRMS to beat at exactly the same rate. Every WR switch uses the for the voltage and 5 ARMS for the current [35], [36]. 5

10-6 A. The Synchrophasor Estimation Algorithm 10 P-class, P = 1 P-class, P = 2 The developed PMUs adopt the e-IpDFT algorithm to 8 ˆ M-class, P = 1 estimate the synchrophasors, i.e., the frequency f, amplitude M-class, P = 2 6 Aˆ, phase angle ϕˆ0 and the Rate-of-Change-of-Frequency (RO-

COF) associated to the fundamental tone of the power system 4 signal under analysis. This technique is specifically designed Phase error [rad] 2 to mitigate the effects of long-range spectral leakage produced 47.5 48 48.5 49 49.5 50 50.5 51 51.5 52 52.5 by the negative image of the fundamental component. Nominal frequency [Hz]

Algorithm 1 The e-IpDFT synchrophasor estimation. Fig. 2. Phase estimation error as provided by e-IpDFT as function of the fundamental frequency, for P and M synchrophasor estimation and when the 1: x[n] := {x(tn) | tn = nTs, n = [0,...N − 1] ∈ N} iteration number P of the compensation process is set equal to 1 and 2. The 2: X(k) = DFT(x[n] · w[n]) additive white Gaussian noise produces a signal-to-noise ratio of 85 dB. 0 0 0 3: {fb , Ab , ϕb0} = IpDFT(X(k)) 4: for p = 1 → P P equal to 2 leads to a significant improvement of the e-IpDFT p− wf p−1 p−1 p−1 5: Xb (k) = (−fb , Ab , −ϕb0 ) estimation accuracy [37]. Therefore, in the developed PMU, p+ p− 6: Xb (k) = X(k) − Xb (k) this compensation routine is performed twice. p p p IpDFT p+ 7: {fb , Ab , ϕb0} = (Xb (k)) It is also worth mentioning that the higher the observation 8: end for interval over witch synchrophasors are measured, the higher the accuracy of the estimates. The IEEE Std. C37.118.1 As described in Algorithm 1, the PMU first acquires a [1] introduces two PMU performance classes: P-class PMUs discrete time-series of samples x[n], where x(t) is the time- are meant for protection applications, and require respon- variant signal under analysis, N is the number of samples that siveness rather than accuracy, M-class PMUs are meant for −1 compose the considered observation interval T and Fs = Ts measurement applications, and require an increased level of is the sampling rate (line 1). The signal is windowed with the synchrophasor estimation accuracy. The PMU proposed in [14] Hanning function w[n] to reduce spectral leakage effects, then adopts an observation interval T of 60 ms, and represents a the DFT of the weighted signal X(k) is computed (line 2). P-class PMU. In the current implementation, the observation A preliminary estimate of the fundamental parameters is interval T can be increased to 100 ms, that is a value typical obtained by processing the highest DFT bins via the IpDFT of M-class PMUs. technique (line 3). Specifically, the fractional correction term In this regard, Fig. 2 compares the phase estimation errors δ, indicating the location of the actual signal frequency with obtained with a P-class and an M-class algorithm, and with P respect to the location of the highest amplitude bin km, is equal to 1 and 2. The test is performed with simulated wave- calculated as follows: forms in steady-state test condition. Specifically, the amplitude

2 · |X(km + ε)| − |X(km)| and the initial phase are equal to 1 pu and 0 respectively, the δ = ε · (3) frequency varies between 47.5 and 52.5 Hz, i.e., within the |X(km + ε)| + |X(km)| PMU pass-bandwidth considering the nominal frequency at The latter is used to estimate the fundamental component 50 Hz and the reporting rate of 50 fps. In order to reproduce parameters based on the following expressions: measurement noise coherent with the signals experimentally ˆ acquired in Section V, the waveforms are corrupted by an f = (km + δ)∆f (4) additive uncorrelated white Gaussian noise, whose is ˆ πδ 2 A = |X(km)| δ − 1 (5) scaled to reproduce an overall signal-to-noise ratio (SNR) of sin(πδ) 85 dB. In this context, it is interesting to observe that for P- ϕˆ0 = ∠X(km) − πδ (6) class configuration the second iteration provides a significant being ∆f = 1/T the DFT frequency resolution. These performance enhancement in case of non-nominal frequency values enable the reconstruction of the component’s negative values, with a phase error not exceeding 5 µrad. For the M- − class configuration, already with P equal to 1 the phase error image Xb (k), whose analytic expression is known for the Hanning function (line 5). The negative image is subtracted does not exceed 4 µrad. from the original DFT bins, that now should account only for Considering the uncertainty balance in Eq. (1), we are able to quantify both ε and ε of our test-bed. The syn- the fundamental component’s positive image Xb +(k) (line 6). alg acq Finally, the IpDFT is applied to such spectrum, resulting in chrophasor estimation uncertainty εalg is rather constant in the an enhanced estimation of the fundamental tone parameters considered spectral bandwidth and lower than 5 µrad, whereas ˆ the measurement noise exceeds the quantization noise of the {f, A,ˆ ϕˆ0} (line 7). The e-IpDFT adopts a reporting rate Fr of 50 frames per second (fps) to report the synchrophasors. acquisition module and thus makes negligible its uncertainty It is worth observing that the compensation of the spectral contribution εacq. interference produced by the negative image of the fundamen- tal component can be repeated a predefined number of times B. The Free-Running Sampling Process P . In the PMU described in [14], the procedure was repeated Regardless of the adopted time dissemination technique, the only once. More recent findings have demonstrated that setting sampling process of the waveforms is free-running and the 6

2 UTC-time synchronization is achieved a posteriori . Specifi- WR-cRIO cally, at the FPGA level, we derive form the UTC-PPS signal a sub PPS square waveform (hereafter called subPPS), locked cRIO to the UTC-PPS and characterized by a frequency correspond- GPS sync ing to the PMU reporting rate Fr. The signal acquisition, the synchrophasor estimation, and the synchrophasor time- PPS10 MHz WRS- 3/18 stamping are triggered by the rising edge of such subPPS. PC However, there is no guarantee that the sampling process is Meinberg GPS180PEX locked to such subPPS signal: there must be an a posteriori time refinement. Fig. 3. The experimental WR network composed of a Meinberg GPS180PEX Specifically, two delays need to be compensated. The first card, a WR switch and a NI-cRIO integrating the WR-cRIO module (i.e., a one results from the fact that the sampling frequency might WR-PMU). drift from its nominal value, due to oscillator degradation or environment conditions variation (such as temperature). We ± 100 ns. That is to say that the NI GPS module provides measure this frequency drift over observation windows of M a continuous time reference characterized by a time polling samples, with M >> N (such as few seconds windows). If the resolution corresponding to the FPGA clock. The subPPS is sampling process was uniform, such window would account locked to the UTC-GPS. for an ideal amount of time MT . In real operating conditions, s The GPS module is coupled with a Trimble’s Bullet III GPS the actual difference between the time instant when the last receiver, an active GPS antenna with a high-gain preamplifier sample is acquired t and the time instant when the first M−1 (35 dB) and dual passband filters [40]. The preamplifier sample is acquired t , might differ from the ideal delay. The 0 enables preserving the GPS signal even for long cable lengths, clock drift is defined as the normalized difference between whereas the filters improve rejection to interfering radio sig- these two delays: nals and reliability. The antenna is mounted on the rooftop (tM−1 − t0) − MTs of DESL laboratory with a full-sky visibility and is coupled fD = (7) MTs to the module via a 30-meters RG-213 shielded cable. The Every time the clock drift is updated, the DFT frequency latter, introduces un unavoidable propagation delay of 5.05 resolution ∆f = 1/T can be adequately compensated as: ns/m, leading to 151.5 ns (suitably compensated). ∆f = ∆f(1 − f ) (8) c D D. PTP Time Synchronization ˆ and therefore the frequency estimation improved fc. The PTP-PMU is based on the cRIO-9039 controller, char- The second delay is due to the possible offset between acterized by a reconfigurable Xilinx Kintex-7 FPGA with the two clocks. Indeed, in ideal operating conditions, i.e., an on-board clock frequency of 40 MHz, 407600 flip-flops, if the sampling process was locked to the subPPS, the time 203800 look-up tables (LUTs), 16020 kbits of block RAM and delay between the rising edge of the subPPS tsubP P S and the 840 DSP slices [41]. The PTP distribution is achieved thanks time instant when the first sample of the related window is to the NI TimeSync library, that synchronizes the timekeeping acquired t0 would be exactly zero. In real operating conditions, clocks of the cRIO. The so-called hardware time-stamping there could be a delay that would result in bad initial phase enables to discipline the FPGA clock directly via the UTC- estimations. We measure this time delay at every subPPS and PTP reference. This enables to timestamp each tick of the 40 compensate for it by updating the estimated phase as follows: MHz clock with real-world time, accurate to within ± 1 µs. ˆ The TimeSynch library provides a continuous time reference, ϕˆ0,c =ϕ ˆ0 + 2πfc(t0 − tsubP P S) (9) however, since the FPGA clock is locked to the UTC-PTP, the resolution of time stamps corresponds to 25 ns. C. GPS Time Synchronization The UTC-PTP reference signal is acquired by connecting The GPS-PMU is based on the cRIO-9068 controller, em- point-to-point the three-speed RJ-45 Gigabit Ethernet Port to bedding a reconfigurable Xilinx Zynq 7020 FPGA with an on- a PTP master clock. The latter is the Network Time Server board clock frequency of 40 MHz, 106400 flip-flops, 53200 NTS 100 manufactured by Tekron [42]. The clock receives the look-up tables (LUTs), 4480 kbits of block RAM and 220 absolute time by a Trimble’s Bullet III GPS receiver, whose DSP slices (each one characterized by a 25 X 18 multiplier, characteristics have been already discussed in Section IV-C. an adder and an accumulator) [38]. The UTC-GPS signal is acquired by means of the NI 9467 GPS time-stamping and E. White Rabbit Time Synchronization synchronization module, that is directly coupled with the on- board FPGA clock [39]. This enables to timestamp each tick The WR-PMU setup is shown in Fig. 3, and is based on of the 40 MHz clock with real-world time, accurate to within the same hardware platform as the GPS-PMU, i.e., cRIO-9068. The WR-UTC signal is provided by the NI WR cRIO module, 2A different approach could be adopted, where the synchronization of the a standalone WR node which can be coupled with the NI cRIO sampling process to the UTC-time is performed on-line by means of dedicated phase-locked loop circuitry. However, that would make the overall system platforms to integrate the WR protocol [43]. The module is architecture more complex. equipped with a Xilinx Spartan-6 FPGA and can be used in 7

all operation modes defined by the WR protocol, i.e., grand- Free-running clock PI controller master, master or slave. Depending on the selected operation Second � counter mode, a different configuration of input and outputs shall be � adopted. A user programmable HDSUB-15 I/O module is Nanosecond + e + � - - × provided, that can acquire the 10 MHz and PPS inputs (in counter case of operating the node in grand-master mode) or any sort sign(e) Femtosecond of external trigger, as well as generate reference clock, PPS + + + Correction counter outputs or generic triggers. The module is also equipped with Correction Integrator a Small Form-factor Pluggable (SFP) cage, for disseminating counter WR messages over optic fiber transceivers. In particular, when Operations Implicit operations (counters update, over/underflow handling) operated in slave mode, such cage is used to connect the module to its master WR switch and to retrieve the time Fig. 4. The structure of the internal clock regulated by the PI controller. information. In the developed PMU, the module is operated in slave mode, and is connected point-to-point to a WR switch operated cRIO, characterized by a finite and deterministic time polling in grand-master mode. The switch is manufactured by Seven Solutions [44]. Ethernet frames are exchanged through 18 Algorithm 2 Retrieving the WR time. ports equipped with SFP sockets, connected directly to a 1: while True Xilinx Virtex-6 FPGA characterized by very low latency. An 2: Go to normal operation ARM CPU running Linux helps with less time-sensitive pro- 3: Start cesses like remote management and keeping the frame filtering 4: while Ttrig database in the FPGA up to date. The clocking resources 5: Wait for node start block contains PLLs for cleaning up and phase-compensating 6: end the system clock, as well as for generating the frequency- 7: Read UTC-WR offset clock. It provides deterministic delivery and a reliable 8: Idle communication using redundant network topology. It allows 9: end many hops (14 tested keeping subnanosecond accuracy). The NTP service, used to determine the absolute time and resolution, not appropriate for PMU applications. date at reboot, is provided by a Windows machine connected To overcome this hardware limitation, an additional internal point-to-point to the RJ-45 management port of the WR switch free-running clock is implemented. Such clock is disciplined via an Ethernet cable. The computer is equipped with a Mein- by the FPGA clock and is implemented at every tick, i.e., berg 180 PEX card that disciplines the system time as well as every 25 ns. As long as the UTC-WR is not updated, the free- the NTP service [45]. The card is coupled with an active GPS running clock governs the PMU time. Every time the UTC- receiver, mounted on the rooftop of DESL laboratory, via a 30- WR is acquired, the free-running clock is overwritten by the meters RG-213 shielded cable. The card compensates for the updated reference time. delay introduced by the cable (as already discussed in Section As it is known, the FPGA clock could drift even in the IV-C). The card further generates reference PPS and 10 MHz short interval between two consecutive triggers, biasing the signals, that are fed to the WR switch. attainable sub-nanosecond accuracy. Therefore, every time the Due to hardware limitations, the UTC-WR polling is limited UTC-WR is acquired, the deviation between the free-running by the module’s FPGA clock running at 50 kHz, therefore, clock and the UTC-WR is computed, and this error is used by the WR cRIO does not provide a continuous time reference. a PI controller to condition the free-running clock. Also, the UTC-WR reading introduces a deterministic delay, The structure of the free-running clock and its PI controller that needs to be compensated. The next paragraph describes is shown in Fig. 4: the PMU time is made of the second the implementation details that enable us to overcome these and nanosecond counters and the correction γ(n) is added two limitations. to a femtosecond counter at each tick. The PMU time is 1) On the Retrieving of the WR Time: To retrieve the UTC- therefore corrected only when this counter has an overflow or WR from the WR cRIO, we generate a trigger characterized by an underflow. The tuning of the PI controller has been done a frequency of 50 kHz, i.e., the maximum value attainable in empirically: the proportional coefficient Kp has been chosen the WR cRIO FPGA. The procedure illustrated in Algorithm 2 to average the error entering the PI controller over a period of is implemented to trigger the UTC-WR acquisition, to freeze 10 ms, which gives a very low jitter. The integrator is built as the time, and to acquire it. Specifically, when the state is a counter that is incremented or decremented according to the Wait for Node Start, the trigger is generated and the WR sign of the error. Its resolution has been set to 1 fs (i.e. the cRIO acquires the reference time. The UTC-WR is frozen highest possible), which allows to compensate the steady-state and acquired in the next states. Then the node is set in Idle error without introducing additional jitter. The implemented mode until the next trigger. The time acquisition process is internal clock is explained in Algorithm 3. not continuous but the UTC-WR is updated in a discrete It is worth noting that, due to hardware limitations, there is manner, determined by the trigger period Ttrig of 20 µs. This a delay in acquiring the UTC-WR. However, the use of the lower bound is limited by the FPGA integrated in the NI FPGA makes this delay time-deterministic (in the order of few 8

� PPS � Meinberg PPS PMU 10 VRMS OMICRON 300 VRMS clock 10 MHz � GPS169PCI Calib CMS356 PXI 6682 PXI � �

GPS PMU NI 9467 Δ�

Tekron sync PTP ������ NTS100 PMU

�����

� trig trig trig trig PPS Meinberg WR sync WR GPS180PEX 10 MHz Switch PMU WR cRIO Fig. 5. The structure of the internal conditioned clock together with the TWR.

Fig. 6. Measurement setup for the performance assessment. The PMU TABLE I calibrator generates user-defined test waveforms, that are amplified up to 300 PERFORMANCEOF TWR AND TPMU . V by the CMS 356 OMICRON amplifier, and then supplied to the three considered PMUs, relying on GPS, PTP and WR time dissemination. TWR TPMU Jitter [ns] 2.89 0.42

kHz, peak amplitude of 10 V, 0 rad phase and frequency microseconds) and, therefore, enables the free-running clock varying in the range [47.5, 52.5] Hz (i.e., the PMU passband to compensate for it (see Fig. 5). considering a nominal frequency at 50 Hz and the reporting In addition to greater resolution, the internal free-running rate of 50 fps). These signals are amplified by a CMS-356 clock also has much less jitter than the WR time, as shown OMICRON precision voltage and current amplifier, charac- in Table I. The calculation of jitter is done by computing the terized by an amplification gain of 30, and simultaneously acquired by the three PMUs under test [47]. The final wave- Algorithm 3 Internal free-running clock. forms are characterized by a signal-to-noise-ratio (SNR) of 85 1: if TWR(n) 6= TWR(n − 1) dB. It is worth pointing out that, as highlighted in Fig. 6, the 2: ε(n) = TWR(n) − TPMU (n) master clocks of the three PMUs and the one of the PMU Pn 3: γ(n) = Kpε(n) + 0 sign(ε(k)) calibrator use separate GPS receivers, thus guaranteeing the 4: end non correlation among the various absolute times. 5: T (n) = T (n) + ∆T + γ(n) PMU PMU As known, the uncertainty requirements are expressed in terms of TVE, Frequency Error (FE), and ROCOF Error time steps of the WR time T and the PMU clock T WR PMU (RFE). However, the analysis of amplitude and phase error (i.e., WR time together with the internal free-running clock) separately provides a deeper understanding about eventual between two successive triggers T . The trig error sources. More specifically, every inaccuracy related to is then computed with 1000 samples. The performance of this a poor time-synchronization of the PMU under test, expresses implementation, characterized by the jitter, is appropriate for itself in a phase error. Also, since the synchrophasor estimation a PMU application. algorithm and the hardware platform are identical for the three PMUs, the time synchronization protocol mainly affects the V. PERFORMANCE ASSESSMENT phase estimation. The performance of the described PMUs is assessed using the test-bed illustrated in Fig. 6, i.e., by means of the dedicated For each PMU configuration and for each nominal fre- PMU calibrator described in [12], that enables us to validate quency we evaluate the phase estimation accuracy over a test the conformity of the PMU under test with respect to the IEEE duration of 24 hours. We present the results by means of Std. C37.118 [1]. The calibrator, generates reference signals two performance indicators. First, we evaluate the cumulative whose true parameters are known with a TVE in the order of distribution function (CDF) of the unbiased normalized phase 3 10−4%, obtained in case of static signals. The true parame- errors . Second, we quantify the stability of the adopted time ters are determined by the well-known Levenberg-Marquardt dissemination technology computing the Allan deviation [48]. algorithm, based on a nonlinear least-squares method. Such procedure, described in [46], has been proven to provide a unique and robust solution within the whole range of static tests required by [1]. 3We normalize the phase error by its mean value calculated in the considered observation interval. By doing so, we focus our analysis on the The forward path of the calibrator generates a set of static standard deviation of the phase error. Indeed, any absolute phase discrepancy reference waveforms characterized by a sampling rate of 500 can be properly compensated at the PMU output. 9

100 -5 10 GPS GPS PTP PTP WR 10-6 WR 75 ) [rad] ( 10-7 50

CDF [%] 10-8

25 Allan deviation 10-9

10-10 0 1 2 3 4 -1 -0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.8 1 10 10 10 10 Normalized phase error [rad] 10-4 [s]

Fig. 7. P-class: phase error cumulative distribution functions as provided by Fig. 10. P-class: phase error Allan deviation as function of the time interval GPS (blue), PTP (red) and WR (green) PMUs over a 24-hour test. The test τ, for GPS (blue), PTP (red) and WR (green) PMUs over a 24-hour test. waveform consists of a single fundamental tone whose amplitude, frequency The test waveform consists of a single fundamental tone whose amplitude, and phase are set equal to 300 V, 50 Hz, and 0 rad, respectively. frequency and phase are set equal to 300 V, 50 Hz, and 0 rad, respectively.

100 10-5 GPS GPS PTP PTP WR 10-6 WR 75 ) [rad] ( 10-7 50

CDF [%] 10-8

25 Allan deviation 10-9

10-10 0 1 2 3 4 -1 -0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.8 1 10 10 10 10 Normalized phase error [rad] 10-4 [s]

Fig. 8. P-class: phase error cumulative distribution functions as provided by Fig. 11. P-class: phase error Allan deviation as function of the time interval GPS (blue), PTP (red) and WR (green) PMUs over a 24-hour test. The test τ, for GPS (blue), PTP (red) and WR (green) PMUs over a 24-hour test. waveform consists of a single fundamental tone whose amplitude, frequency The test waveform consists of a single fundamental tone whose amplitude, and phase are set equal to 300 V, 47.5 Hz, and 0 rad, respectively. frequency and phase are set equal to 300 V, 47.5 Hz, and 0 rad, respectively.

100 10-5 GPS GPS PTP PTP WR 10-6 WR 75 ) [rad] ( 10-7 50

CDF [%] 10-8

25 Allan deviation 10-9

10-10 0 1 2 3 4 -1 -0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.8 1 10 10 10 10 Normalized phase error [rad] 10-4 [s]

Fig. 9. P-class: phase error cumulative distribution functions as provided by Fig. 12. P-class: phase error Allan deviation as function of the time interval GPS (blue), PTP (red) and WR (green) PMUs over a 24-hour test. The test τ, for GPS (blue), PTP (red) and WR (green) PMUs over a 24-hour test. waveform consists of a single fundamental tone whose amplitude, frequency The test waveform consists of a single fundamental tone whose amplitude, and phase are set equal to 300 V, 52.5 Hz, and 0 rad, respectively. frequency and phase are set equal to 300 V, 52.5 Hz, and 0 rad, respectively. 10

100 10-5 GPS GPS PTP PTP WR WR 10-6 75 ) [rad] ( 10-7 50 -8 CDF [%] 10

25 Allan deviation 10-9

10-10 0 1 2 3 4 -1 -0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.8 1 10 10 10 10 [s] Normalized phase error [rad] 10-4

Fig. 13. M-class: phase error cumulative distribution function over a 24-hour Fig. 14. M-class: phase error Allan deviation over a 24-hour test, for GPS test, for GPS (blue), PTP (red) and WR (green) PMUs. The test waveform (blue), PTP (red) and WR (green) PMUs. The test waveform consists of a consists of a single fundamental tone whose amplitude, frequency and phase single fundamental tone whose amplitude, frequency and phase are set equal are set equal to 300 V, 50 Hz, and 0 rad, respectively. to 300 V, 50 Hz, and 0 rad, respectively.

To this end, we consider the M-sample variance, defined as: In the same test conditions, Fig. 10, 11 and 12 evaluate the Allan deviation as function of the time interval τ. Co- M−1 M−1 !2 1 X 1 X herently with the previous results, we notice how the WR is σ2(τ) = δ2 (m, τ) − δ (m, τ) M − 1  ϕ M ϕ  characterized by the lowest variability, whereas PTP and GPS m=0 m=0 provide comparable performance. For instance, at 50 Hz the ϕ(mT + τ) − ϕ(mT ) WR Allan deviation decreases from 0.5 µrad up to 0.7 nrad, δ (m, τ) = r r (10) ϕ τ if we enlarge the time interval from 101 up to 104 s. This performance enhancement provided by the WR time dissem- where ϕ(mTr) is the phase estimate associated to mTr time ination becomes more significant as τ increases, particularly instant, expressed as function of the reporting period Tr, M is the sample number for the variance computation, and τ is the when asynchronous sampling conditions are considered. time deviation between two consecutive phase estimates. The Allan variance refers to the specific case where M and Tr are B. Normal Operating Conditions, M-class set equal to 2 and τ, respectively, and the Allan deviation is its Given a fundamental frequency of 50 Hz, Fig. 13 and square root [48]. We evaluate the phase estimation accuracy 14 show the CDF and the Allan deviation for the M-class over different time intervals, varying τ between 101 to 104 s. configuration, respectively. The choice of limiting the analysis Our analysis has been conducted by coupling for several to a synchronous sampling condition enables us to limit the days the three devices with the PMU calibrator and in the uncertainty coming from the synchrophasor extraction process following paragraphs, we present the results obtained for three and focus primarily on the stability of the time synchronization different scenarios. The first two paragraphs are meant to source. evaluate the performance during normal operating conditions As expected, the M-class configuration provides better per- and refer to P- and M-class PMUs, respectively. The third formance than the P-class one, leading to errors roughly 1 paragraph instead refers to P-class PMUs during the worst- µrad lower for every considered timing technology. As shown case condition that has been recorded over various tests and in Fig. 13 the distribution of the errors is in this case sharper is meant to assess the maximum phase uncertainty that can be and less disperse than the results presented in the previous introduced by GPS, PTP and WR synchronization schemes. Section V-A. As regards the Allan deviation, the WR still provides enhanced stability, over any of the considered time A. Normal Operating Conditions, P-class intervals. In the first test, we compare the phase estimation accuracy of P-class PMUs obtained in normal operating conditions as C. Worst-Case Operating Conditions, P-class function of the fundamental frequency. Specifically, Fig. 7, 8 In the third scenario, we extract the worst-case performance and 9 present the phase error CDFs for 50, 47.5 and 52.5 Hz, associated to each time dissemination and compare them in respectively. Independently from the fundamental frequency order to experimentally determine the accuracy limit provided values, the WR enables us to keep the normalized phase error by GPS, PTP and WR-PMUs. As in Section V-B, we limit our within ±15 µrad, whereas PTP and GPS might exceed 30 analysis to coherent sampling, i.e., we keep the fundamental µrad. It is also worth observing that the GPS tends to outper- frequency to the nominal value of 50 Hz. form the PTP and this performance discrepancy becomes more As shown in Fig. 15, the three PMUs are characterized evident, as we consider non-nominal test conditions, when the by different trends of the distribution of the absolute phase sampling rate is not locked to the fundamental frequency. error. As expected, the PTP-PMU is characterized by the most 11

disperse distribution, with a standard deviation of 26 µrad. TABLE II The GPS and PTP-PMUs are characterized by non-symmetric PHASE ERROR [µRAD]-STATISTICAL DISTRIBUTION FEATURES tails and a non-null mean value, because the time evolution of Normal Worst-Case the phase errors is characterized by a non-symmetric trend P-class M-class P-class with respect to the respective mean value. The WR-PMU 47.5 Hz 50 Hz 52.5 Hz 50 Hz 50 Hz instead always reports a symmetric behavior, thus leading to min -85.5 -90.8 -97.5 -28.9 -84.7 a balanced CDF. Finally, the WR-PMU exhibits the sharpest GPS max 68.8 85.9 109.2 31.9 92.0 CDF trend with a standard deviation of 8 µrad, demonstrating std 11.3 7.4 13.3 6.4 18.1 once again that such synchronization technique is the most min -59.1 -40.4 -64.1 -31.6 -80.1 deterministic one. PTP max 49.0 32.6 57.8 27.5 55.7 std 12.3 8.2 15.7 7.2 25.9 In general, the results in Fig. 15 reflect the accuracy min -22.2 -25.0 -36.3 -21.1 -27.9 specification of the adopted time synchronization techniques, WR max 24.6 26.7 31.6 22.9 43.6 in the sense that the lower the accuracy of the timing module, std 5.9 5.8 7.1 5.1 8.1 the more disperse the phase error distribution. As discussed in Section IV-A, the error introduced by the algorithm is dominating and masking the potential improvement of the interesting to observe that GPS is typically characterized by phase estimate given by the WR technology. Nevertheless, an a lower standard deviation, but a larger min-max range than improvement of 10 µrad is achieved for the WR-PMU with PTP. This phenomenon is due to the fact that even if GPS respect to the GPS counterpart. estimates are characterized by a reduced variability, they might Similar considerations are valid also for the Allan deviations present sudden variations or outliers that affect the definition presented in Fig. 16: independently from the considered time of maximum and minimum error. interval τ, the WR confirms to be characterized by a lower 4 phase variability (equal to 1 nrad at 10 s), whereas PTP and VI.CONCLUSION GPS provide nearly coincident performance. The paper presented the use of the WR time synchronization Finally, in Table II, we report the main features of the protocol for synchrophasor networks. The WR is characterized phase error statistical distributions obtained in the different by a time accuracy of 1 ns, that is superior to those of operating conditions and PMU configurations. For each time state-of-the-art time dissemination technologies used for PMU dissemination technique and fundamental frequency value, we applications, i.e., 50 ns for GPS and 1 µs for PTP. compute the minimum and maximum phase error, as well as The IEEE Std. C37.118.1 requires a maximum synchroniza- its standard deviation. In all the considered configurations, the tion uncertainty of 1 µs for PMUs operating in transmission WR outperforms PTP and GPS, with a worst-case standard networks, but this value is lowered to 10 ns for distribution deviation of 8.1 µrad. In this regard, it is worth noticing PMUs. Therefore, the WR is a suitable time distribution how the WR synchronization produces a phase variability that technique for PMUs operating at any power system level. is comparable with the synchrophasor accuracy limit (i.e., 5 The paper has presented the integration of the WR protocol µrad for P-class and 4 µrad for M-class). In other words, the in a specifically developed WR-PMU, and has assessed its WR-PMU is capable of minimizing the time dissemination performance with respect to a GPS-PMU and a PTP-PMU. uncertainty contribution and thus optimizing the performance The three PMUs are characterized by the same synchrophasor of the actual synchrophasor estimation algorithm. It is also estimation algorithm and by the same hardware platform,

100 10-5 GPS GPS PTP PTP WR -6 WR 75 10 ) [rad] ( 10-7 50

CDF [%] 10-8

25 Allan deviation 10-9

0 10-10 -1 -0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.8 1 101 102 103 104 Normalized phase error [rad] 10-4 [s]

Fig. 15. Worst-case scenario: phase error cumulative distribution function over Fig. 16. Worst-case scenario: phase error Allan deviation over a 24-hour test, a 24-hour test, for GPS (blue), PTP (red) and WR (green) PMUs. The test for GPS (blue), PTP (red) and WR (green) PMUs. The test waveform consists waveform consists of a single fundamental tone whose amplitude, frequency of a single fundamental tone whose amplitude, frequency and phase are set and phase are set equal to 300 V, 50 Hz, and 0 rad, respectively. equal to 300 V, 50 Hz, and 0 rad, respectively. 12

with the exception of the time synchronization technique. The [22] M. Lixia, A. Benigni, A. Flammini, C. Muscas, F. Ponci, and A. Monti, results demonstrate the advantage of using the WR instead of “A software-only PTP synchronization for power system state estimation with PMUs,” IEEE Transactions on Instrumentation and Measurement, GPS, as it is characterized by a more deterministic phase error, vol. 61, no. 5, pp. 1476–1485, May 2012. experimentally quantified in 8 µrad. [23] CERN, “The White Rabbit Project,” 2017, accessed: 2018-08-10. [Online]. Available: http://white-rabbit.web.cern.ch/ [24] F. Ramos, J. L. Gutierrez-Rivas,´ J. Lopez-Jim´ enez,´ B. Caracuel, and REFERENCES J. D´ıaz, “Accurate timing networks for dependable smart grid applica- tions,” IEEE Transactions on Industrial Informatics, vol. 14, no. 5, pp. [1] “IEEE standard for synchrophasor measurements for power systems,” 2076–2084, May 2018. IEEE Std C37.118.1-2011 (Revision of IEEE Std C37.118-2005), pp. [25] J. L. Gutierrez-Rivas,´ J. Lopez-Jim´ enez,´ E. Ros, and J. D´ıaz, “White 1–61, Dec 2011. rabbit HSR: A seamless subnanosecond redundant timing system with [2] M. Pignati, L. Zanni, S. Sarri, R. Cherkaoui, J.-Y. Le Boudec, and low-latency data capabilities for the smart grid,” IEEE Transactions on M. Paolone, “A pre-estimation filtering process of bad data for linear Industrial Informatics, vol. 14, no. 8, pp. 3486–3494, Aug 2018. power systems state estimators using PMUs,” in Power Systems Com- [26] D. M. Anand, K. G. Brady, C. Nguyen, E. Song, K. Lee, Y. Li-Baboud, putation Conference (PSCC), 2014, Aug 2014, pp. 1–8. A. Goldstein, and G. FitzPatrick, “Measurement tools for substation [3] “Time Synchronization in the Electric Power System,” NASPI Time equipment: Testing the interoperability of protocols for time transfer and Synchronization Task Force, 2017. communication,” in 2018 IEEE International Symposium on Precision [4] “Synchrophasor Monitoring for Distribution Systems: Technical Foun- Clock Synchronization for Measurement, Control, and Communication dations and Applications,” NASPI Distribution Task Team, 2018. (ISPCS), Sep. 2018, pp. 1–6. [5] S. Barreto, M. Pignati, G. Dan, M. Paolone, and J.-Y. Le Boudec, [27] E. F. Dierikx, A. E. Wallin, T. Fordell, J. Myyry, P. Koponen, M. Mer- “Undetectable PMU timing-attack on linear state-estimation by using imaa, T. J. Pinkert, J. C. J. Koelemeij, H. Z. Peek, and R. Smets, rank-1 approximation,” IEEE Transactions on Smart Grid, vol. PP, “White rabbit precision time protocol on long-distance fiber links,” IEEE no. 99, pp. 1–1, 2017. Transactions on Ultrasonics, Ferroelectrics, and Frequency Control, [6] J. Serrano, P. Alvarez, M. Cattin, E. G. Cota, P. M. J. H. Lewis, vol. 63, no. 7, pp. 945–952, July 2016. T. Włostowski et al., “The white rabbit project,” in Proceedings of [28] G. Fantino, G. Cerretto, and C. D. Costa, “White rabbit time transfer ICALEPCS TUC004, Kobe, Japan, 2009. on medium and long fibre hauls at INRIM,” in Proceedings of the [7] M. Lipinski,´ T. Włostowski, J. Serrano, and P. Alvarez, “White rabbit: 46th Annual Precise Time and Time Interval Systems and Applications a PTP application for robust sub-nanosecond synchronization,” in 2011 Meeting, Boston, Massachusetts, Dec 2014, pp. 45–51. IEEE International Symposium on Precision Clock Synchronization for [29] H. Li, G. Gong, W. Pan, Q. Du, and J. Li, “Temperature effect on white Measurement, Control and Communication, 2011, p. 25–30. rabbit timing link,” IEEE Transactions on Nuclear Science, vol. 62, [8] M. Lipinski,´ E. van der Bij, J. Serrano, T. Włostowski, G. Daniluk, no. 3, pp. 1021–1026, June 2015. A. Wujek, M. Rizzi, and D. Lampridis, “White rabbit applications and [30] “Copper SFP modules,” accessed: 2019-05-05. [Online]. Available: enhancements,” in 2018 IEEE International Symposium on Precision https://www.ohwr.org/project/white-rabbit/wikis/CopperSFP Clock Synchronization for Measurement, Control, and Communication [31] M. Rizzi, M. Lipinski,´ P. Ferrari, S. Rinaldi, and A. Flammini, “White (ISPCS), Sep. 2018, pp. 1–7. rabbit clock synchronization: Ultimate limits on close-in phase noise and [9] “IEEE Standard for a Precision Clock Synchronization Protocol for short-term stability due to FPGA implementation,” IEEE Transactions Networked Measurement and Control Systems,” IEEE Std 1588-2008 on Ultrasonics, Ferroelectrics, and Frequency Control, vol. 65, no. 9, (Revision of IEEE Std 1588-2002), pp. 1–300, July 2008. pp. 1726–1737, Sep. 2018. [10] “IEEE Standard for Ethernet,” IEEE Std 802.3-2015 (Revision of IEEE [32] O. Ronen and M. Lipinski,´ “Enhanced synchronization accuracy in Std 802.3-2012), pp. 1–4017, March 2016. IEEE1588,” in 2015 IEEE International Symposium on Precision Clock [11] “Timing Characteristics of Synchronous Ethernet Equipment Slave Synchronization for Measurement, Control, and Communication (IS- Clock (EEC),” ITU-T Rec. G.8262, August 2007. PCS), Oct 2015, pp. 76–81. [12] G. Frigo, A. Derviskadiˇ c,´ D. Colangelo, J.-P. Braun, and M. Paolone, [33] G. Daniluk and T. Włostowski, “White rabbit: Sub-nanosecond syn- “Characterization of uncertainty contributions in a high-accuracy PMU chronization for embedded systems,” in Proceedings of the 43rd Annual validation system,” Measurement, vol. 146, pp. 72 – 86, 2019. Precise Time and Time Interval Systems and Applications Meeting, Nov [13] R. Razzaghi, A. Derviskadiˇ c,´ and M. Paolone, “A white rabbit synchro- 2011. nized PMU,” in 2017 IEEE PES Innovative Smart Grid Technologies [34] “National Instruments CompactRIO Controller,” accessed: 2019-05-05. Conference Europe (ISGT-Europe), Sept 2017, pp. 1–6. [Online]. Available: http://www.ni.com/en-us/shop/select/compactrio- [14] P. Romano and M. Paolone, “Enhanced interpolated-DFT for syn- controller chrophasor estimation in FPGAs: Theory, implementation, and valida- [35] “National Instruments NI 9225 C Series Voltage Input Module,” tion of a PMU prototype,” IEEE Transactions on Instrumentation and accessed: 2019-05-05. [Online]. Available: http://www.ni.com/en- Measurement, vol. 63, no. 12, pp. 2824–2836, Dec 2014. us/support/model.ni-9225.html [15] E. Costa, “Simulation of the effects of different urban environments on [36] “National Instruments NI 9227 C Series Current Input Module,” GPS performance using digital elevation models and building databases,” accessed: 2019-05-05. [Online]. Available: http://www.ni.com/en- IEEE Transactions on Intelligent Transportation Systems, vol. 12, no. 3, us/support/model.ni-9227.html pp. 819–829, Sept 2011. [37] A. Derviskadiˇ c,´ P. Romano, and M. Paolone, “Iterative-interpolated [16] X. Jiang, J. Zhang, B. J. Harding, J. J. Makela, A. D. Domı et al., DFT for synchrophasor estimation: A single algorithm for P- and M- “Spoofing GPS receiver clock offset of phasor measurement units,” IEEE class compliant PMUs,” IEEE Transactions on Instrumentation and Transactions on Power Systems, vol. 28, no. 3, pp. 3253–3262, 2013. Measurement, vol. 67, no. 3, pp. 547–558, March 2018. [17] D. Mills, J. Martin, J. Burbank, and W. Kasch, “Network time protocol [38] “National Instruments CompactRIO 9068 Controller,” accessed: 2019- version 4: Protocol and algorithms specification,” Tech. Rep., 2010. 05-05. [Online]. Available: http://www.ni.com/en-us/support/model.crio- [18] S. Barreto, A. Suresh, and J.-Y. Le Boudec, “Cyber-attack on packet- 9068.html based time synchronization protocols: The undetectable delay box,” [39] “National Instruments NI 9467 C Series Synchronization Module,” 2016 IEEE International Instrumentation and Measurement Technology accessed: 2019-05-05. [Online]. Available: http://www.ni.com/en- Conference Proceedings, pp. 1–6, 2016. us/support/model.ni-9467.html [19] “IEEE Standard Profile for Use of IEEE 1588 Precision Time Protocol [40] “Trimble Bullet III GPS Antenna,” accessed: 2019-05-05. [Online]. in Power System Applications,” IEEE Std C37.238-2017 (Revision of Available: https://www.trimble.com/timing/bullet-gps-antenna.aspx IEEE Std C37.238-2011), pp. 1–42, June 2017. [41] “National Instruments CompactRIO 9039 Controller,” accessed: 2019- [20] H. F. Albinali and A. P. S. Meliopoulos, “Resilient protection system 05-05. [Online]. Available: http://www.ni.com/en-us/support/model.crio- through centralized substation protection,” IEEE Transactions on Power 9039.html Delivery, vol. 33, no. 3, pp. 1418–1427, June 2018. [42] “Tekron NTS100,” accessed: 2019-05-05. [Online]. [21] P. Castello, P. Ferrari, A. Flammini, C. Muscas, and S. Rinaldi, “A Available: https://tekron.com/news/release/tekron-nts100-network-time- new IED with PMU functionalities for electrical substations,” IEEE server-new-product Transactions on Instrumentation and Measurement, vol. 62, no. 12, pp. [43] “CompactRIO White Rabbit (CRIO-WR),” accessed: 2019-05-05. 3209–3217, Dec 2013. [Online]. Available: https://www.ohwr.org/projects/crio-wr/wiki 13

[44] “Seven Solutions White Rabbit Switch,” accessed: 2019-05-05. [Online]. Mario Paolone (M’07, SM’10) received the M.Sc. Available: http://sevensols.com/index.php/products/white-rabbit-switch/ (with Hons.) and Ph.D. degrees in electrical en- [45] “Meinberg GPS180PEX: Low Profile GPS gineering from the University of Bologna, Italy, Clock,” accessed: 2019-05-05. [Online]. Available: in 1998 and 2002, respectively. In 2005, he was https://www.meinbergglobal.com/english/products/pci-express-gps- appointed Assistant Professor in power systems with clock.htm the University of Bologna, where he was with the [46] G. Frigo, D. Colangelo, A. Derviskadiˇ c,´ M. Pignati, C. Narduzzi, power systems laboratory until 2011. In 2010, he and M. Paolone, “Definition of accurate reference synchrophasors for received the Associate Professor eligibility from the static and dynamic characterization of PMUs,” IEEE Transactions on Polytechnic of Milan, Italy. Since 2011, he joined Instrumentation and Measurement, vol. 66, no. 9, pp. 2233–2246, Sept the Swiss Federal Institute of Technology, Lausanne, 2017. Switzerland, where he is currently Full Professor, [47] “CMS 356 Voltage and current amplifier,” accessed: 2019-05-05. Chair of the Distributed Electrical Systems Laboratory, Head of the Swiss [Online]. Available: https://www.omicronenergy.com/en/products/cms- Competence Center for Energy Research Future Swiss Electrical infrastructure 356/ and Chair of the EPFL Energy Centre Directorate. He has authored or co- [48] D. W. Allan, “Statistics of atomic frequency standards,” Proceedings of authored over 300 papers published in mainstream journals and international the IEEE, vol. 54, no. 2, pp. 221–230, Feb 1966. conferences in the area of energy and power systems. His research interests focus on power systems with particular reference to real-time monitoring and operation aspects, power system protections, dynamics and transients. Dr. Paolone is the Editor-in-Chief of the Elsevier journal Sustainable Energy, Grids and Networks.

Asja Derviskadiˇ c´ (M’15) was born in Sarajevo, Bosnia and Herzegovina, in 1990. She received the B.Sc. and M.Sc. degrees (Hons.) in electrical engi- neering from the University of Rome “La Sapienza,” Rome, Italy, in 2012 and 2015, respectively. She is currently pursuing the Ph.D. degree with the Distributed Electrical Systems Laboratory (DESL) of the Swiss Federal Institute of Technology of Lausanne (EPFL), Switzerland. Her research inter- ests include the development of enhanced Phasor Measurements Units (PMUs) for active distribution networks.

Reza Razzaghi (M’10) received the Ph.D. degree in electrical engineering from the Swiss Federal Institute of Technology of Lausanne (EPFL), Lau- sanne, Switzerland in 2016. In 2017, he joined Monash University, Melbourne, Australia, where he is currently a Lecturer (Assistant Professor) with the Department of Electrical and Computer Systems En- gineering. His research interests include power sys- tem protection and control, synchrophasor networks, and electromagnetic transients in power systems. He has been the recipient of multiple awards including the 2018 Best Paper Award of the IEEE Transactions on EMC and the 2013 Basil Papadias Best Paper Award from the IEEE PowerTech Conference.

Quentin Walger (S’19) was born in 1995. He received the B.Sc. degree in electrical engineering from the Swiss Federal Institute of Technology of Lausanne (EPFL), Switzerland, in 2017. He is currently pursuing his M.Sc. degree in electrical engineering from EPFL and collaborating with the Distributed Electrical Systems Laboratory (DESL). His fields of interest include power system monitor- ing and control.