DEGREE PROJECT IN ELECTRICAL ENGINEERING, SECOND CYCLE, 30 CREDITS STOCKHOLM, SWEDEN 2017
Time Synchronization for Ethernet Communication based Line Differential Protection
SURYA SEETHARAMAN
KTH ROYAL INSTITUTE OF TECHNOLOGY SCHOOL OF ELECTRICAL ENGINEERING Abstract
Most of the industrial manufactures and designers of Power Substation Automation Protec- tion Control products are under pressure regarding the existing communication architecture of the Line Differential Protection Application. Currently the widely used technologies include SONET/SDH which are circuit switched networks, that are no longer predominant. They are been replaced with Packet Switched Networks everywhere. Moreover dedicated network equip- ment is required to be maintained for using SDH/SONET technologies. Hence it is becoming increasingly hard for the utilities to get access to and maintain the SDH/SONET networks.
The main objective of this thesis work is to develop an alternative communication solution in order to migrate away from these SDH/SONET networks. The project work is done such that the requirements for the Line Differential Protection Communication are carefully analysed and a solution is designed such that it fulfils those requirements. Following that the imple- mentation and evaluation of the solution is performed. In the end, a testing of this solution is also done on a real-time WAN network running the Line Differential Protection function, in order to verify the working of the solution and study the challenges of a WAN deployment scenario in future.
i Abstrakt
De flesta industriella tillverkare och designers till produkter av system f¨orstationsautomation ¨ar under press ang˚aendeden existerande kommunikationsarkitekturen f¨orLinjedifferentialskydds- till¨ampningar. De f¨orn¨arvarande utbredda teknologierna som anv¨ands,som SONET/SDH, ¨ar kretskopplade n¨atverkvilket inte l¨angre¨arden dominerande n¨atverksprincipen. Dessa ¨arp˚av¨ag att bli ersatta av paketf¨ormedlade n¨atverk.Dessutom kr¨avsdet att dedikerad n¨atverksutrust- ning m˚aste underh˚allasf¨oratt anv¨andasmed SONET/SDH-teknologier. D¨arf¨orb¨orjar det bli kr¨avandeoch sv˚artf¨orverktygen att ges ˚atkomst samt underh˚allaSONET/SDH-n¨atverk.
Huvudm˚aletmed denna uppsats ¨ar att utveckla en alternativ kommunikationsl¨osningi syfte att byta ut dessa SDH/SONET-n¨atverk.Arbetet ¨arutf¨ortgenom att f¨orstnoggrant analysera kraven f¨orLinjedifferentialskydds-kommunikation f¨oratt sedan finna och designa en l¨osning som uppfyller dessa krav. D¨arefterf¨oljeren implementation och utv¨arderingav den funna l¨osningen. I sista fasen genomf¨orsen testning av l¨osningeni ett realtids WAN-n¨atverksom k¨orLinjedifferentialskydds-funktionen, detta f¨oratt verifiera hur den funna l¨osningenfungerar samt att studera utmaningar f¨oren framtida WAN implementation.
ii Acknowledgement
This thesis work has been conducted in collaboration with ABB Substation Automation Prod- ucts in V¨aster˚asas well as with the Industrial Information and Control Systems department at KTH in Stockholm. This project would not have been possible without a lot of people for whom I hold a great amount of respect. Hence I would like to express my gratitude towards them and acknowledge their roles in making this thesis work happen.
Johan S¨aljhas been my supervisor at ABB. He has been a pillar of great support throughout the thesis project specially when it came to technical design and guidance. Bjorn Lexelius from ABB has also been very helpful and skilled when it comes to the IED communication area. I am very thankful to both of them for sharing their broad expertise and for supervising my thesis in a systematic and dedicated manner. Moreover, it has been a great pleasure, working in the team of Henrik B¨ackstrand. In his capacity as a line manager he has helped me with all administration matters as well as with the arranging of the required technical equipment. His timely advice has always kept me motivated.
Furthermore, my supervisor from KTH Fabian Hohn has been really impressive and of note- worthy help to me. During the course of my thesis work, he has always been there - giving suggestions for improvisation, clarifying doubts in case I got stuck on technical aspects, en- suring I stay on track with the project. Most importantly he has constantly encouraged me to do well right from the beginning. Professor Lars Nordstr¨omwho is head of the department of Electric Power and Energy Systems at KTH was acting as the responsible examiner of the thesis. I would like to thank him for having given me this great opportunity to conduct the thesis with ABB.
Lastly, I would like to thank all my colleagues from ABB, friends from KTH and my lov- ing family in India for supporting and encouraging me throughout.
iii Contents
Abstract i
Abstrakt ii
Acknowledgement iii
List of Figures viii
List of Tables x
Nomenclature xi
1 Introduction 1 1.1 Problem Statement ...... 1 1.2 Problem Description ...... 1 1.3 Project Goals and Scope ...... 2 1.4 Project Outline ...... 2
2 Background 4 2.1 Line Differential Protection ...... 4 2.1.1 Working Principle ...... 4 2.1.2 Requirements of the Protection Interface ...... 5 2.2 Existing Industrial Solutions : SDH/SONET ...... 5 2.3 Motivation for IP-based Ethernet Networks ...... 6 2.4 Time Synchronization ...... 7 2.4.1 Inter-Range Instrumentation Group - B ...... 8 2.4.2 Network Time Protocol ...... 8 2.4.3 SyncE ...... 9 2.5 Precision Time Protocol ...... 9 2.5.1 Protocol Details ...... 10 2.5.1.1 PTP devices ...... 10 2.5.1.2 PTP Messages ...... 11 2.5.1.3 PTP Communication Path ...... 12
iv 2.5.1.4 PTP Delay Mechanisms ...... 12 2.5.2 Working Principle ...... 17 2.5.2.1 Best Master Clock Algorithm ...... 18 2.5.2.2 Synchronization ...... 19 2.5.3 PTP Profiles ...... 20 2.5.3.1 PC37.238 IEEE-1588 : PTP Power Profile ...... 20 2.5.3.2 IEC/IEEE 61850-9-3:2016 : PTP Power Utility Profile . . . . 21 2.5.3.3 IEEE G.8275.1/G.8275.2 : PTP Telecom Profile ...... 21 2.6 PTP versus existing methods ...... 22
3 Methodology 24 3.1 Proposed Solution ...... 24 3.2 Satisfying the Requirements of the Protection Interface ...... 24 3.3 Approach ...... 25 3.3.1 Communication : Packet Switched Networks ...... 25 3.3.2 Synchronization : Precision Time Protocol ...... 25
4 Implementation 26 4.1 Resources Used ...... 26 4.1.1 Hardware Network Components ...... 26 4.1.2 Software Tools ...... 32 4.2 Network Topologies ...... 35 4.2.1 N1: Point to Point Connection between two IEDs ...... 35 4.2.2 N2: Introducing L2 switches between the IED’s ...... 37 4.2.3 N3: Three Ended Application - GPS to IED ...... 38 4.2.4 N4: Three Ended Application - GPS to switch ...... 39 4.2.5 N5: Hardware-in-the-Loop Trial with OMNET++ ...... 40 4.2.6 N6: Unlocked from GPS time reference ...... 41 4.3 Measurement Conditions ...... 42 4.3.1 Network Device Settings ...... 42 4.3.1.1 IED Configuration ...... 42 4.3.1.2 Switch Configuration ...... 43 4.3.1.3 GPS Clock Configuration ...... 43 4.3.2 Oscilloscope Settings ...... 43 4.3.3 Differential Current Settings ...... 43
5 Evaluation 44 5.1 E1: Functionality Testing ...... 44 5.1.1 Time Synchronization using PTP ...... 44 5.1.2 Line Differential Protection Communication over Ethernet ...... 44 5.2 E2: Performance Testing ...... 46
v 5.2.1 Time to Synchronize at Start-Up ...... 46 5.2.2 Time to Re-Synchronize ...... 46 5.2.3 Time synchronization Accuracy ...... 48 5.2.3.1 N1 : For the point to point Set-Up ...... 48 5.2.3.2 N2 : For introduction of L2 switches Set-Up ...... 49 5.2.3.3 N3 : Three-ended Application Set-Up ...... 50 5.2.3.4 N4 : Three-ended Application Set-Up with GPS to switch . . 51 5.2.3.5 N6 : Unlocked to UTC ...... 52 5.3 E3: Stress Testing ...... 54 5.3.1 Effect on Application Communication ...... 56 5.3.2 Effect on PTP Communication ...... 57
6 Feasibility for WAN Deployment 61 6.1 WAN Description and Set-up ...... 61 6.2 Testing and Results ...... 62 6.2.1 Case 1 : With the ABB v1.2 IEDs ...... 63 6.2.2 Case 2 : With the ABB v2.2 IEDs ...... 64 6.2.2.1 Case 2.1 : Data and Sync messages over Primary route . . . 66 6.2.2.2 Case 2.2 : Data over Secondary and Sync over Primary routes 66 6.2.2.3 Case 2.3 : Data and Sync messages over Secondary Route . . 67 6.3 Challenges during Testing ...... 69 6.4 Customer Requirements after the Testing ...... 70 6.5 General Industry Challenges for WAN Deployment ...... 70
7 Conclusion and Discussion 72 7.1 Conclusion ...... 72 7.2 Future Work ...... 72
Bibliography 73
Appendix 73
vi List of Figures
2.1 Line Current Differential Protection [3] ...... 4 2.2 PTP Clocks [16] ...... 11 2.3 PTP End-to-End Delay Mechanism [19] ...... 13 2.4 Hardware Timestamping during the transport of a Sync message [19] . . . . . 14 2.5 Working of peer-delay messages in Peer-to-Peer Delay Mechanism [20] . . . . . 15 2.6 PTP Peer-to-Peer Delay Mechanism [20] ...... 15 2.7 PTP Working Principle [16] ...... 17 2.8 PTP BMC Working ...... 19
4.1 RED 670 IEDs ...... 27 4.2 LDCM, IRIG-B and GTM Modules (left to right order) ...... 28 4.3 AFS 660 switch ...... 29 4.4 GPS clock and AFS677 switch ...... 29 4.5 GPS Antenna ...... 29 4.6 GPS clock signal relied to all network nodes in the lab ...... 30 4.7 Oscilloscope ...... 30 4.8 Communication Media ...... 31 4.9 Single Phase Relay Tester ...... 32 4.10 ABB AFS Finder 2.2.0.7 ...... 33 4.11 Wireshark capturing PTP sync messages ...... 34 4.12 Sender Application ...... 35 4.13 Point to Point set-up between two IEDs ...... 36 4.14 Introducing PTP compatible L2 switches ...... 37 4.15 Three ended application ...... 38 4.16 Three ended application with the GPS signal provided directly to the network . 39 4.17 HiL Network Setup ...... 40 4.18 Unlocked to UTC with IED acting as a PTP Master ...... 41
5.1 Functionality Testing ...... 45 5.2 Time required for Re-synchronization ...... 47
vii 5.3 Density Plot for the PTP Slave’s deviation or time sync accuracy from PTP Master ...... 49 5.4 Density Plot for the PTP Slave’s deviation or time sync accuracy from PTP Master ...... 50 5.5 Density Plot for both the PTP Slaves’ deviation or time sync accuracy from PTP Master ...... 51 5.6 Density Plot for the three PTP Slaves’ deviation or time sync accuracy from PTP Master ...... 52 5.7 Density Plot for the two PTP Slaves’ deviation or time sync accuracy from PTP Master ...... 53 5.8 Topology for Stress Testing ...... 54 5.9 Measuring Line Differential Current during Stress Testing ...... 57 5.10 Measuring Time Sync Accuracy during Stress Testing ...... 58 5.11 Time Synchronization Accuracy measured during stress testing ...... 60
6.1 WAN test network ...... 61 6.2 WAN set-up for v1.2 IEDs testing ...... 63 6.3 WAN set-up for v2.2 IEDs testing ...... 65 6.4 WAN set-up for v2.2 IEDs testing ...... 69
viii List of Tables
2.1 PTP Message Types ...... 12 2.2 Comparison of Time Sync Mechanisms ...... 22
5.1 Diff and Bias currents measured from N4 set-up...... 45 5.2 Time to synchronize to 1 µs accuracy with the GPS signal after boot up of the network devices...... 46 5.3 Re-synchronization time for the three Slaves with respect to the deviation en- countered with the GPS signal was removed for various time periods...... 47 5.4 Configuration Parameter Values for Performance Testing ...... 48 5.5 Time Sync Accuracy (ns) for PTP Slave from PTP Master ...... 49 5.6 Time Sync Accuracy (ns) for PTP Slave from PTP Master ...... 50 5.7 Time Sync Accuracy (ns) for both the PTP Slaves from PTP Master . . . . . 51 5.8 Time Sync Accuracy (ns) for the three PTP Slaves from GPS clock ...... 52 5.9 Time Sync Accuracy (ns) for both the PTP Slaves from PTP Master . . . . . 53 5.10 Configuration Parameter Values for one instance of Sender Application . . . . . 55 5.11 Configuration Parameter Values for Stress Testing ...... 56 5.12 Diff and Bias currents measured from N4 set-up during Stress Testing...... 57 5.13 Time Sync Accuracy (ns) Results for the three Slaves with variation in Addi- tional Traffic load ...... 59
6.1 Results for differential current from v1.2 IEDs over the primary route ...... 64 6.2 Results for differential current from v2.2 IEDs ...... 66 6.3 Results for differential current from v2.2 IEDs - part 2 ...... 67 6.4 Results for differential current from v2.2 IEDs - part 3 ...... 67 6.5 Results for differential current from v2.2 IEDs - part 4 ...... 68
7.1 Communication and PTP Configuration for RED670 IED ...... 77 7.2 Global Configuration Parameter Values for AFS665 L2 switch ...... 77 7.3 BC Configuration Parameter Values for AFS665 L2 switch ...... 78 7.4 BCv2 port configuration parameters ...... 79 7.5 TC Configuration Parameter Values for AFS665 L2 switch ...... 80 7.6 TCv2 port configuration parameters ...... 80
ix 7.7 PTP Configuration for Meinberg LANTIME M3000 ...... 83 7.8 Oscilloscope configuration parameters ...... 84
x Nomenclature
AFS ABB FOX Switch
AP Access Point
BC Boundary Clock
BMCA Best Master Clock Algorithm
CPU Central Processing Unit
CSN Circuit Switched Network
FPGA Field Programmable Gate Array
GPS Global Positioning System
GTM GPS Time Synchronization Module
GM Grandmaster
HiL Hardware in the Loop
HMI Human Machine Interface
IEC International Electrotechnical Commission
IED Intelligent Electronic Device
IEEE Institute of Electrical and Electronics Engineers
IP Internet Protocol
IRIG Inter-Range Instrumentation Group
LDCM Line Data Communication Module
LDP Line Differential Protection
LibPTP A Library for PTP Simulation
xi MAC Media Access Control
MII Media Independent Interface
MT-RJ Mechanical Transfer Registered Jack
OC Ordinary Clock
OMNeT++ Objective Modular Network Testbed in C++
OS Operating System
PC Personal Computer
PCM Protection and control IED manager
PPS Pulse Per Second
PSN Packet Switched Network
PTP Precision Time Protocol
SDH Synchronous Digital Hierarchy
SFP Small Form-factor Pluggable
SNTP Simple Network Time Protocol
SONET Synchronous Optical Networking
SyncE Synchronous Ethernet
TC Transparent Clock
UTC Universal Time Coordinate
WAN Wide Area Network
xii Chapter 1
Introduction
1.1 Problem Statement
The main problem that this thesis work is trying to solve is in the area of Line Differential Protection (LDP) :
1. Most of the LDP relays currently communicate over SONET/SDH.
2. This requires dedicated communication equipment that is a wastage of resource.
3. Furthermore SONET/SDH are now becoming outdated and are being replaced by new technologies.
4. Hence there is a need to find an alternative method for the LDP relays to communicate.
1.2 Problem Description
Today most of the Line differential protection relays communicate over SONET (Synchronous optical Networking) or SDH (Synchronous Digital Hierarchy). These technologies provide a symmetric communication delay in both directions since they use the Time Division Mul- tiplexing mechanism. The symmetric delay is utilized to achieve time synchronization for measurement data between the IEDs. One drawback is that dedicated communication equip- ment is needed both inside and outside the IEDs.
Furthermore, SONET/SDH network equipment is being replaced by new technologies (like Ethernet/IP based networks). Thus it is hard for the utilities to get access to SONET/SDH networks. In addition, these IP networks can be shared for other communication requirements in power systems unlike the CSNs.
1 1.3 Project Goals and Scope
The main goals of this project are to :
1. Find a solution to the stated problem
2. Make pilot implementation set-ups using the ABB IEDs and get the proposed solution for LDP communication working :
(a) Get LDP working on a point to point set-up (b) See the effect of adding asymmetry (switches) to the network (c) Use a simulator to test on a larger network (d) Try a WAN testing of the solution
3. Perform an evaluation of the important parameters that are required to study the feasi- bility of the solution.
4. Validate the solution on a WAN network and find out the challenges in using this solution for a future WAN deployment.
Scope: This thesis mainly revolves around developing a modern-day solution for the com- munication of LDP Application, implementing it, and doing a laboratory based testing of the solution. The scope of the solution’s implementation and evaluation do not go beyond pilot laboratory set-ups and measurements.
1.4 Project Outline
The thesis project work is divided into Design, Implementation and Evaluation phases. The total duration of the thesis is twenty two weeks. The first four weeks are used to perform lit- erature study and propose a solution. The next nine weeks are used to design and implement the solution while the remaining nine weeks are used for the evaluation and testing.
Design Phase : This phase is divided into two parts - first is to do a literature study and come up with a feasible solution to the problem and the second is to perform a background study and get acquainted with the ABB internal tools which are required for the work.
Implementation Phase : This phase mainly involves working with the IEDs and other hard- ware/software equipments to set up pilot network models to test the LDP over the proposed solution, keeping in mind the requirements as per the standards. The main goal of this phase is to try out variants of the solution and ensure the working and monitoring of the behaviour.
Evaluation Phase : This phase involves the measurement of some of the performance metrics
2 as well as a full on testing in a WAN deployment scenario. Possible future work is also analysed which is mentioned in the Conclusion Chapter.
3 Chapter 2
Background
2.1 Line Differential Protection
Due to the quick and inherently selective fault detection, Line Differential Protection is an im- portant protection scheme. According to ANSI/IEEE C37.2-2008 [1], the communication and its corresponding time synchronization are two of the most important parts of this protection scheme.
2.1.1 Working Principle
The working principle behind Line Differential Protection is based on Kirchhoff’s Current Law. This law can be stated as; “The sum of the current flowing into a node has to be equal to the current flowing out of this node” [2]. The operating principle of differential protection scheme is shown in Figure 2.1.
Figure 2.1: Line Current Differential Protection [3]
Figure 2.1 shows the simplified version of the working principle of the fault/delta current by summing up the measured inflow/outflow current values. If the inflow current equals the out-
4 flow current, they cancel each other and on the other hand a fault occurs if the line differential current delta is not zero.
In order to transfer and evaluate the measurement data, two different methods can be used. One is to calculate phasors for each period or a predefined number of periods and only trans- mit this phasor value. Second is to transfer all the sampled values instead of a single value. The second method has an advantage over the first, since transfer of all the sampled values would contribute towards providing more options for the future algorithms that detect faults [4]. Hence having a high bandwidth channel and a precisely accurate time synchronization mode are two of the most important requirements to be able to transfer all the sampled values within the stipulated time accuracy.
2.1.2 Requirements of the Protection Interface
The communication between the relays is the most essential factor for the working of the line differential protection. Only if the data from the remote relay is communicated correctly along with a precise timestamp, the relay can calculate the difference in current.
Taking the communication architecture into consideration, there are two very important phe- nomenons [2] - the path delay, also called the latency and the path delay variation (PDV), also called as jitter. Latency can be defined as a delay occurring in the communication of the data information to the remote relay. This can cause a variation in the tripping time (based on the amount of delay) and hence is an important point that has to be overcome or taken care of in the fault detection algorithms that are designed. The recommended latency values are stated in the IEC standard 61850-90-1 [5]. Based on the amount of applied voltage, usual range requirement for the maximum latency is between 5 ms and 10 ms.
Like latency has an influence on the communication of data through the channel, jitter has an influence on the time synchronization’s accuracy. The recommended jitter values are stated in the IEC standard 61850-90-1 [5]. Based on the required fault current sensitivity, usual range requirement for the synchronization accuracy is less than 10 µs. Hence lesser the jitter, higher the time synchronization accuracy and consequently quicker and more precise detection of faults is possible.
2.2 Existing Industrial Solutions : SDH/SONET
A traditional and widespread means of communication, in the field of teleprotection is SDH. In simple terms, SDH can be defined as a standard technology used for data transmission in a synchronous manner using optical communication media [6]. SONET is a similar protocol standard like SDH, which is used in the US. Before these technologies, there was the tradi-
5 tional PDH (Plesiochronous Digital Hierarchy) technology that was used. The SDH/SONET technologies were invented in order to provide faster communication with lesser costs than PDH.
SDH communication network is based on TDM (Time Division Multiplexing) which, using fixed time interval slots, blocks chunks of bandwidth for the various services running in the network that are in need of them. Due to the fact that TDM needs a fixed bandwidth, which is reserved for the needed timeslots, it provides a symmetric communication delay in both direc- tions. This symmetric delay is utilized to achieve time synchronization between the protection devices [2]. A major drawback of SDH is that it does not provide effective bandwidth utilization.
The SDH protocol has a complex structure due to the interweaving of the header part across the data part. However this feature allows for the data that is encapsulated to have its own freedom in terms of the frame structure and rate [6]. The interweaving is also advantageous since it allows only very less latency for the encapsulated data. The delay for the data passing through is atmost 32 µs, compared to a frame rate of 125 µs. Other complementary protocols usually buffer the data during transits for at least one frame or packet before sending it on unlike SDH. The data and frame rates are different due to which extra padding is provided for the data so that it can glide within the frame. Although the complexity increases with the allowance of this extra padding, the performance is also enhanced. While SDH is still used for protection applications, the network providers discontinue native SDH services successively and change over to the PSN technology.
2.3 Motivation for IP-based Ethernet Networks
IP-based Ethernet networks offer more advantages than CSNs and also remove the existing problems caused by the outdated SDH networks. In the case of PSNs, the available bandwidth can be utilized more efficiently thus requiring fewer network devices and connections for the same network load. This further leads to lesser floor space utilization for the network deploy- ment as well as a reduced usage of electricity. Ciena Corporation compares the legacy SDH equipment with the PSN technology in a white paper [7]. The comparison and case study done in the paper reveals the following figures - 95% floor saving and 90% energy saving. Hence, it can be concluded that it is advantageous to communicate natively over PSN technology and discontinue SDH for LDP application.
From the Line Protection point of view, different approaches have already been tested in the past which use PSN technologies. Some laboratory based tests have been performed in the past using Wi-Fi and WiMAX technologies [8], [9]. When using Wi-Fi technologies, usage of a centrally controlled radio communication can be a disastrous disadvantage since it is po- tentially liable to a single point of failure. Also the range of distance that a Wi-Fi signal can
6 span plays and important role. The 87L protection scheme done, based on Ethernet communi- cation using echo mode for round trip measurement in order to establish time synchronization was published by Fukushima et al. in [10]. The stated method in the paper uses a PSN network and this academic research work shows that using PSNs for LDP is also a plausible solution that can be considered.
Quality of Service for prioritizing traffic and routing of packets are some of the major perks of building networks on top of the IP Layer 3 and upwards in the ISO/OSI model. One can also route traffic through different ports in case multiple applications are running at the same time on the network. Also, the same Ethernet network can be used for connecting the LDP protec- tion relay device with the Personal Computer in use. This option of using an IP based network has been tested upon in [2] and it is the approach most closely related to the methodology of this thesis work.
Unlike CSNs, using the PSNs for LDP will require the usage of a time synchronization mecha- nism that can get the clocks in the network synchronized. As mentioned in Section 2.1.2, a very precise time synchronization is needed for the transfer of data samples. Using devices having GPS receiver and then synchronizing them through the GPS signal is an alternative solution. However the aim here is to develop an independent solution and also, it may not be possible to have GPS antenna and receivers in several situations since it becomes an added requirement on the hardware. Besides here, the focus is on synchronization via the communication path.
2.4 Time Synchronization
In modern power systems, precise time synchronization has become an essential phenomenon. Although there are several methods available today to achieve synchronization between the intelligent electronic devices (IEDs), recently there has been a great deal of interest in trying to provide the time synchronization using the same infrastructure channel through which the data messages are communicated.
Most of the power system applications involving IEDs usually require that the IEDs have an accurate time source, using which they operate in sync with each other. Examples of such applications requiring micro and nano second accuracy include travelling wave fault location, synchrophasors and sampled values. In order to distribute such accurate time stamp values, the traditional methods that have been popularly used in the past include IRIG-B (which needs an extra cable different from the cable used for the data communication) and NTP (which does not provide sub micro-second range accuracy). In modern times, Ethernet based communication technologies are becoming more prevalent in substation automation industrial environments. When using PSN networks, the challenge lies in retaining “high reliability, deter- minism, and availability” [11]. A protocol used for time synchronization over the Ethernet that
7 satisfies these qualities is the PTP (Precision Time Protocol or IEEE 1588). This is becoming a dominant and standard technology for synchronization over the same data communication path. It provides an accuracy of 1 µs or lesser depending on the implementation.
The upcoming sections discuss the existing time synchronization solutions like NTP, IRIG- B and Synchronous Ethernet followed by a detailed protocol analysis of the Precision Time protocol. It includes information regarding the protocol’s working, designing and special ter- minologies used by it in the network. PTP provides accuracy within the sub-microsecond range. It has also been defined as several different profiles in the standard out of which the Power profile, Power Utility profile and Telecom profile for power system applications are being mentioned here.
2.4.1 Inter-Range Instrumentation Group - B
IRIG-B technology, developed by the Inter-Range Instrumentation Group (IRIG), is one of the most widely used time codes in the electrical industry. It is often used to provide time synchro- nization solutions in the power systems domain for phasor measurement units, protection relays or similar utilities. Each IRIG-B signal is a frame, 100 bits in size having precise information regarding the date, time and sometimes the control functionalities [12].
IRIG-B can provide accuracies in the order of 1 µs or lesser depending on the implemen- tation. However there are a lot of details like method of connecting the network devices to the time source, correct usage the transceivers, receivers and repeaters etc. to which care- ful attention has to be given during the implementation phase. Unfortunately, due to the requirements of these specifics it makes the overall design, implementation, installation and maintenance of these IRIG-B networks costly [12]. Another disadvantage is that the IRIG-B signals are not compatible with the existing data networks. Apart from these issues, due to a technological migration happening now, towards PSN based technologies in the industry, IRIG-B is also slowly being substituted by Ethernet based solutions for time sync that use the same communication channel as the data.
2.4.2 Network Time Protocol
The NTP protocol was created around the 1980s and since then has been a widely accepted protocol used for offering time synchronization solutions in several domains which use Ethernet based networks [12]. This protocol, instead of just copying and pasting the timestamp received into the device’s internal clock, makes use of an algorithm that controls the time frequency in order to keep the timestamping error to a minimum and hence ensures there is no disruption caused. However this leads to a slight delay in the conjunction of time. NTP transmits time messages through the Ethernet medium along with the other data and by taking the time mea- sured for sending and receiving these messages in the network into account, it balances the
8 possible network latency. Apart from this NTP also calculates an approximate clock deviation and provides a steady time sync even if the signal to the time source is lost.
SNTP is another subset protocol to the NTP, which has lesser algorithm constraints and does not provide all the features like NTP does. However the IEDs using Ethernet commu- nication for synchronization usually use SNTP instead of NTP due to its simplicity and less complexity. In fact, the IEC 61850 standard itself recommended this protocol for the IEDs at first. Unfortunately, the SNTP (neither the NTP) does not provide the required time synchro- nization accuracy wanted by the LDP application and hence does not satisfy the protection interface requirements. It provides an accuracy at a range of 1 ms or lesser. Under higher traffic loads, in practice it attains only an accuracy between 2 ms and 10 ms.
2.4.3 SyncE
Synchronous Ethernet is another technology that is gaining popularity amongst the time syn- chronization solutions. It usually operates along the physical layer of the OSI layer structure in order to distribute the time sync messages from the clocks to the nodes in the network [13]. Similar to SDH/SONET technologies, SyncE also uses a local clock for every network device to learn the clocking rate from each interface in the network device. Usually the system clock is obtained from one of these interfaces in the network device depending on the accuracy. The architectural design of SDH and SyncE are quite similar. Its capability is said to be similar to that of SDH since it does not get affected by other higher traffic load in the network. Also, SyncE is mainly bothered only about the time signal’s precision between the devices.
The synchronization architecture for SyncE is such that any network device should have two local clocks at least and the network interfaces on the devices should be able to produce their own time synchronization signal if the reference clock’s signal fails. This state of the network device is called “holdover” [13]. In order to ensure that this signal generated does not fade away in the communication medium, the signal should be subjected to a filtering at regular intervals and revived.
2.5 Precision Time Protocol
Precision Time Protocol (PTPv2) is defined in the IEEE 1588-2008 standard [14] as a “Pre- cise Clock Synchronization protocol for Networked Measurements and Control Systems”. The PTPv1 was published in 2002 [15] and then its second version PTPv2 was released in 2008. Both these versions are not compatible and thus cannot be used interchangeably in the net- work. Since PTPv2 has more enhanced features and is more popularly used, that is the version referred to and used in this thesis work.
9 The aim behind the origin of PTP is to synchronize the different types of clocks varying in precision and stability, in the PSNs. PTP does not need much bandwidth and neither does it cause much of an overhead in the communication aspect. Hence it can be a perfect choice for a time synchronization solution in the distributed network systems. Based on the type of hardware clocking in the network devices and the kind of PTP stack implementation in them, this protocol can provide an accuracy on a sub-microsecond level.
2.5.1 Protocol Details
The PTP standard includes details regarding the principles and operation sequences that hap- pen to make all the clocks in the network synchronize with each other in a real-time run. All the clocks in the network work on the basis of a “master-slave synchronization hierarchy”. The main clock in the network that acts as the time source for the whole system is called a grandmaster. By sending timing messages to other devices from this clock, the whole network synchronizes to the time advertised by this grandmaster. This protocol works within a subnet scope which is termed as a “PTP domain”.
2.5.1.1 PTP devices
A network running PTP can have both PTP compatible devices called as PTP devices and PTP incompatible devices which are known as non-PTP devices. The non-PTP devices are the ordinary switches and routers and other network gear. Some of the PTP devices include Grandmaster Clocks, Ordinary Clocks, Boundary Clocks and Transparent Clocks. These are described in more detail below [16]:
1. PTP Grandmaster Clock: The role of the grandmaster clock within a PTP network is to act as the time reference or time source to which the rest of the clocks in the network would synchronize too. Hence this clock needs to have a very accurate time feed, which is usually a GPS signal feed or an internal atomic clock. So basically this is an external source of time reference.
2. Ordinary Clock (OC) : A PTP clocking device having a single port is called an ordinary clock. Such a clock can act as either a master i.e become a time source or as a slave i.e synchronize to a master clock. The BMC algorithm (explained in Section 2.5.2.1) is what decides whether the OC becomes a master or a slave for configuration. These are the most common PTP nodes in the network since they act as end devices and also connect to various other external devices.
3. Boundary Clock (BC) : A PTP clocking device that has multiple ports with each port connecting the clock to a different communication channel is called a boundary clock. It can be thought of as the equivalent of a standard router or switch in the PTP network. A BC passes all ordinary traffic along with the PTP traffic. Such a clock can act as
10 either a master i.e become a time source or as a slave i.e synchronize to a master clock. A slave BC synchronises itself to a master and also transmits the time signal to the other master ports in the device. It also uses the BMC algorithm to decide which port becomes master and which one becomes the slave. The master ports in BC synchronize the clock ports in a downstream manner while the slave ports in BC synchronize the clock ports in an upstream manner.
4. Transparent Clock (TC) : The most important manner in which PTP counteracts the network delay and compensates for it, is by using the transparent clock. Its role is to keep updating the time interval field of the PTP event messages. However this update then recompenses for the network delay thus providing an accuracy of 1 picosecond. A TC conveys the transit time of the PTP event message to the receiver device which helps in calculating the propagation delay. There are two types of TCs :
(a) Peer-to-peer transparent clock (P2P TC) : A P2P TC, apart from calculating the propagation delay, also measures and corrects the propagation information as well as the port link delay of the receiver port. With the Peer delay measurement mechanism described in Section 2.5.1.4, the delay between slave TCs, slave-master TCs are calculated. (b) End-to-end transparent clock (E2E TC) : An E2E TC only calculates the transit time for the PTP event message and does not provide any corrections for the propagation delay like P2P TC. As opposed to P2P TC, it uses the Delay Request-Response mechanism which is described in Section 2.5.1.4.
Figure 2.2: PTP Clocks [16]
2.5.1.2 PTP Messages
This section describes the different PTP messages that are exchanged between the PTP devices for achieving time synchronization [16]. The are mainly two types of PTP Messages : (1) Event Messages which are time critical and they include the first four rows in Table 5.10; (2) General Messages which are not time dependent and they comprise the rest of the rows in Table 2.1.
11 Hex message type
00 Sync
01 Delay Req
02 Pdelay Req
03 Pdelay Resp
08 Follow Up
09 Delay Resp
0a Pdelay Resp Follow Up
0b Announce
0c Signaling
0d Management
Table 2.1: PTP Message Types
2.5.1.3 PTP Communication Path
In simple terms, a network cable that has a master element at one end and a slave element at the other is called a PTP communication path [16]. To this network cable if a TC is added in between, then the connection from the master element could be branched out to several slave elements. However an important point to remember here is that a PTP communication path may have a TC in between but not a BC. This is because while a TC splits the path, a BC ends it. For instance, if a GM talks to a BC and this BC talks to an OC slave clock, there are two cables used here and the GM is not directly talking with the OC slave clock. In fact all the PTP messages received by the OC slave will have the BC in its address.
2.5.1.4 PTP Delay Mechanisms
In order to perform clock synchronization in a network, it is essential to know the delay cause in communicating the time messages between them. Such a delay in communication is other- wise called path delay or latency. In PTP terminology, this path/network delay measurement mechanism is what is known as the PTP delay mechanism. The delay that is of importance here is the port-to-port delay on a path instead of the end to end delay between two distant devices [17]. This Delay Mechanism concept is what is behind the reason for the accuracy of the PTP protocol functionality. In other words the Hardware Timestamping mechanism is
12 what makes the IEEE1588 protocol so trustworthy [18]. There are mainly two types of delay mechanisms :
1. End-to-End Delay Measurement Mechanism : The standard also calls this the delay request-response mechanism. This delay measurement mechanism uses message types Sync, Follow-up (optional), Delay Req, and Delay Resp as shown in Figure 2.3. An exchange of messages between the master and slave is portrayed. The time at which a message is sent and received on both ends is recorded and stored as four timestamps - t1, t2, t3 and t4. The Follow Up and Delay Response messages take the timestamps to the slave clock from the master. Using this timestamp data, the slave clock adjusts its time according to the time of the master which it calculates by using the following formula that gives the time deviation between itself and the master :
offset = (t2 + t3–t1–t4)/2
Figure 2.3: PTP End-to-End Delay Mechanism [19]
Unfortunately, there is one shortcoming in this equation. It presumes the fact that the time taken for the messages transmission from master to slave is the same as the time taken for the messages transmission from slave to master, i.e it presumes a symmetrical delay in both the directions. However these are not symmetrical. Any difference in the delay between both the paths - which could happen due to the queues in the routers, switches or even in the network stacks at the end devices - results in an error in deter- mining the time difference between the master and the slave. This is taken care of by the hardware timestamping. How this works is shown in Figure 2.4.
13 Figure 2.4: Hardware Timestamping during the transport of a Sync message [19]
The Media Independent Interface (MII) between the data link layer and the physical layer in the network device helps in the hardware timestamping mechanism of PTP. Whenever a message leaves or arrives at a network port, special “PTP hardware” in the device creates a timestamp from the internal clock in the MII. This wipes out any uncertainty regarding the response time of the OS. Besides this, the switches supporting PTP, better known as TCs, also timestamp the PTP messages. One such TC is shown in Figure 2.4, which rectifies the time spent by updating the field in the PTP messages. The BCs use the PTP messages to adjust their own internal clock and then send it to the other slave which need to be synchronized. This delay mechanism focuses the PTP load on the master since it does most of the work. Due to this it is not so scalable with an increase in the number of network devices.
2. Peer-to-Peer Delay Measurement Mechanism : This mechanism calculates the delay between two connected network ports, like for instance the ports of two OCs that are connected, with no middle router or switch. The P2P delay mechanism scales with the increase in the size of the network since the network load is not focused on the master port alone like is the case with the E2E delay mechanism.
Like in E2E mechanism, in the P2P mechanism also the master and slave exchange Sync and Follow Up messages and the slave calculates its deviation from the master according to the following formula.
slave time = master time + network delay.
14 Unlike in E2E mechanism where the slave sends delay measurement messages to the master, in the P2P mechanism all the devices in the network send peer-delay measure- ment messages to each other. Hence there is no requirement for the four timestamp calculation that was seen in E2E mechanism. Using the peer delay messages, all the de- vices can calculate their offset with respect to the neighbouring devices in the network. Figure 2.5 shows how this works.
Figure 2.5: Working of peer-delay messages in Peer-to-Peer Delay Mechanism [20]
In all the ports on every device, a periodic prompt for sending peer delay messages happens. Next, when the Sync messages go into the device, the peer delay factor is discarded by resetting the fields in either the Sync or the Follow Up message. However in case the device is a switch, the peer-delay is not present in the exit port since the next device that is connected to the switch will do this. This way the resetting being done twice can be avoided. The order of peer-delay messages is shown in Figure 2.6.
Figure 2.6: PTP Peer-to-Peer Delay Mechanism [20]
15 In this scenario, Clock A is trying to find its offset from Clock B. A sends a Pdelay Req message to B at time t1 and this message reaches B at time t2. A stores t1 and B stores t2 which is sends at time t3 through the PDelay Resp message to A that reaches at t4 which now receives t2. Following this message a Pdelay Resp Follow Up message is also sent by B to A carrying information about the timestamp t3. So finally A ends up having t1, t2, t3, t4 timestamps and it calculates its deviation from B as described in Section 2.5.1.4, point 1 under end-to-end delay mechanism. Next B can do the same with A to calculate its offset from A and in this manner each device can calculate its offset from the neighbouring devices.
Another presumption made is that the communication delay for these messages to reach from A to B and vice versa are the same which is not the case in reality. However this is different from the E2E case since this presumption is done only for a point to point direct cable. So since there will be no queues, this presumption will hold. Here, as with the end-to-end mechanism, the assumption is made that the time it takes for the peer-delay messages to get from one clock to the other is the same in each direction. If supposing the device at the end of this direct cable is a switch, then this presumption can fail. However that is the reason why P2P mechanism only works well with TCs or BCs; since TCs and BCs can take care of the delay. OCs cannot be used for P2P delay mechanism since they cannot handle the peer-delay messages.
An E2E TC supports and takes part only in the end-to-end delay mechanism while a P2P TC supports and takes part only in the peer-to-peer delay mechanism. A BC on the other hand can have several master ports and slave ports. Hence both end-to-end delay mechanism as well a peer-to-peer delay mechanism can be configured on those ports. An OC that has only a single port can either have peer-to-peer delay mechanism or end-to-end delay mechanism. Any one PTP communication path cannot have a combination of both mechanisms. It can either run peer-to-peer delay or run end-to-end delay at any given time. A peer-to-peer communication path can have exactly two ports while an end-to-end communication path can have atleast two ports.
End-to-End versus Peer-to-Peer : E2E delay mechanism is more user friendly since OCs, routers, switches etc.. can also manage this mechanism. However only TCs and BCs usually work well with the P2P delay mechanism. Inspite of that P2P delay mechanism can be more useful in some scenarios where it is more advantageous :
1. When the networks are dynamically changing, even if a network path has changed, this periodic outburst of peer-delay messages from the devices, ensures that all the devices are aware of their own offsets from the neighbouring devices. Also, these peer-delay messages are transmitted in such a way that there are no loops (by using the Spanning Tree protocol).
16 2. Unlike in E2E mechanism where the Sync message and the Delay Request message can end up being on different paths, in P2P it is not a problem as there is no Delay Request message.
3. As in P2P, one has to send only the Sync and Follow Up messages the overhead is less and there is more simplicity during congestion times which makes it more scalable while for E2E mechanism, it has to deal with the Delay Request messages responses.
2.5.2 Working Principle
The IEEE 1588v2 standard describes a “hierarchical master-slave architecture” for PTP time synchronization. Under this architecture, the devices in the network act as either a master (thus relaying time to other devices) or a slave (thus synchronizing to the time relayed by the master devices). These master and slave devices then exchange a set of messages as explained under the delay mechanisms to synchronize with each other. A sample PTP Network in the power domain containing grandmaster clock, BC switches, IEDs etc. is shown in Figure 2.7 which also shows the working of PTP time synchronization. Here if the Master 1 looses its GPS signal, then all the devices start synchronizing to Master 2 which is the second best clock to take upon the role of a grandmaster based on its time quality [16].
Figure 2.7: PTP Working Principle [16]
The working principle mainly revolves around two steps for synchronizing the devices using IEEE 1588 protocol : (1) Check and decide which device will become the grandmaster clock and which will become master/slave in the network, and (2) perform delay measurement calculations to find the deviations of the internal clocks in devices with respect to the other network devices and correct this time difference in order to synchronize with each other. In
17 order to execute the first step, PTP uses the BMC algorithm which is described in the section section.
2.5.2.1 Best Master Clock Algorithm
The Best Master Clock algorithm is an important part of PTP. This algorithm determins which node in the network will become the GM. The algorithm runs at regular intervals in the network and updates itself in case there is another better GM available [21].
In order to determine which node has a better time quality, the following parameters are chosen by BMC and evaluated for each node :
1. Priority One Field : It is a user definable value with possible ranging from 0 to 255. The default value is 128 for a device that needs to compete for the master role. For the slaves its set to 255. So basically lower the number, higher the priority of it becoming the master.
2. Clock Class : The class of a clock is determined by its current state. A clock having a stable source of time which is really accurate is said to have a higher class than a local clock running wildly. For instance a clock connected to UTC, has more class than a clock connected to a local reference. Hence higher a clock class, more probability that the clock becomes the master.
3. Clock Accuracy : This field specifies the accuracy level of a clock and more the accuracy more the chances are of the clock becoming the master. Usual range for PTP is between 20ns to 100 ns.
4. Clock Variance : This field indicates how stable the clock is or how much it is affected by the jitter and latency in the network. Lesser the variance the better it is.
5. Priority 2 Field : This is similar to the Priority 1 Field that can also be set by the user. It is just a back up option for providing a second clock as a choice in case for some reason the first clock is not able to become the master.
6. Source Port ID : It is the MAC address of the source port in the clocking device. It is also acting as a tie breaker here in case all the above mentioned values end up being equal in two or more nodes fighting to become the master.
7. Steps Removed : If two nodes are having the same GM in the network, then whichever node is connected to the GM through least hops has better chances of becoming a master (in case the GM looses its GPS signal).
According to the BMC algorithm, when the existing GM looses its GPS signal, another clock with the next best time quality takes upon the role of the GM according to the above rules. The set of possible states that any clock can go into are shown in Figure 2.8.
18 When the clock is switched on, or restarted, it goes into the listening state where it lis- tens to announce messages from other clocks around it. An announce message contains the above mentioned fields and properties of the clock sending it. So when a clock receives an announce message from another clock, it can compares the remote clock’s properties with itself and determine if the announce messages was a better one or a worse one. If that announce message was a better one, which means the remote clock has a better time quality, this clock goes into the slave state making the remote clock as its master. If the announce message was worse one which means the remote clock has worse time quality than this clock, then it ignores this messages and waits. The wait time for getting a better announce message than itself is called the announce time out interval. Within this interval the clock should get a better quality master, else it takes over the role of a GM in the network. This algorithm runs continuously in the network and also it is important to note that the announce time out interval should be greater than the announce interval for better results.
Figure 2.8: PTP BMC Working
2.5.2.2 Synchronization
Like mentioned under the working principle, there are two steps for PTP synchronization out of which the first step is to run the BMC and determine the master-slave architecture for the network. The second stage is where each device calculates its offset from another device using the delay mechanisms in order to achieve time synchronization in the overall network. So this second stage also known as the synchronization process is divided into two phases [22]. First phase is the “offset correction” phase that has been described in Section 2.5.1.4 by calculating
19 the delay measurements. Second phase is the actual network synchronization where the clocks using the delay measurements and offset calculation done, update their hardware and software clocks and synchronize themselves to their respective masters. Hence all the devices in the network get synchronized in this manner.
2.5.3 PTP Profiles
A PTP profile as defined by the IEE-1588 standard is “the set of allowed PTP features ap- plicable to a device”. PTP has different profiles like the power profile, power utility profile, telecom profile etc. to satisfy the needs of different kinds of applications. However all these profiles are just variations of the PTP standard or restrictions imposed of several features in PTP in order to enable the usage of PTP in various domains like the power domain. The IEEE standard and the description of PTP given in Section 2.5 all refer to the standard profile also known as the default profile. This profile is the most widely used on in a lot of applications. Almost all manufactures and vendors manufacture PTP devices based on this default profile, since it also provides an easier standard way of comparison and testing. All other profiles are created by modifying this default profile by various other standards like the IEC for their own purposes.
A PTP profile is created by modifying one or more of the following features from the de- fault PTP profile usually [16].
• BMCA parameters
• PTP delay mechanism configuration parameters
• PTP management features
• default values of all the variable parameters
• PTP device types
• extra PTP features
• communication rules for PTP
Some of such profiles that are relevant in this thesis work are :
2.5.3.1 PC37.238 IEEE-1588 : PTP Power Profile
The IEEE Power Profile [23] defines “specific or allowed values for PTP networks used in power substations”. Some of these values that are tampered with from the default profile include the BMCA parameters, higher layer protocols in the power communication domain for running PTP messages etc. This profile is configured with the aim of providing accurate time synchronization using PTP over country wide substation automation networks over WANs.
20 The profile also defines the usage of FPGA and PHY interface for the network node operations on a physical layer. In this profile mode, the devices uses the configuration values defined in the IEEE-1588 Power Profile standard [23]. The PTP messages are transported over layer 2 using P2P delay mechanism in the TCs.
2.5.3.2 IEC/IEEE 61850-9-3:2016 : PTP Power Utility Profile
The IEC/IEEE 61850-9-3 Standard [24] defines the PTP Power Utility Profile (PUP) for the power utility automation field. It is derived from the default profile and is a subset of it apart from the redundancy link method that is a new addition in this profile. The reason behind adding this redundancy method is because of the high demands of time sync from the substa- tion automation environment. The IEC 61850 defines “Parallel Redundancy Protocol) (PRP) and High-availability seamless redundancy (HSR)”. PRP is used in cases where the network devices are from a combination of two separate LANs. Since the two nodes could be in differ- ent LANs, the delay principle in the default profile is compromised as the delay calculation in different LANs could change differently. To avoid this, the IEC 62439-3 Annex A changes this 1588 principle only for the PTP messages. The same change is done for the HSR protocol as well. More about this can be read in [25].
Another difference from the default profile is that although the default profile defines a lot of parameters like the P2P, E2E, synchronization modes etc. it does not pay attention to the performance factors or parameters. This PTP PUP profile takes such performance character- istics like time accuracy for each PTP device for instance.
This PTP PUP profile along with the modification regarding PRP and HSR protocols has benefited the substation automation section in the power industry. It has enabled an increase in the performance of the time synchronization solution which is very important in specially the protection function applications.
2.5.3.3 IEEE G.8275.1/G.8275.2 : PTP Telecom Profile
The IEEE G.8275.2 is the PTP Telecom profile [26] which is created when “phase or time- of-day synchronization is required”. The previous telecom profile version is the G.8275.1 in which every node in the system may not participate in the PTP synchronization. The G.8275.2 version uses unicast mode with IPv4/v6 protocols running on layer three in the OSI model. The network devices running this v2 of the protocol need not be connected with a direct fibre since they use a “partial timing support” for synchronization. Some applications of this pro- file are Wi-Fi systems and LTE technology running networks that need synchronization in the telecommunication sector.
The kind of PTP devices used in this profile include OCs (GM and partial time support-
21 ing slaves) and BCs (GM BC, slave BC) [26]. Another interesting difference in this profile is the ABMCA (alternate best master clock algorithm) that is used instead of the BMCA algorithm which is the common algorithm used by all the other profiles.
Some other features of this profile include [27]; usage of both hybrid and non-hybrid modes of PTP, usage of redundant time sources for each device although at any given time the device is synchronized only to one of those sources. This profile does not give much of importance to performance metrics.
2.6 PTP versus existing methods
PTP delivers a clock synchronization accuracy of 1 µs which is somewhere in the middle of the accuracy provided by NTP protocol and GPS based synchronization systems [11]. Instead of using GPS in order to synchronize the whole system, it is easier to use PTP because only one GPS signal is needed in the PTP system for the GM and it is easy to maintain than ensuring every device in the network having a GPS antenna. It also makes it cost friendly. Also this way the whole system does not become a predator to a bad GPS signal; only the GM would loose its signal which would be easy to figure out. Also PTP system can stay independent of GPS. More about this will be covered in the upcoming Chapters. An approximate comparison of the sync accuracies are given in Table 2.2.
Technology Accuracy
atomic clock 0.1 - 1 ns per day
GPS (expensive) 10 ns
PTP (inexpensive) 100 ns - 1 µs
NTP 0.2 ms - 10 ms
IRIG-B 1 µs
Table 2.2: Comparison of Time Sync Mechanisms
While PTP emphasizes on day time based synchronization like the UTC, SyncE method em- phasizes on only frequency based synchronization. However due to the similarity with SDH, it is still a huge competition for PTP in some domains like telecommunication. Just like SDH uses TDM at the physical layer, Sync-E also works on a similar principle as was explained in Section 2.4.3. According to [17], PTP supports upto 1GigE while SyncE supports upto 10GigE. However Sync-E is not so compatible as it requires hardware changes in the network
22 devices. PTP is cheaper in this aspect since it requires only minor hardware changes. It is due to this main reason that PTP is more cost efficient and popular.
23 Chapter 3
Methodology
3.1 Proposed Solution
From the literature study phase of this thesis work which is summarized in Chapter 2, it is evident that using PTP with PSNs can be a great alternative method for SDH/SONET technologies in order to run LDP application. So the proposed solution here it to use Packet switched networks for communication of LDP data and use PTP for time synchronization just like its outlined in [2]. A detailed description of the technical aspects of PSN and PTP are discussed in Sections 2.3 and 2.5.
3.2 Satisfying the Requirements of the Protection In- terface
As discussed in Section 2.1.2, the latency and jitter should be minimum while higher bandwidth and accurate synchronization are absolutely necessary for the LDP communication at the protection interface. In fact according to the requirements specified in the IEC 61850 standards, a time synchronization solution of the order of 10 µs or less is recommended. Hence this solution of using PTP for time synchronization, is beneficial as it fulfils the requirements of the LDP protection interface. Also the networks no longer need to be symmetrical like for SDH/SONET networks. Apart from the time synchronization requirements, PSNs allow routing and flexibility through the QoS features that can ensure a reduced latency and jitter depending on the implementation conditions and network settings.
24 3.3 Approach
3.3.1 Communication : Packet Switched Networks
Since running LDP communication over PSN technologies would mean working at the different ISO/OSI model layers of the network, in this thesis work, it has been decided to run the application over the UDP protocol (at the Transport Layer), IP (at the Network Layer) and Ethernet 802.3 (at the Link Layer). This option of using an IP based network has been tested upon in [2] and it is this approach that is most closely related to the methodology of this thesis work. The academic evaluation done in that paper has been looked upon from an industrial perspective in this thesis project.
3.3.2 Synchronization : Precision Time Protocol
Synchronization over Ethernet using the same medium as that used by the data messages is the main criterion that was considered for choosing PTP as the time sync solution. The approach on using PTP for implementation was again by installing the PTP stack on all the network devices.
A detailed description of the implementation of this solution using the above mentioned ap- proach is done in the next Chapter.
25 Chapter 4
Implementation
The implementation part of this project consists of creating pilot network set-ups in the com- munication laboratory of ABB, using various resources. These set-ups are mainly used, to implement line differential protection over UDP(L4)/IP(L3)/Ethernet(L2) based networks and achieve time synchronization using PTP protocol. This Chapter explains in detail all the re- sources that are used and how the different network topologies are implemented. It also covers details regarding the measurement conditions like the configuration settings in the various network devices.
4.1 Resources Used
A combination of hardware and software tools have been used for implementation and evalua- tion purposes. Before going into the implementation set-ups, a briefing of the resources used is done below.
4.1.1 Hardware Network Components
The various hardware equipments that are used to create the different network topologies and to perform measurements include :
– Intelligent Electronic Device (IED) : An IED is generally defined as a microprocessor- based controlling device that can control power system equipments such as tap changers, switches, drives, valves, transformers, circuit breakers or capacitor banks [28]. Some of the IEDs are also the protective relaying types using the microprocessor to perform several protective and control functions. A typical IED usually has around 5-12 protection functions, 5-8 control functions controlling separate devices, an auto-reclose function, self monitoring function, communication functions etc. That is why they are named as Intelligent Electronic Devices. With respect to the communication functions that they can perform, some of the latest IEDs are designed to support the IEC61850 standard
26 for substation automation thus providing interoperability and advanced communication capabilities. One such IED called the RED670 [29], which belongs to the Relion [30] protection and control product family is what is used for line differential protection application, as shown in Figure 4.1. The RED670 IED is designed for protection, monitoring and control of overhead lines and cables in all types of networks. In addition, this IED is capable of handling transformer feeders, generator and transformer blocks. It provides an extensive functionality with configuration opportunities and expandable hardware to meet our specific requirements. For this thesis work, three RED670 pilot version 2.2 (laboratory models) IEDs that support PTP and have Ethernet optical ports are used.
Figure 4.1: RED 670 IEDs
Within the IEDs, there are several slots into which different modules can be fit as per our requirements. Each module (mainly containing FPGAs) is used to bring about a specific functionality in the IEDs. The modules that are explicitly used in this project are the following. They can also be seen in Figure 4.2.
– Line Differential Communication Module (LDCM) : This module is used for the communication of the LDP’s data packets in the form of C37.94 frames [31]. LDCM uses the C37.94 as the protocol for optical fiber communication. It is these C37.94 frame formats carrying the LDP communication data, that are then sent over UDP packets. – Inter-Range Instrumentation Group (IRIG-B) : This module is mainly used for achieving time synchronization. The message in the IRIG-B will give an indication when a new second is present, with the pattern “on-time”. This pattern then
27 triggers a storage of time within the module and generates a signal for a capture register. An optical PPS input used by itself or in combination with the IRIG-B input can synchronize other equipments in a station if such a device is present. This method is what is used at first before using PTP; during the initial phase of the project. – GPS Time Synchronization Module (GTM) : This module as the name sug- gests is also used for establishing time synchronization between the IEDs. The main notable hardware I/O of this module that is used is the PPS input and output like in IRIG-B module. The probes are connected between this module and the oscil- loscope, and using the PPS out signals that are generated, several measurements are done during the evaluation phase.
Figure 4.2: LDCM, IRIG-B and GTM Modules (left to right order)
– L2 Switch : Layer 2 switches are generally used for storing and forwarding L2 packets in the TCP/IP suite. The L2 AFS switches from ABB are optimized for the data communication of electrical utilities. With their robust hardware and powerful operating system they are able to withstand extremely harsh environmental conditions as mentioned in [32]. The integration of the redundancy protocols PRP and HSR allows uninterrupted data communication and therefore guarantees the reliability and interoperability of the system. Precision time synchronization in accordance with IEEE 1588v2, synchronizes sensors, drives and measuring equipment. Gigabit Ethernet enables a fast connection to
28 the backbone, while connections to terminal equipment are made via 100 BASE-TX – either alone or in combination with 100 BASE-FX.
For this project, two PTP compatible L2 AFS665 LAN switches with 11 ports each are used. These switches are mainly used as BCs or TCs thus allowing them to be syn- chronized to a precision of approximately 100ns. However these switches only support Multicast PTP thus making the load on the network due to PTP sync messages negli- gible. An AFS660 switch is shown in Figure 4.3, which after slight modification makes the AFS665 switch.
Figure 4.3: AFS 660 switch
– GPS Clock and Receiver : A Meinberg LANTIME M3000 GPS clock [33] is used as the PTP grandmaster for all the implementation set-ups. For some of the set-ups at the evaluation stage, the GPS bulb shaped antenna is also used. The front view of the GPS Clock and Antenna used along with the AFS677 switch are shown in Figures 4.4 and 4.5. The Clock is connected to an AFS677 switch (that acts as a BC) [34] and then the GPS signal is relied to all the IEDs in the lab as shown in Figure 4.6.
Figure 4.4: GPS clock and AFS677 switch Figure 4.5: GPS Antenna
29 – Personal Computer : Two desktop PCs (with CPUs having 8 cores) are used to create and configure the network topologies. One of these PCs is running Windows OS while the other is running Linux Ubuntu OS. The Software tools described in the following section are also installed and run on these machines.
Figure 4.6: GPS clock signal relied to all network nodes in the lab
– Oscilloscope : In order to measure the performance metrics like the time sync accuracy, an Agilent Technologies’ DSO1024A oscilloscope [35] is also used as shown in Figure 4.7. A PPS signal is obtained from the GTM boards of the IEDs to perform the measurements.
Figure 4.7: Oscilloscope
– Connection Media :There are mainly three types of connection media that are used and they are shown in Figure 4.8:
– CAT5e UTP Cable with an RJ 45 Connector : The Ethernet over twisted pair cables are used for connecting the L2 switches with each other and to connect the switches to the PCs as well. The ones used for this project are 100BASE-TX, supporting speeds of 100Mbit/s in full-duplex fashion.
30 – MTRJ, Duplex Multimode Fiber Optic Cable and Patch Cord : These fiber optic cables are used to connect two IEDs together and also to connect the IEDs to the switches. – SFP Transceiver : Small Form-Factor Pluggable Transceiver are basically used to standardise the transformation from the optical domain of the communication medium to the electrical domain of the integrated circuit. This transceiver is com- pact and hot-pluggable. It is widely used in the field of Data Communication and Networking. It mainly acts as an interface between a networking device and its interconnecting cable. In the course of this project it is was mainly plugged into the ports of the IEDs, thus acting as an interface to the fiber optic cable.
(a) Ethernet RJ 45 Cable (b) SFP Transceiver
(c) MTRJ, Fiber Optic Cable (d) MTRJ, Fiber Optic Patch Cord
Figure 4.8: Communication Media
– Single Phase Relay Tester : The SVERKER650 [36] relay tester as shown in Figure 4.9, is used to supply a single phase current in order to test if LDP application works well or not during the evaluation phase. The tester is directly connected to the IEDs and the differential current and bias current are measured using remote IED access clients.
31 Figure 4.9: Single Phase Relay Tester
4.1.2 Software Tools
Several Software tools are also used for implementing the network topologies and perform- ing measurements using them. Out of these, some are used for monitoring and configuring purposes. The software tools used are described in detail below :
– The OMNeT++ : In order to perform Hardware in the Loop testing, OMNeT++ [37], which is an Open Source network simulator is used. The OMNeT++ discrete event simulation environment has been publicly available since 1997. Some of the research application areas for which this simulator was built include communication networks, multiprocessors and other distributed systems. In addition to that the simulator is also quite flexible and general. It has been used in numerous domains like queuing network simulations, wireless networks, ad-hoc network simulations, business process simulations, peer-to-peer networks, optical switch and storage area network simulations.
In this thesis specifically, OMNeT++ is used with the INET framework model suite [38] for creating large LAN networks and LibPTP [39] for creating PTP compatible devices in the simulator.
– PCM600 Series: In order to configure the IEDs and perform easier read and write operations on them, the PCM600 [40] tool is used. This tool is installed into a PC and the IEDs are accessed and connected to it remotely. It allows setting of the parameters
32 in the IEDs and analysing and monitoring of the various functionalities performed by the IEDs. PCM600 is compliant with IEC 61850, which simplifies IED engineering and enables information exchange with other IEC 61850 compliant tools. The user interface, workflow and the IEC 61850 based data model in PCM600 are designed according to the same philosophy as the IED itself, ensuring smooth integration between the tool and the IEDs. The configuration settings of the IEDs using PCM600 tool are described in Section 4.3.
– AFS Switch Finder : The AFS Switch Finder [41] is another software tool released by ABB in order to configure the settings inside an AFS switch. The ABB AFS Finder 2.2.0.7 is a free program that detects the AFS switches from the connected optical network. The program displays information such as switch ID, MAC address, IP address, net mask, default gateway etc. as shown in Figure 4.10. It also shows settings within the switch such as the details regarding the ports, time synchronization options like PTP or SNTP, switching and fabrication, system settings etc. This tool is installed on a PC and the connected switches in the same network are discovered automatically and configured using a web interface during which two switches have been discovered. The configuration settings of the AFS switches using AFS Finder tool are described in Section 4.3.
Figure 4.10: ABB AFS Finder 2.2.0.7
– Wireshark : Wireshark [42] is the tool used for troubleshooting and monitoring the UDP data communication packets and PTP synchronization messages during the imple- mentation and testing phases of the project. Wireshark is a free and open source packet analyzer that is used mainly for network troubleshooting and analysis. A screenshot containing the flow of PTP messages between the IEDs that is captured using Wireshark is shown in Figure 4.11.
– Sender Application : This application is mainly used during the part of stress testing, in order to stress the network with additional traffic. The tool is just a basic tool used at ABB, and its parameter settings are shown in Figure 4.12. The sender application is used to send UDP traffic with the flexibility to adjust the size of packets to be sent and the frequency at which they are supposed to be sent. The IP address and port number of the destination have to entered.
33 Figure 4.11: Wireshark capturing PTP sync messages
– Internal ABB debug tools : Apart from the above mentioned specific tools, there were also some other internal ABB tools that were used during the course of the whole project in order to mainly monitor and configure the IEDs and the communication between them. Some of these tools include :
– Debug client : an interactive terminal based tool to remotely connect to the IEDs and upon setting of the debug flags, it would print statements and information related to the status of the IED with respect to those flags. – HMI client : A remote interface tool which is an alternative to actually sitting in front of the IEDs and pressing the buttons on it. Was useful in setting the communication parameters for the used APs in the IEDs. – LDCM client : This tool was mainly used to monitor the data communication packets and time synchronization parameters of the IEDs by remotely connecting to them. It gave detailed information on how many packets were dropped. sent or received and most important of all, it indicated whether the LDP was blocked or not and whether there was a communication failure or not. – I/O Client : It was mainly used to set the attribute values that needed to be mea- sured in the IEDs and hence enabled the observance and storage of line differential current, bias current and the phase angles.
34 Figure 4.12: Sender Application
4.2 Network Topologies
Using the Resources mentioned in Section 4.1, for implementing and verifying the feasibility of the proposed combination of Ethernet communication with PTP for time synchronization, the following network set-ups are created, analyzed and evaluated in the communication lab of ABB. However the details regarding the configurations of the network devices have been explained in Section 4.3.
4.2.1 N1: Point to Point Connection between two IEDs
This is a very simple network topology where two IEDs are connected to each other by a MTRJ, duplex multimode fiber optic cable. One of the IED’s is then connected to the GPS clock through two BCs (one IED and one switch), as shown in Figure 4.13. The steps involved in the implementation of this set-up include :
1. LDCM for Communication and IRIG-B for time sync : Initially, the two IEDs are made to communicate through their LDCM modules that are connected using a duplex multimode fiber optic patch cord. For time synchronization, at first the IEDs are made to run in the echo ping pong mode and then they are changed to run over the GPS mode using IRIG-B modules. So the IRIG-B module is inserted into both the IEDs and their time sync mode is set to GPS. Then the parameters of IRIG-B module is set to GPS and encoding is set to 1344 [43]. Next the IEDs are connected to the GPS clock through a splitter and then both the clocks are synchronized with the GPS clock time;
35 i.e the hardware clock (the internal clock within the IED) and the software clock (which is the connected GPS clock externally) of each IED are synchronized to the GPS clock and hence they are also synchronized to each other. So once the synchronization time difference of the hardware and software clocks of the IEDs from the GPS time becomes less than 16 micro seconds, the differential protection application starts running. Until then, the application is kept blocked.
2. UDP for communication : Now this set up is replaced by making the data communi- cation run over the Ethernet. The hardware LDCM modules are removed and a script to insert software LDCM modules is installed into the IEDs in order to run in UDP mode on the target. The duplex multimode fiber optic patch cord between the LDCM modules is replaced by a MTRJ, duplex multimode fiber optic cable that is connected to each of the Ethernet ports (or Access Points) at the back of the IEDs. Following that, the scripts were installed into the LDCM driver of the IEDs making it send the LDCM type packets (C37.94 frames) over UDP/Ethernet. Note that there is no change in the nature of the packets. It is the same LDCM format except that it is made to run over UDP protocol. The installed UDP configuration script for each IED contains the IP address of the remote client, the port number for sending on the remote client and port number for receiving. The sample scripts for enabling the target to run on UDP mode is given in Appendices A and B.
Figure 4.13: Point to Point set-up between two IEDs
3. PTP for time sync : Once the line differential protection application’s data messages started running over the UDP, the PTP mode was switched on at all the used ports on the IEDs after removing the IRIG-B modules which were no longer needed. Thus PTP synchronization messages are also now exchanged using the same interface that
36 is used for data communication over UDP. The IEDs are now synchronized using PTP protocol to the GPS clock. Finally the splitter connecting both the IEDs to the clock was removed and a single Ethernet port to port connection was made from one of the IEDs to the switch acting as the boundary clock, thus relying the GPS clock’s synchronization messages as shown in Figure 4.13. Upon doing this, the BMC algorithm ran automatically thus making the IED connected to the GPS clock the PTP master and the other IED connected to this master IED as the PTP Slave. The PTP Slave after a couple of seconds are synchronized to the PTP master and once the accuracy of the time difference between them got below 16 micro seconds, the LDP worked.
So now the two IEDs communicate the LDP data frames and PTP synchronization messages through UDP/Ethernet.
4.2.2 N2: Introducing L2 switches between the IED’s
Figure 4.14: Introducing PTP compatible L2 switches
The aim of this topology set-up is to see the effect of introducing PTP compatible L2 switches (see Section 4.1.1) in the network. By doing so, the so called network symmetry which existed in topology N1 was removed. The L2 switches will now perform the store and forward mech- anism, thus creating delay and jitter into the network which in turn introduces a path delay variation that can affect the synchronization accuracy. In order to ensure that such asymmetric disadvantages in the network do not effect the time synchronization, in case of PTP compatible devices, the standard IEEE 1588-2008 [14] defines for this case so called BCs and TCs (see Sec-
37 tion 2.5.1.1) which correct the time by adding information about the holding time in the switch.
So as shown in 4.14, similar to N1, the IED connected to the GPS clock which is the grand- master, acts as the PTP Master according to the BMC algorithm. The switch connected to this PTP Master can be either configured as BC or TC (more details in Section 4.3.1.2). In this case both the switches are configured as BCs. So the port of the switch connected to PTP Master, automatically aligns itself as PTP slave port, and relies time to its other port that is connected to the second switch, thus making that port a PTP Master port. Hence the port of the second switch attached to the first switch’s Master port, will align itself as PTP Slave port and relies time to its other port that is connected to the second IED thus making this port as a PTP master port. Finally the second IED that is connected to the second switch becomes the PTP Slave and all the devices in the network are now synchronized to the GPS grandmaster clock that is locked to the UTC time reference.
4.2.3 N3: Three Ended Application - GPS to IED
Figure 4.15: Three ended application
Compared to topology N2, in this case another third IED is added due to which now there is a difference in the LDP Application, as the calculation for the Line Differential Current would now have to take into account the third IED as well. Also in this set up, the UDP configuration scripts and software LDCM configuration scripts on each IED will be slightly different than for N1 and N2 topologies. There will be second LDCM slot defined in the Software LDCM script. In each of the UDP configuration scripts there will now be information for two remote clients
38 and each IED will now have two ports for sending and receiving. So basically the three ended application is configured such that the three IEDs are connected to each other in a triangle. As can be seen from Figure 4.15 this network is a little more complex since there are two PTP slave IEDs that are being synchronized to the PTP Master IED through the BC switches in between. The PTP Master is then synchronized to the GPS grandmaster and hence the entire network is synchronized to UTC Time reference.
4.2.4 N4: Three Ended Application - GPS to switch
Comparing N3 with this topology which is shown in Figure 4.16, they may not look so different. This topology is also a three ended application except that here, the GPS grandmaster is directly connected to one of the ports of the first L2 switch. Thus this scenario actually represents a case where there is a single GPS signal into the network which is relied to all the devices that act as slaves. This is the general trend in real world deployment. Hence there is a single network PTP grandmaster and all the three IEDs become PTP Slaves. The main motivation behind this setup is to see the effect on Time Sync Accuracy between PTP Slave 1 and PTP Slaves 2/3 as there is a difference of one boundary clock coming in between slaves 1 and 2/3. Or in other words, PTP Slave is connected to the same switch into which the GPS signal comes in while PTP Slaves 2 and 3 are on another switch. This could have a possible effect on accuracy of time sync and time required for re-sync.
Figure 4.16: Three ended application with the GPS signal provided directly to the network
39 4.2.5 N5: Hardware-in-the-Loop Trial with OMNET++
This network topology as shown in Figure 4.17 is created in order to facilitate HiL testing. Keeping in mind that PTP compatible L3 devices are quite expensive, other alternative ways to create larger networks were explored than to buy the hardware network equipment. One such alternative method is the use of a network simulator in order to simulate larger networks containing PTP compatible routers and switches and then to connect the simulator to the hardware IEDs and switches that were present, through emulation. During the search, this open source discrete event simulator called OMNet++ (see Section 4.1.2 for more details) was discovered which also had libraries that supported the creation of PTP compatible BC and TC nodes [44].
Figure 4.17: HiL Network Setup
The OMNeT++ simulator is installed on a Desktop PC running Linux Ubuntu 16.04 LTS which has two Ethernet ports. These ports are then used for emulation. As a first step, on the simulator a L3 router is created with two Ethernet interfaces. The network is then designed into two subnets. All the devices on left side of the PC belonged to Subnet One and all the devices on the right side of the PC belogned to Subnet Two (left and right terminology is according to Figure 4.17). The ports of the PC act as gateways for the complementary subnets in order to allow the end IEDs to communicate with each other. These interfaces are then emulated to the Ethernet ports of the PC by using the code written in Network Description (NED) language [45] as shown in Appendix C. A routing file for the simulated router is also created as show in Appendix D. Now, the Ethernet ports on the PC are connected to the L2 hardware switches that are present, as portrayed in Figure 4.17. Following this, the port
40 forwarding between the ports within the PC is disabled thus forcing the packets to go through the simulated router. Upon configuring the network to perform LDP communication between the two end IEDs along with time sync, the following problems were encountered.
• The PC did not have any support for hardware timestamping due to which it was hard for the PTP Time sync to work. As is widely known, using software timestamping results in a less precise synchronization accuracy, which has outliers up to 200 micro seconds [22] which is not compatible with our requirements thus rendering it useless. Hence the PC in itself becomes a PTP non-compatible equipment in the network.
• Inspite of the first problem, both the Data Communication packets and PTP Synchro- nization packets were forward by the ports into the simulated router which did perform its function of routing. But the major problem was the with the simulator itself. The packets getting routed through the simulated router was visible, but it was too slow and not really compatible for the real time network traffic in this case. Thus the time synchronization was not possible since some of the sync messages were dropped and/or delayed by the simulated router as it could not take that much load. Thus since syn- chronization did not work, the LDP application also got blocked as the PTP Slave IED was never synced with the PTP Master IED that were on different subnets.
Due to the above two problems, it was concluded that under the above circumstances, the usage of a completely software based network simulator was not going to yield any expected results and this network setup thus reached a dead end and no performance metrics were measured from the N5 topology.
4.2.6 N6: Unlocked from GPS time reference
Figure 4.18: Unlocked to UTC with IED acting as a PTP Master
41 This unique set-up’s aim is to try and remove dependency on the GPS signal and satellite system. Coming up with a time sync solution independent of GPS is also one of the require- ments in the thesis project proposal. This set-up is done in order to fulfill that requirement. In Figure 4.18, one can see a promising solution : if the GPS signal is removed and the IEDs are connected over a network, then one of the IEDs (the one having the best time quality among the three IEDs - and in case multiple IEDs have the same time quality, then the IED with the lowest MAC address wins the competition) takes upon the role of PTP Master (thus becoming a local time reference unlocked from the UTC) and the other IEDs and devices can synchronize to this PTP Master IED. As long as the devices in the network are synchronized from one end to the other, the LDP application should not be effected or blocked (unless the time sync quality drops below 17 micro seconds). In this manner, the LDP communication can be made independent of the GPS system. In order to achieve this in this project, the PTP and communication sync code within the IED had to be changed in the relevant portions which cannot be disclosed.
Let us consider another general case of two different subnets or two different substations connected over a network. As long as both the substations have a common/same local time reference (same GmId in the code given in Appendices E and F) and each of the substation devices are synchronized to those respective local time references, both the substations can also be synchronized with each other without needing to depend on GPS. If this solution is further developed it does have potential in the real world power network deployment for a GPS independent Time Synchronization system.
4.3 Measurement Conditions
This section describes the various settings and configuration parameters of the network devices that are then later used for performing evaluations.
4.3.1 Network Device Settings
4.3.1.1 IED Configuration
The settings in the IEDs for Ethernet Communication and PTP synchronization are given in Table 7.1 in Appendix E.1. Those parameters which have their values as subjective are variable in nature and take upon values during the runtime of the network. The rest of the objective values are mentioned along with their default values to which the IEDs are set to during the implemenation set-ups.
42 4.3.1.2 Switch Configuration
The configuration parameters of the AFS 665 Layer 2 Switch which contains 11 ports in total (out of which 3 are optical rest are galvanic) are provided in Tables 7.2, 7.3 and 7.5 in Appendix E.2. These are mostly the PTP parameters that this thesis work is concerned with. Apart from these, depending on whether the PTP mode is BC or TC, there is also the possible settings for each port provided in Tables 7.4 and 7.6.
4.3.1.3 GPS Clock Configuration
The Meinberg LANTIME M3000 GPS clock that was used as the grandmaster in all the network set-ups, had the following set of possible parameters as shown in Table 7.7 given in Appendix E.3 as a part of the PTP configuration set-up. The Clock had its antenna connected to a GPS receiver and was locked on a GPS signal, thus having an accurate UTC Time reference quality of the order of 4ns. Now using this sync to GPS signal, the PTP configuration was done to make the clock to work as the PTP grandmaster in all our implementation set-up (except N6) thus allowing the whole network to become synchronized to the UTC time.
4.3.2 Oscilloscope Settings
The basic settings on the oscilloscope that were used during all the evaluation scenarios are given in Table 7.8 in Appendix E.4.
4.3.3 Differential Current Settings
The sample Relay tester was used to provide 0.4Amps of current across the two connected IEDs at a phase angle of 180 degrees. This is read as 400Amps of bias current on the IEDs.
43 Chapter 5
Evaluation
This Chapter mainly covers the evaluations that are done on the different network topologies described in Chapter 4 and configured as per the default setting values described in Section 4.3. The evaluations performed have been categorized and described in the following sections along with the results and graphs.
5.1 E1: Functionality Testing
The proposed solution in order to run the LDP Application is to use PSNs for communication and to compensate for the symmetric path delay calculation in CSNs by using the PTP protocol for time synchronization in PSNs. This functionality test is a basic test to see if this solution allows the functioning of LDP application without any hindrance and as per the requirements of the IEC61850-9-3 profile. The network topology N4 was used to verify the functionality of LDP. The data communication and PTP communication are described in two different sub-sections.
5.1.1 Time Synchronization using PTP
Firstly the network LDP needs to have the IEDs synchronized with each other and the time difference between each pair of IEDs should be less than 17 µs, else the LDP application will be blocked. Once the devices were connected and configured, there was no problem with establishing PTP communication, infact the PTP sync messages were given priority over other Ethernet traffic and the IEDs were synchronized to the PTP grandmaster and became PTP Slaves as shown in Figure 4.16. The time accuracy was well below the 1 µs requirement for LDP as per the IEC61850-9-3 standards.
5.1.2 Line Differential Protection Communication over Ethernet
Once the IEDs were synchronized using PTP, in order to test the functionaly of LDP, a Single Phase Relay Tester SVERKER650 as described in Section 4.1.1 was used to supply the 3-ended
44 IED application with approximately 0.5 A of current. The connection set-up is shown in Figure 5.1.
Figure 5.1: Functionality Testing
The Relay Tester was connected to only two out of the three IEDs; one of them at a phase angle of 180 degrees from the other. The Differential Current and Bias Current measured from the three IEDs/PTP Slaves are shown in Table 5.1. The results show that the LDP works as expected over the Ethernet.
Parameters Slave1 (s) Slave2 (s) Slave3 (s)
Diff. Current (mA) 6.82 6.63 6.95
Bias Current (mA) 403.3 402.5 401.8
Table 5.1: Diff and Bias currents measured from N4 set-up.
45 5.2 E2: Performance Testing
The network topologies N1 to N4 mentioned in Chapter 3 are used to measure some per- formance metrics that are of importance to the line differential protection application. The three main metrics that are measured include the time synchronization accuracy for the various set-ups, time taken to re-synchronize in case the network looses the GPS signal and time taken to synchronize after the booting up of the IEDs at start-up.
5.2.1 Time to Synchronize at Start-Up
The time required for the whole system to synchronize at Start-Up is an interesting factor as it portrays the efficiency of taking up this solution of running LDP over Ethernet using PTP. This metric was measured using the N4 set-up and the results for the time required to synchronize is measured from just after the IEDs finish booting. The measurements were done manually and hence are accurate only to the degree of a second. The results are shown in Table 5.2. It can be seen that the time taken by Slave1 is less than the time taken by Slaves 2 and 3. This behaviour can be because Slave1 lies on the same side as the GPS signal to the switch as shown in Figure 4.16.
Slave1 (s) Slave2 (s) Slave3 (s)
55 57 61
Table 5.2: Time to synchronize to 1 µs accuracy with the GPS signal after boot up of the network devices.
5.2.2 Time to Re-Synchronize
Since Line Differential Protection is now run over the Ethernet, the time synchronization becomes a very vital phenomenon to ensure that the application does not be blocked. Thus it is important to analyze the case of how long it takes for the system to re-synchronize when the network looses the GPS signal and when the signal comes back within a certain amount of time. The N4 topology was used to measure the resynchronization factor. The GPS signal was removed for specific amounts of time and the time drift from origin was recorded during that time. Now from the moment when the GPS signal was plugged back in until the system got synchronized within the one micro-second range is the time to re-synchronize. This result for the three PTP Slaves is shown in Table 5.3. It can be seen that, the rate of drifting is 10 µs apart for every minute the signal is lost. It can also be seen that the time to re-synchronize which is measured in seconds is slightly better for the PTP Slave 1 by a couple of seconds than for the other two, which is mainly because as you can see in Figure 4.16, PTP Slave 1 is connected to the same switch as the GPS signal.
46 Time period Deviation (µs) Slave1 (s) Slave2 (s) Slave3 (s) out of sync (s)
20 3.33 24 27 27
30 5 26 29 30
60 10 34 34.5 34.5
120 20 38 40 38
300 50 55 57 55
3600 600 77 79 79
241200 40200 105 117 115
Table 5.3: Re-synchronization time for the three Slaves with respect to the deviation encoun- tered with the GPS signal was removed for various time periods.
The plot of measured re-sync time versus time period out of sync is shown in Figure 5.2. It can be observed from this plot that there is a minimum time bound required at a base level for resync, around 25 s no matter how long the GPS signal is lost. This could probably be because the BMC algorithm needs to be run every time the network is connected to UTC. On the other hand, even after being out of sync for almost 2.5 days, it took only about 1.5 mins to become synchronized. So the upper bound is also in limit and cannot be much worse than that.
Figure 5.2: Time required for Re-synchronization
47 5.2.3 Time synchronization Accuracy
The time synchronization accuracy was measured by connecting probes from the oscilloscope to the GTM board of the IEDs and/or to the GPS clock’s PPS output. The Master’s PPS signal was used as reference and the deviation of the slave’s PPS signal from the Master was measured and recorded as the time synchronization accuracy. The network equipment was configured basically using the setting parameters for Communication and PTP as described in Section 4.3. The value of those setting parameters that were used for this particular test E2, are mentioned in Table 5.4. Those that are not explicitly mentioned, have the same default vales as mentioned in Section 4.3. The other values which are mentioned as subjective in those tables, differ between each clock and IED like the MAC addresses.
Parameters Values
PTP settings in the IEDs
AP Operation ON
Network Protocol UDP/IPv4(L3)
PTP ON
PTP settings in the L2 Switch
PTP Operation ON
PTP Mode v2-transparent-clock
PTP Management ON
PTP Status Is Synchronized
Delay Mechanism P2P
Network Protocol IEEE 802.3
Syntonize ON
Synchronize local clock ON
Table 5.4: Configuration Parameter Values for Performance Testing
5.2.3.1 N1 : For the point to point Set-Up
Two probes were used to connect the GTM boards of both the IEDs to the oscilloscope. The PPS signal from PTP Master was used as reference and the deviation of PPS signal from PTP Slave (terminology according to Figure 4.13) with respect to the reference was recorded. The
48 results are portrayed in Table 5.5. They are compiled by collecting a sample subset of 287 sample values which are measured using the oscilloscope.
Parameter Value (ns)
x¯ -24.24
σ 131.06
max (x) 352
min (x) -672
Table 5.5: Time Sync Accuracy (ns) for PTP Slave from PTP Master
The density plot obtained from those 287 samples is shown in Figure 5.3. The time sync peak is almost towards the origin though slightly tilted towards the negative side. On an average it can be seen that the time synchronization accuracy of the PTP Slave to have a deviation range between -200 ns and 200 ns.
Figure 5.3: Density Plot for the PTP Slave’s deviation or time sync accuracy from PTP Master
5.2.3.2 N2 : For introduction of L2 switches Set-Up
Once the direct connection was removed between the IEDs and both the switches were intro- duced between them which brought about a store and forward mechanism of packets in the network, the time synchronization accuracy of the PTP Slave from the PTP Master (termi- nology based on Figure 4.14) was measured using the oscilloscope. The results are shown in Table 5.6. They are compiled by collecting a sample subset of 287 sample values which are measured using the oscilloscope. The standard deviation is 178.14 which is a bit more than
49 the standard deviation when there were no switches. So to a certain extent in the order of nanoseconds, there is an effect on the time synchronization with the introduction of switches. Even the maximum deviations towards either sides are larger than for the N1 case.
Parameter Value (ns)
x¯ -29.6
σ 178.14
max (x) 448
min (x) -752
Table 5.6: Time Sync Accuracy (ns) for PTP Slave from PTP Master
The density plot obtained from those 287 samples is shown in Figure 5.4. The time sync peak is almost towards the origin though slightly tilted towards the negative side (a little more than N1’s average value). On an average it can be seen that the time synchronization accuracy of the PTP Slave to have a deviation range between -250 ns and 250 ns although there is a larger deviation pattern with switches than for without switches. The density curve becomes more broader in Figure 5.4 than in Figure 5.3.
Figure 5.4: Density Plot for the PTP Slave’s deviation or time sync accuracy from PTP Master
5.2.3.3 N3 : Three-ended Application Set-Up
The Time Sync Accuracy for both PTP Slave1 and PTP Slave 2 was measured with respect to PTP Master (terminology according to Figure 4.15). This time three probes, one from Master IED and other two from the Slave IEDs were connected to the oscilloscope to obtain
50 the results which are shown in Table 5.7. They are compiled by collecting a sample subset of 287 sample values which are measured using the oscilloscope. The results for PTP Slave 2 seem to have larger deviation than PTP Slave 1.
Parameters PTP Slave1 (ns) PTP Slave2 (ns)
x¯ 11.29 37.45
σ 144.75 160.59
max (x) 552 776
min (x) -536 376
Table 5.7: Time Sync Accuracy (ns) for both the PTP Slaves from PTP Master
The density plot obtained from those 287 samples is shown in Figure 5.5. The time sync peak is almost towards the origin though slightly tilted towards the positive side for both the IEDs. On an average it can be seen that the time synchronization accuracy of the PTP Slaves to have a deviation range between -400 ns and 400 ns. The density curve is more broader and less sharper for PTP Slave 2 than for PTP Slave 1.
Figure 5.5: Density Plot for both the PTP Slaves’ deviation or time sync accuracy from PTP Master
5.2.3.4 N4 : Three-ended Application Set-Up with GPS to switch
The Time Sync Accuracy for PTP Slave1, PTP Slave 2 and PTP Slave 3 was measured with respect to PTP Master (terminology according to Figure 4.16). This time four probes, one from GPS clock and other three from the Slave IEDs were connected to the oscilloscope to obtain the results which are shown in Table 5.8. They are compiled by collecting a sample subset of 287 sample values which are measured using the oscilloscope.
51 Parameters PTP Slave1 (ns) PTP Slave2 (ns) PTP Slave3 (ns)
x¯ 2.46 -36.32 54.56
σ 138.62 174.85 163.02
max (x) 408 624 776
min (x) -624 -752 -416
Table 5.8: Time Sync Accuracy (ns) for the three PTP Slaves from GPS clock
The density plot obtained from those 287 samples is shown in Figure 5.6. The time sync peak is almost towards the origin for PTP Slave 1, though slightly tilted towards the positive side for PTP Slave 3 and towards negative side for PTP Slave 2. On an average it can be seen that the time synchronization accuracy of the PTP Slaves to have a deviation range between -400 ns and 400 ns. The density curve seems to be more perfect and sharper, less broader for PTP Slave 1 than for the other two slaves. This could be because both the PTP Slave 2 and 3 are on the opposite side of the GPS signal while PTP Slave 1 is connected to the same switch as the GPS signal. This is why the average of PTP Slave 1 is almost zero and the standard deviation is also not as much as for PTP Slaves 2 and 3.
Figure 5.6: Density Plot for the three PTP Slaves’ deviation or time sync accuracy from PTP Master
5.2.3.5 N6 : Unlocked to UTC
The Time Sync Accuracy for PTP Slave1 and PTP Slave 2 was measured with respect to PTP Master (terminology according to Figure 4.18). This time three probes, one from PTP Master IED and other two from the Slave IEDs were connected to the oscilloscope to obtain
52 the results which are shown in Table 5.9. They are compiled by collecting a sample subset of 287 sample values which are measured using the oscilloscope. Since in this case the network is unlocked from UTC time reference, the time quality would be bad, but as long as the two Slave IEDs are in sync with the IED that became the Master through BMC, the application should not be blocked.
Parameters PTP Slave1 (ns) PTP Slave2 (ns)
x¯ -126.52 -39.86
σ 107.46 147.85
max (x) 248 488
min (x) -624 -440
Table 5.9: Time Sync Accuracy (ns) for both the PTP Slaves from PTP Master
The density plot obtained from those 287 samples is shown in Figure 5.7. The curves look really bad when compared to the situation when the system is locked to UTC. They are not so symmetrical when compared to the other cases. However, here the whole concept is relative to the master and as long as the slaves are within the good time accuracy range of the master, the application will not have a problem.
Figure 5.7: Density Plot for the two PTP Slaves’ deviation or time sync accuracy from PTP Master
53 5.3 E3: Stress Testing
Figure 5.8: Topology for Stress Testing
This category of testing as the name suggests was done to test the performance of Ethernet based LDP communication with PTP for time synchronization in a heavy traffic-oriented network scenario. The N4 network topology was used for this test with some necessary additions to the network like an extra PC as shown in Figure 5.8. The main aim of this setup is to stress the single communication medium between the two switches so as to check the effect these extra heavy traffic load have on the data communication messages and PTP synchronization messages. In order to do that, the Sender application (see Section 4.1.2 for more details) is installed on the sender host 1, which is directed to send additional UDP traffic packets to the receiver host 2’s IP address. The exact configuration details of the Sender Application which amounts to the number of additional UDP packets sent for this test E3, are shown in Table 5.10. Note that multiple instances of this application were launched on host 1 to vary the traffic load in order to utilize the link (single communication line between the two switches) throughput from 0% to 100%. The amount of traffic sent can be adjusted, by increasing/decreasing the size and frequency of packet bursts in the application.
Parameters Values
Packet Settings in Sender Application
54 Host IP subjective
Port Number subjective
Packet Size 1500
Cycle in ms 100
Packets per cycle Start 1000
Packets per cycle End 1000
Duration INF
Status START/STOP(default)
Table 5.10: Configuration Parameter Values for one instance of Sender Application
The network equipment was configured based on the setting parameters for Communication and PTP as described in Section 4.3. The value of those setting parameters that were used for this particular test E3, are mentioned in Table 5.11. Those that are not explicitly mentioned, have the same default vales as mentioned in Section 4.3. The other values which are mentioned as subjective in those tables, differ between each clock and IED like the MAC addresses. Another important point is that the GPS clock used for this test is different from the clock used for Performance and Functionality Tests. Since the deviation offsets of both the clocks are different from one another, the values for this test cannot be compared with the values obtained in the previous tests as for this new clock, the deviation factor never goes to the negative side.
Parameters Values
PTP settings in the IEDs
AP Operation ON
Network Protocol UDP/IPv4(L3)
PTP ON
PTP settings in the L2 Switch
PTP Operation ON
PTP Mode v2-boundary-clock
55 PTP Management ON
PTP Status Is Synchronized
Two Step ON
Time Source ptp
UTC Offset Valid ON
Time Traceable OFF
Frequency Traceable OFF
PTP Timescale ON
Table 5.11: Configuration Parameter Values for Stress Testing
The results of the effect this had on LDP Application and Time Synchronization are explained in the following subsections.
5.3.1 Effect on Application Communication
The effect of additional traffic load on the data communication messages was studied by initially starting with 0% additional UDP traffic load and then increasing it to 98%. The Single Phase Relay Tester was connected to two (Slave 2 and 3) out of the three IEDs as shown in Figure 5.9 just as described in the E1 evaluation test. With the variation in traffic load, the values of differential current and bias current were recorded. Until 18% traffic load, both the differential and bias current values were as expected and all the data communication messages that were sent, were also received. But when the traffic load was increased beyond 18%, the values for the differential current and bias current started shifting randomly and unexpected fluctuating values were recorded at high traffic loads for the Slaves 2 and 3 although the average values for Slave 1 remained almost the same irrespective of the additional load. This can be attributed to the fact that no current was actually applied on Slave 1, so the local current at Slave 1 was zero. Once the connected between Slave 1 to Slave 2 and 3 were broken, the remote current values at Slave 1 also became zero and hence the total differential current at Slave 1 was zero. Another interesting result was that only the communication between Slaves 2 and 3 were effected completely. The communication between Slave 1 to 2 and 3 was still active and not completely broken although most of the packets were delayed or dropped. However all the packets between Slaves 2 and 3 were dropped and the the differential current of these two Slaves were affected the most on high traffic load. Overall scenario was that some of the data packets were delayed while others were dropped at 98% traffic flooding. This is because unlike for the PTP messages that are given priority, the data communication messages were
56 not prioritized. Perhaps if they were prioritized, then irrespective of additional traffic load, they would have been delivered correctly. The results are shown in Table 5.12.
Figure 5.9: Measuring Line Differential Current during Stress Testing
0% Additional Traffic Load
Parameters Slave1(s) Slave2(s) Slave3(s)
Diff. Current (mA) 6.82 6.63 6.95
Bias Current (mA) 403.3 402.5 401.8
18% - 98% Additional Traffic Load
Parameters Slave1(s) Slave2(s) Slave3(s)
Diff. Current (mA) 7.02 390.2 391.8
Bias Current (mA) 391.5 389.3 394.5
Table 5.12: Diff and Bias currents measured from N4 set-up during Stress Testing.
5.3.2 Effect on PTP Communication
The effect of additional traffic load on PTP messages was studied by starting the percentage of additional traffic load sent into the network at 0% and then slowly increasing it to 100% to see the effect this has on the time sync of the three PTP Slaves with the GPS clock as shown in Figure 5.10. The PPS output signal from the three PTP Slave IEDs and the GPS clock were connected to an oscilloscope (see Section 4.1.1 for more details and Chapter 4.3 for
57 configuration settings of the oscilloscope) by using probes. The oscilloscope was then used to measure the Time Synchronization Accuracy by monitoring the deviation of the PPS pulses of the Slave nodes from the main GPS signal’s PPS. The time sync accuracy did not differ much throughout the increase of the additional traffic load (except for small fluctuations in values) which led to the fortunate conclusion that no amount of additional traffic sent (even when the link was tried to flooded completely) affected the PTP messages in any manner. The results of this are portrayed in Table 5.13 at traffic load percentages 0%, 18%, 66% and 98%. The reasoning behind why the PTP messages were not affected is that according to the IEEE 1588 and IEC61850 standards for communication, the PTP messages are given priority over other traffic unless configured otherwise.
Figure 5.10: Measuring Time Sync Accuracy during Stress Testing
Priority of PTP messages in PSNs : There are a couple of reasons as to why the PTP messages were not affected by the heavy load traffic unlike the data communication mes- sages. One of them is the same reason which is attributed to its accuracy i.e the hardware timestamping mechanism as described in Section 2.5.1.4. The dedicated hardware approach is unaffected by operating system or network traffic latency. Another reason is also the usage of the QoS protocol in Ethernet networks for the IEEE 1588 protocol as mentioned in [46]. The PTP messages are sent as 802.1Q tagged Ethernet frames with a default VLAN 0 and default priority 4 in Ethernet 802.3 with Ethertype 0X88F7. Also 802.1AS is a subset of the IEEE 1588 Precision Time Protocol which is restricted to IEEE 802 networks (e.g. Ethernet, Wi-Fi). The 802.1Qav/s priority is inserted into the Ethernet header used for VLANs and then switches use this priority information to shape the traffic being transmitted out a port at that priority level.
58 PTP Slave 1
% of load 0% load 18% load 66% load 98% load
x¯ (ns) 491.49 482.14 438.28 415.88
σ (ns) 122.07 131.85 130.27 121.76
max (x) (ns) 860 856 1002 800
min (x) (ns) 112 168 144 8
PTP Slave 2
% of load 0% load 18% load 66% load 98% load
x¯ (ns) 552.88 573.47 480.67 436.02
σ (ns) 116.54 132.04 122.74 129.02
max (x) (ns) 1003 992 928 848
min (x) (ns) 272 232 112 120
PTP Slave 3
% of load 0% load 18% load 66% load 98% load
x¯ (ns) 559.35 618.68 530.52 514.76
σ (ns) 138.02 145.54 140.50 135.20
max (x) (ns) 960 1009 1006 1008
min (x) (ns) 152 280 80 200
Table 5.13: Time Sync Accuracy (ns) Results for the three Slaves with variation in Additional Traffic load
As is visible in Table 5.13, the values for the average time synchronization accuracy, standard deviation of time synchronization accuracy and maximum time synchronization accuracy devi- ations to the right and left sides from the reference GPS signal. These values were calculated after collecting a subset of 287 samples for all the three PTP Slaves using the oscilloscope.
59 (a) No Additional Traffic (b) 18% Additional Traffic
(c) 66% Additional Traffic (d) 98% Additional Traffic
Figure 5.11: Time Synchronization Accuracy measured during stress testing
The density plot based on those set of 287 samples is shown in Figure 5.11. From Table 5.13 and Figure 5.11, it is clear that the PTP Synchronization messages were not affected with any amount of traffic flooding in the network. Except for slight fluctuations in the values at the different percentiles of traffic, the plots do look pretty much similar. Also it is good to note that in all the four plots, the blue curve, which is that of PTP Slave 1 which lies on the same side as the GPS clock signal, has the least deviation. The blue peak is more towards the left than the green and red peaks.
60 Chapter 6
Feasibility for WAN Deployment
This chapter focuses on analysing if the proposed solution of using Ethernet based IP networks combined with PTP for time synchronization is a feasible option for a WAN deployment scenario by the industrial companies dealing with the LDP application. Although lot of academic research has happened in this area [2], no large scale deployment has happened in reality to the best of my knowledge. Even from the perspective of the thesis work, it is one thing to create small scale pilot set-ups for evaluation but a totally different thing to evaluate on a real WAN network. Therefore the following sections analyse and test the proposed solution on a large scale WAN at a system operator’s facilities.
6.1 WAN Description and Set-up
The simplified version of the WAN set-up is depicted in Figure 6.1.
Figure 6.1: WAN test network
The below points describe the network and the customer company’s set-up in more detail:
1. This Packet Switched Wide Area Network was mainly used for services like Surveillance
61 and Video-On-Demand. Apart from the PSN technology, they also parallely have the SDH network just for running the LDP communication. The maintenance of this SDH network was getting more and more difficult. So they wanted to dispose the SDH network and use the existing PSN for running the LDP application too.
2. The Ethernet/IP based WAN network was over 400 km to 500 km long; designed in a ring shaped topology, although it could also be changed into a mesh topology upon requirement like through connecting the possible short route as shown in Figure 6.1.
3. The network had 31 hops (R1 to R31), such that each router was a Cisco ASR920 model router with PTP Telecom Profile compatibility.
4. Each hop Rx shown in the figure, represents an entire substation in reality. Though this is still a test network, the customer company has got some of the substations running live.
5. There were mainly two ways for communication from any Rx to Ry node - either clockwise (secondary route) or anti-clockwise (primary route).
6. The network had a central Meinberg LANTIME 3000 clock which was connected to a GPS antenna and this clock was the PTP grandmaster, thus synchronizing all other nodes in the network.
7. Another point to be noted is that the substations used the RED670 ABB IEDs v1.2 in their network, which were not PTP compatible. Hence the workaround solution for the customer to then synchronize the nodes in the Ethernet network was by using Synchronous-Ethernet to achieve time synchronization.
8. Thus using the Sync-E for time synchronization, the LDP application was run by the customer on this WAN and when a current of 0.4 A was applied across their IEDs of v1.2, using a relay tester (also called a Freja; on a scale of 1:1000), a differential current of 15 A was recorded.
9. They had to maintain symmetry even for the Sync-E solution and often when doing route switching from primary to secondary route, they encountered trips in the application due to the asymmetry.
10. Also, every time the customer company tried to switch routing from primary to secondary and vice versa, they encountered a blocking for a relatively long time of about 0.5 mins, and so while switching, they had to turn off the LDCM communication modules.
6.2 Testing and Results
In addition to the customer company’s network equipment, the following gears were also used for the testing :
62 1. ABB RED670 IEDs v1.2 - two
2. ABB RED670 IEDs v2.2 prototype with Ethernet ports - two
3. ABB AFS 665 L2 switch - one
4. Meinberg LANTIME 3000 GPS clock - one
5. required converters and cables were also used.
The test network was such that the beginning of the ring i.e R1 and the end of the ring i.e R31 routers were in the same center of operation of the visit along with their (customer’s) central Meinberg clock. Infact their Meinberg clock was connected to the R1 node. So the second Meinberg clock was connected to the R31 node. Hence both these Meinberg clocks acted as boundary clocks in converting the telecom profile of PTP to power profile. So in all the test cases, each of these meinberg clocks were connected to one of the IEDs. Basically both the IEDs were at the two extreme ends of the ring network. The aim of the tests was to get them both to communicate the data messages and get synchronized over the WAN network. Then when current was applied across them, the differential current measured was recorded. There were also some changes in the set-ups done to check the effect of route switching on the application tripping and blocking.
6.2.1 Case 1 : With the ABB v1.2 IEDs
The first scenario that was tested was using the ABB RED670 v1.2 IEDs. The set up for this test is shown in Figure 6.2.
Figure 6.2: WAN set-up for v1.2 IEDs testing
63 Since the 1.2 IEDs do not support PTP, the solution used for the time synchronization was something different. At the R31 end where the second Meinberg clock was present, it had an IRIG-B output that was configured and connected to the ABB IED on that end. On the other end, it was slightly more complicated since the customer’s Meinberg clock did not have an IRIG-B output. Hence an ABB v2.2 IED was first connected through the AFS switch (which acted as a converted between the RJ 45 cable and optical cable) to the clock and was synchronized using PTP. The PPS signal obtained from the GTM board of this 2.2 IED was then fed into the 1.2 IED at this end. Hence both these 1.2 IEDs were synchronized through the primary route. The data messages were communicated well through the LDCM modules (since these IED versions also did not have Ethernet ports) and when a current of 0.4 A was applied from the Freja to the IEDs, the results recorded are shown in Table 6.1.
Parameters Value
Applied Current(A) 0.4
Freja’s Scaling 1:3000
Bias Current(A) 1200
Diff. Current(A) 65
Tx. Delay 1.9ms
Asymmetry 0
Table 6.1: Results for differential current from v1.2 IEDs over the primary route
In this case the differential current recorded was 65 A; which was a slightly larger value than expected from the LDP application’s perspective. This larger value could be attributed to lesser accuracy arising out of the additional connection done through the v2.2 IEDs as shown in Figure 6.2. Since we add the v2.2 IED which is configured to run in the 2 Mbit mode, as opposed to the v1.2 IED which is configured to run in the 64 Kbit mode, the application data first gets compressed to fit the “legacy” 64 kBit communication link by v1.2 IED and then is uncompressed by v2.2 IED. This compression in the v1.2 IED leads to small errors in the current representation. However in the next scenario where we do not use the v1.2 IEDs, we can see better results due to the absence of any kind of compression.
6.2.2 Case 2 : With the ABB v2.2 IEDs
The next case was to test it with the v2.2 prototype IEDs which had the Ethernet ports capable of transmitting data over Ethernet/UDP by installing and running the UDP scripts. The test set-up is shown in Figure 6.3. On the R1 side, the IED is synced through PTP with the
64 rest of the network by using the AFS switch as the converter between the 100BASE-TX and 100BASE-FX cables. Hence here the Meinberg clock converts the Telecom profile supported by the devices in the WAN network to the Power profile of PTP which is compatible with the Power Utility profile in the ABB IEDs. On the R31 end due to the absence of another AFS L2 PTP compatible switch, IRIG-B output from the Meinberg clock was used to synchronize the IED on that end. So in Case 2 testing the LDP communication data messages are running over Ethernet network with the IEDs communicating through UDP datagrams (LDCM frames wrapped in UDP header) and the time sync PTP messages also running on the same Ethernet medium (refer point 3 under Section 6.3). Now upon applying the current using the Freja, the differential current measured was recorded. However this Case 2 is further divided into three sub-cases based on the type of routes used to send the data and time sync packets over the network. The results for each of those cases are described in detail below :
Figure 6.3: WAN set-up for v2.2 IEDs testing
Parameters Value
Applied Current(A) 0.4
Freja’s Scaling 1:3000
Primary Route for data and time sync
Bias Current(A) 1200
Diff. Current(A) 5
65 Tx. Delay 0.4ms
Asymmetry 0
Secondary Route for data and Primary Route for time sync
Bias Current(A) 1200
Diff. Current(A) 5
Tx. Delay 4ms
Asymmetry 2.7ms
Table 6.2: Results for differential current from v2.2 IEDs
6.2.2.1 Case 2.1 : Data and Sync messages over Primary route
There is a direct connection between hops R1 and R31 as shown in Figure 6.3. This route is called the primary route and first the LDP data messages and time sync messages were routed to go through this primary route. The results of the differential current measured in this scenario when the IEDs were in sync are shown in Table 6.2. A differential current of 5 A was recorded hence reducing the measured differential current from 15 A to 5 A. Infact the data packets transmission delay when using the Ethernet/UDP has also gone down to just 0.4 ms as opposed to the 1.9 ms that was recorded in Case 1. Since it was over the primary route, there was no asymmetry in this case.
6.2.2.2 Case 2.2 : Data over Secondary and Sync over Primary routes
The next scenario is where the data messages alone were configured to run over the secondary route, thus through all the 31 hops from R1 to R31, while the time synchronization was still achieved through the primary route with just one hop. Now when a current of 0.4 A from the Freja was applied across the two IEDs, the results recorded are shown in Table 6.2. When a bias current of 1200 A was recorded, a differential current of 5 A was recorded which is basically the same when the data messages were configured to run over the primary route. So considering that the current values are transmitted across the 500 km WAN, these results are really good for LDP application over the Ethernet. The transmission delay is also pretty small for the secondary route, only being around 4 ms. Since now there are 31 hops, there was an asymmetry of 2.7 ms that was detected.
Though the ABB v2.2 IEDs were expected to provide a differential current lesser than 5 A, the slightly higher value recorded in this scenario was attributed to the fact that the TRM board on these pilot prototype IEDs were not calibrated. Hence to override that little glitch,
66 a current of 1 A was applied from the Freja to the IEDs (scale being 1:1000 this time), so a bias current of 1000 A was detected on the IEDs with a differential current of only 2 A which was a much better result. These results are shown in Table 6.3.
Parameters Value
Applied Current(A) 1
Freja’s Scaling 1:1000
Secondary Route for data and Primary Route for time sync
Bias Current(A) 1000
Diff. Current(A) 2
Tx. Delay 4ms
Asymmetry 2.7ms
Table 6.3: Results for differential current from v2.2 IEDs - part 2
6.2.2.3 Case 2.3 : Data and Sync messages over Secondary Route
The last sub-case under Case 2 was when the time synchronization was also changed and the sync messages were made to go through the secondary route. At first, everything seemed good and when the differential current was applied the following results were recorded as shown in Table 6.4.
Parameters Value
Applied Current(A) 1
Freja’s Scaling 1:1000
Secondary Route for data and time sync
Bias Current(A) 1000
Diff. Current(A) 2
Tx. Delay 3ms
Asymmetry 1ms
Table 6.4: Results for differential current from v2.2 IEDs - part 3
67 However after a while (approximately 10 mins), the second Meinberg clock started to go out of sync from the first Meinberg clock. However since the internal oscillator of the ABB Meinberg was still having a good time quality, the results in Table 6.4 could be recorded which was just temporary. After a while the time sync was lost and LDP application got blocked and the results changed from the values in Table 6.4 to as shown in Table 6.5.
Parameters Value
Applied Current(A) 1
Freja’s Scaling 1:1000
Secondary Route for data and time sync
Bias Current(A) 1000
Diff. Current(A) 150
Tx. Delay 3ms
Asymmetry 1ms
Table 6.5: Results for differential current from v2.2 IEDs - part 4
The exact reason behind this behaviour of why the time sync over secondary route did not work, could not be found, and needs to be further investigated. Nevertheless, the focus has been shifted towards the WAN setup to figure out the issue. In order to figure which part of the WAN was having the error, a different set-up was tried out as shown in Figure 6.4.
An extra Meinberg clock which was at R21 node was now used as a PTP Master instead of the Meinberg connected to R1, in order to sync the ABB Meinberg at R31. So basically the sync was now through the this temporary tertiary route as shown in Figure 6.4. Then the sync worked and the results went back to as they were in Table 6.4. So the reason of why it did not work over the secondary route is still unknown, though suspicion is with respect to some configuration issues of Cisco routers in the WAN.
Another interesting result was that whenever there was a route switching from the secondary route back to the primary route, for the customer, the average blocking time period was half a minute and when our proposed solution was tested, it took around 2 s. Also the blocking time period when switching from primary to secondary was 100 ms.
68 Figure 6.4: WAN set-up for v2.2 IEDs testing
6.3 Challenges during Testing
Following were the main challenges and set backs that were faced during the testing :
1. The initial plan was to test the synchronization using PTP in both the v2.2 IEDs. Unfortunately due to the absence of a second AFS switch, there was a lack of a converter on one side and hence the IRIG-B output had to be used to establish the sync on R31 end of the network. Hence one of the set backs was not being able to completely deploy PTP sync between the two IEDs over the network - It was instead PTP on one end and IRIG-B on the other end.
2. There were some issues with the prototype code that was used to implement the N6 case of being unlocked from the UTC time reference. The v2.2 IEDs as per logic compared if the two ends had the same GmId when using PTP for time sync. However the Meinberg clocks did not completely transfer the GmId value. Even though the ABB Meinberg clock at the R31 end was synced to the customer’s Meinberg clock (being the grandmaster in the network) at R1 end, the ABB Meinberg just retained its own MAC address as the GmId and did not get the GmId value set to the customer Meinberg clock’s MAC address. Hence since both the clocks did not have the same GmId, the end IEDs used to keep the LDP application blocked. Thus some adjustments had to be with the code and then the changed source code was compiled and used.
69 3. The Meinberg clocks which were just used as boundary clocks for the PTP profile conver- sion did not allow the passage of any other type of packets and hence seperate channels from the IEDs to the R1 and R31 routers had to be connected for the data messages to be transported. However once within the network both the data and PTP time sync messages were along the same path.
4. Due to the absence of an accurate time measurement device, the exact time period of blocking of LDP during the route switching from primary to secondary and vice versa, could not be recorded.
6.4 Customer Requirements after the Testing
After the testing, it was good to get some customer feedback on the proposed solution of using Ethernet with PTP for LDP application. Following were the expressed points :
1. The existing network equipment, i.e the IEDs were not PTP compatible and the customer was interested in a solution for the existing set-up.
2. An important point of concern was the fact that the v2.2 IEDs do not support the Telecom Profile of PTP. They wanted telecom profile support in the future versions.
3. Using Meinberg clocks as converters between profiles is too costly.
6.5 General Industry Challenges for WAN Deploy- ment
This Chapter portrayed the specific case of WAN testing and the challenges that arouse from it. Looking at the bigger picture, in general, there could be plausible difficulties of deploying this thesis work’s solution on a large scale in the industry (which could also be a reason as to why there are not many deployments using PSN+PTP, existing today) and these general issues are discussed below.
While doing a large scale network deployment, the following points can be challenging :
• PTP equipment migration - Due to the fact that switches with IEEE 1588 support are costly and not available in most of the current network infrastructures, one would have to change the existing nodes and hardware.
• Using PTP converters would not be a viable option in the current networks as it would just increase the number of nodes and these converters can be too costly.
• The Ethernet setup can cause communication failures and more regulations in terms of monitoring and control and the synchronization errors could increase, depending on the
70 amount of traffic and the number of network devices on the sync path. Since its packet switched networks, packet dropping is a common phenomenon that can cause the LDP application to be blocked.
• Other security challenges, like security of data messages send, DDoS Attacks on the network, network carrying the messages and PTP protocol messages could get compro- mised.
• Prioritizing the data messages over other messages in the network
• Running multiple services in parallel can cause bandwidth issues.
• Compatibility between multiple PTP profiles.
71 Chapter 7
Conclusion and Discussion
7.1 Conclusion
The solution proposed in this thesis work is a promising alternative solution to the existing communication architectures like SDH/SONET for the LDP application. Although some hur- dles regarding PSNs and PTP compatibility with WAN deployment would have to be overcome as stated in Section 6.5, the solution does hold a deployment future. Using PTP for time syn- chronization is trending in a lot of other domains as well due to its compatibility with the Ethernet network infrastructure. It is not long before the network devices around the world also start supporting the PTP stack, following which it will become easier for implementing this solution on a country wide network.
7.2 Future Work
Given that the solution seems feasible, more testing could be done in WAN scenarios, in order to improve several aspects like response time of PSN and PTP in asymmetric environments and ponder upon the plausible challenges like cost of QoS maintained to provide priority to the LDP+PTP traffic amongst other ordinary network traffic. It would also be useful to test the solution with TCP instead of UDP and to see how the TCP handshaking and header might effect the network load.
One more interesting area where more investigation needs to be done is with the N6 topology. The solution proposed in order to migrate away from the dependency on the GPS signal seems promising, although more testing and handling of the scalability issues of the proposed solution need to be done.
Another point to work on is to see if other PTP profiles need to be included in the net- work devices (specially the ABB IEDs from the thesis perspective) and maybe also utilize the
72 redundancy links to see if it is possible to change links depending on which link offers better time quality.
With regards to the testing done in this thesis, more rigorous Stress tests could be done in the future since the results of why Slave 1 was not affected and only Slaves 2 and 3 were affected could not be explained properly as portrayed in Section 5.3.1. Another test to be focussed upon is to try and get the Case 2.3 described in Section 6.2.2.3 working.
73 Bibliography
[1] “IEEE Standard Electrical Power System Device Function Numbers, Acronyms, and Contact Designations”. In: IEEE Std C37.2-2008 (Revision of IEEE Std C37.2-1996) (2008), pp. 1–48. doi: 10.1109/IEEESTD.2008.4639522. [2] A. Aichhorn et al. “Realization of Line Current Differential Protection over IP-based networks using IEEE 1588 for synchronous sampling”. In: 13th International Conference on Development in Power System Protection 2016 (DPSP). 2016, pp. 1–6. doi: 10. 1049/cp.2016.0026.
[3] ABB. Line Protection. Accessed on : 3.08.2017. url: https://www.scribd.com/ document/342881606/Lineprotectionbasics-June2008-150422025540-Convers ion-Gate02. [4] M. G. Kanabar and T. S. Sidhu. “Performance of IEC 61850-9-2 Process Bus and Corrective Measure for Digital Relaying”. In: IEEE Transactions on Power Delivery 26.2 (2011), pp. 725–735. issn: 0885-8977. doi: 10.1109/TPWRD.2009.2038702. [5] IEC TR 61850-90-1:2010. Communication networks and systems for power utility au- tomation - Part 90-1: Use of IEC 61850 for the communication between substations. Accessed on : 5.07.2017. url: https://webstore.iec.ch/publication/6007. [6] T. C. Wright. “The synchronous digital hierarchy standard”. In: Second IEE National Conference on Telecommunications 1989. 1989, pp. 297–302.
[7] Ciena Corporation. SONET/SDH Network Modernization Is Long Overdue, Whitepaper, 2013, Maryland, USA. Accessed on : 6.07.2017. url: http://www.ciena.com/insig hts/white-papers/SONETSDH-Network-Modernization-Is-Long-Overdue_prx. html. [8] K. M. Abdel-Latif et al. “Laboratory Investigation of Using Wi-Fi Protocol for Transmis- sion Line Differential Protection”. In: IEEE Transactions on Power Delivery 24.3 (2009), pp. 1087–1094. issn: 0885-8977. doi: 10.1109/TPWRD.2009.2013665. [9] R. H. Khan, T. S. Ustun, and J. Y. Khan. “Differential protection of microgrids over a WiMAX network”. In: 2013 IEEE International Conference on Smart Grid Communi- cations (SmartGridComm). 2013, pp. 732–737. doi: 10.1109/SmartGridComm.2013. 6688046.
74 [10] S. Fukushima et al. “Development of line current differential relay over native Ethernet”. In: 12th IET International Conference on Developments in Power System Protection (DPSP 2014). 2014, pp. 1–5. doi: 10.1049/cp.2014.0030. [11] S. T. Watt et al. “Understanding and applying precision time protocol”. In: 2015 Saudi Arabia Smart Grid (SASG). 2015, pp. 1–7. doi: 10.1109/SASG.2015.7449285. [12] C. A. Outra et al. “Substation time synchronization in today and future architectures”. In: 12th IET International Conference on Developments in Power System Protection (DPSP 2014). 2014, pp. 1–6. doi: 10.1049/cp.2014.0082. [13] Alcatel.Lucent. Technology White Paper : Synchronization Over Ethernet Networks. Accessed on : 30.06.2017. url: https://www.scribd.com/document/171599706/ Synchronization-Over-Ethernet. [14] “IEEE Standard for a Precision Clock Synchronization Protocol for Networked Measure- ment and Control Systems”. In: IEEE Std 1588-2008 (Revision of IEEE Std 1588-2002) (2008), pp. 1–269. doi: 10.1109/IEEESTD.2008.4579760. [15] “IEEE Standard for a Precision Clock Synchronization Protocol for Networked Measure- ment and Control Systems”. In: IEEE Std 1588-2002 (2002), pp. i–144. doi: 10.1109/ IEEESTD.2002.94144. [16] Cisco. Cisco Connected Grid Switches System Management Software Configuration Guide. Accessed on : 15.06.2017. url: http://www.cisco.com/c/en/us/td/docs/ switches/connectedgrid/cg- switch- sw- master/software/configuration/ guide/sysmgmt/CGS_1000_Sysmgmt/sm_ptp.pdf. [17] Jeff Laird. White Paper : PTP Background and Overview. Accessed on : 19.06.2017. url: https://www.iol.unh.edu/sites/default/files/knowledgebase/1588/ ptp_overview.pdf. [18] Y. Li et al. “A study of precision of hardware time stamping packet traces”. In: 2014 IEEE International Symposium on Precision Clock Synchronization for Measurement, Control, and Communication (ISPCS). 2014, pp. 102–107. doi: 10 . 1109 / ISPCS . 2014.6948700.
[19] Douglas Arnold. End-to-End Versus Peer-to-Peer. Accessed on : 19.06.2017. url: htt ps://blog.meinbergglobal.com/2013/09/19/end-end-versus-peer-peer/.
[20] Douglas Arnold. End-to-End Versus Peer-to-Peer. Accessed on : 19.06.2017. url: htt ps://blog.meinbergglobal.com/2013/09/19/end-end-versus-peer-peer/.
[21] Douglas Arnold. What Makes a Master the Best? Accessed on : 19.06.2017. url: https: //blog.meinbergglobal.com/2013/11/14/makes-master-best/.
[22] Hirschmann. Precision Clock Synchronization – IEEE 1588. Whitepaper, Rev. 1.2. url: http://www.industrialnetworking.com/pdf/Hirschmann_IEEE_1588.pdf.
75 [23] “IEEE Standard Profile for Use of IEEE 1588 Precision Time Protocol in Power System Applications”. In: IEEE Std C37.238-2017 (Revision of IEEE Std C37.238-2011) (2017), pp. 1–42. doi: 10.1109/IEEESTD.2017.7953616. [24] “IEC/IEEE International Standard - Communication networks and systems for power utility automation Part 9-3: Precision time protocol profile for power utility automation”. In: IEC/IEEE 61850-9-3 Edition 1.0 2016-05 (2016), pp. 1–18. doi: 10.1109/IEEESTD. 2016.7479438. [25] “IEC/IEEE Draft Communication Networks and Systems for Power Utility Automation - Part 9-3: Precision Time Protocol Profile for Power Utility Automation”. In: IEC/IEEE
P61850-9-3F DIS, F ebruary2016 (2016), pp. 1–20. [26] Cisco. G.8275.2 Telecom Profile. Accessed on : 7.07.2017. url: http://www.cisco. com/c/en/us/td/docs/routers/asr920/configuration/guide/timing/b- timing-sync-xe-16-5-asr920/g_8275_2.pdf. [27] Iometrix Sebastien Jobert. TELECOM PROFILE FOR TIME/PHASE ITU-T G.8275.1. Accessed on : 7.07.2017. url: https://www.atis.org/wsts/docs/2014/4- 4- Iometrix_Jobert_iTU_G.8275.1.pdf. [28] “IEEE Standard for Intelligent Electronic Devices Cyber Security Capabilities”. In: IEEE Std 1686-2013 (Revision of IEEE Std 1686-2007) (2014), pp. 1–29. doi: 10.1109/ IEEESTD.2014.6704702. [29] ABB Group Relion. Line differential protection RED670 2.1 IEC Product guide. Accessed on : 23.05.2017. url: https://library.e.abb.com/public/76afcb6f5b2349a88b e57fa44a78a060/1MRK505346-BEN_B_en_Product_Guide__Line_differential_ protection_RED670_2.1.pdf.
[30] ABB Group. Relion product family. Accessed on : 23.05.2017. url: http : / / new . abb . com / substation - automation / products / protection - control / relion - product-family. [31] “IEEE Standard for N Times 64 Kilobit Per Second Optical Fiber Interfaces Between
Teleprotection and Multiplexer Equipment”. In: IEEE Std C37.94-2002 (2003), 01–. doi: 10.1109/IEEESTD.2003.94245.
[32] ABB Group. AFS66X Switch. Accessed on : 24.05.2017. url: http://new.abb.com/ network- management/communication- networks/optical- networks/afs660- switch. [33] Meinberg Funkuhren GmbH & Co KG. IMS - LANTIME M3000: Versatile and Modular Time and Frequency Synchronization Platform. Accessed on : 24.05.2017. url: https: //www.meinbergglobal.com/english/products/modular-3u-sync-system.htm.
76 [34] ABB Group. AFR677 Router. Accessed on : 24.05.2017. url: http://new.abb.com/ network- management/communication- networks/optical- networks/afr677- router. [35] Agilent Technologies. Agilent Technologies 1000 Series Portable Oscilloscopes. Accessed on : 24.05.2017. url: http://www.testequipmenthq.com/datasheets/Agilent- DSO1024A-Datasheet.pdf.
[36] Megger. Single Phase Relay Tester. Accessed on : 14.06.2017. url: http://us.megge r.com/single-phase-relay-tester-sverker650. [37] Andr´asVarga and Rudolf Hornig. “An Overview of the OMNeT++ Simulation En- vironment”. In: Proceedings of the 1st International Conference on Simulation Tools and Techniques for Communications, Networks and Systems & Workshops. Simutools ’08. Marseille, France: ICST (Institute for Computer Sciences, Social-Informatics and Telecommunications Engineering), 2008, 60:1–60:10. isbn: 978-963-9799-20-2. url: http://dl.acm.org/citation.cfm?id=1416222.1416290. [38] Dominik Klein and Michael Jarschel. “An OpenFlow Extension for the OMNeT++ INET Framework”. In: Proceedings of the 6th International ICST Conference on Simulation Tools and Techniques. SimuTools ’13. Cannes, France: ICST (Institute for Computer Sciences, Social-Informatics and Telecommunications Engineering), 2013, pp. 322–329. isbn: 978-1-4503-2464-9. url: http://dl.acm.org/citation.cfm?id=2512734. 2512780. [39] Wolfgang Wallner. “Simulation of the IEEE 1588 Precision Time Protocol in OM- NeT++”. In: CoRR abs/1609.06771 (2016). url: http://arxiv.org/abs/1609. 06771. [40] ABB Group. Protection and Control IED Manager PCM600 Product Guide. Accessed on : 25.05.2017. url: http://www04.abb.com/global/seitp/seitp202.nsf/0/ d13cebbc495afa9bc1257b1e0028edfa/\$file/PCM600_pg_756448_ENh.pdf.
[41] ABB Group. ABB AFS Finder 2.2. Accessed on : 25.05.2017. url: http://abb-afs- finder.software.informer.com/2.2/. [42] Angela Orebaugh et al. Wireshark & Ethereal Network Protocol Analyzer Toolkit. Syn- gress Publishing, 2007. isbn: 1597490733, 9781597490733. [43] Meinberg Funkuhren. General IRIG-B Dataframe and IEEE1344 format. Accessed on : 23.05.2017. url: https://www.meinbergglobal.com/download/docs/other/ ieee-1344.pdf. [44] W. Wallner, A. Wasicek, and R. Grosu. “A simulation framework for IEEE 1588”. In: 2016 IEEE International Symposium on Precision Clock Synchronization for Measurement, Control, and Communication (ISPCS). 2016, pp. 1–6. doi: 10.1109/ISPCS.2016. 7579516.
77 [45] OMNeTpp. The NED Language. Accessed on : 30.05.2017. url: https://omnetpp. org/doc/omnetpp/manual/#cha:ned-lang. [46] Audinate. Evolving networks to Audio Video Bridging (AVB). Accessed on : 6.08.2017. url: http://www.lectrosonics.com/Support/phocadownload/audinate%20avb% 20white%20paper%20v1%202.pdf.
78 Appendix
A) Script for inserting Software LDCM
#need to get a ldcm in hwCfgList in order to run in UDP mode on t a r g e t
Slot Type In Out OptStr OverrideHw p31:2 670LDCM8 8 2M 0
B) Script for configuring UDP mode
# this script is for IED:t7 (192.168.1.7 on ap1) slave # # t7:ldcm312 = 192.168.1.7:7312 <−−> 192.168.1.8:8312 = t8: ldcm312 #
# port to listen to (on all interfaces) compAttrWriteWithTrans ”App1.LDCM.312” ,”PARAM.21. Value”,”7312”
# port on remote side to conect to compAttrWriteWithTrans ”App1.LDCM.312” ,”PARAM.22. Value”,”8312”
# IP on remote side compAttrWriteWithTrans ”App1.LDCM.312” ,”PARAM.23. Value ”,”10.1.2.214”
C) NED script for OMNeT++ emulation between PC and simulated router import inet.nodes.inet.Router;
79 import libptp .src.Components.BasicBlocks.PTP BasicNode ; network Network { @display(”bgb=375,271”); submodules: router: Router { parameters: numExtInterfaces = 2; ext[0].device = ”ens5”; ext[0].filterString = ”udp or ptp”; ext[1].device = ”enp9s0”; ext[1].filterString = ”udp or ptp”; @display(”p=167,93”); } connections allowunconnected: }
D) Simulated L3 router’s routing file
i f c o n f i g : name : ext0 i n e t addr: 10.1.2.209 Mask: 255.255.255.248 mtu : 1500 Metric : 1 BROADCAST MULTICAST name : ext1 i n e t addr: 10.1.2.217 Mask: 255.255.255.248 mtu : 1500 Metric : 1 BROADCAST MULTICAST
ifconfigend.
r o u t e :
80 10.1.2.208 10.1.2.209 255.255.255.248 G 0 ext0 10.1.2.216 10.1.2.217 255.255.255.248 G 0 ext1
routeend .
E) Measurement Conditions
E.1) IED Configuration
IED Settings for Ethernet Communication and PTP Sync.
Parameters Possible Values
Access Point AP-1/2/3/4
Description Station Bus
Operation ON/OFF (default)
Redundancy None
Physical Ports subjective
IP Address subjective
Subnet Mask subjective
Default Gateway subjective Ethernet MAC Address subjective Communication PCM600 Access ON (default)/OFF
MMS ON (default)/OFF
GOOSE ON (default)/OFF
DNP3 ON (default)/OFF
FTP ON (default)/OFF
FSTAccess ON/OFF (default)
SNTPServer ON/OFF (default)
PTP ON/OFF (default)
81 Domain 0 to 255 (default is 0)
PTP Config Priority 1 0 to 255 (default is 128)
Priority 2 0 to 255 (default is 128)
Current Sync Source PTP@portnum
PTPGMId subjective
PTPGMClass subjective
PTPGMAccuracy <100ns
PTPParentId subjective
PTPSteps 1/2 (default)
PTPOwnID subjective
Table 7.1: Communication and PTP Configuration for RED670 IED
E.2) Switch Configuration
PTP Global settings
Parameters Possible Values
PTP Operation ON/OFF (default) v2-boundary-clock/ PTP Mode v2-transparent-clock (default) Operation Sync Lower Bound (ns) 0 to 999999999 (default is 30)
Sync Upper Bound (ns) 31 to 999999999 (default is 5000)
PTP Management marked/unmarked (default)
PTP Status Is/Not Synchronized
Status PTP Time Reference UTC Time (using GPS clock)
Max Offset Absolute (ns) subjective
Table 7.2: Global Configuration Parameter Values for AFS665 L2 switch
82 PTPv2 BC settings
Parameters Possible Values (some are default)
Priority 1 0 to 255 (default is 128)
Operation Priority 2 0 to 255 (default is 128)
Domain Number 0 to 255 (default is 0)
Two Step ON/OFF (default)
Steps Removed subjective Status Offset to Master (ns) subjective
Delay to Master (ns) subjective
Clock Identity MAC address of switch
Identities Parent Port Identity MAC address of master port in switch
Grandmaster Identity MAC address of GPS clock
Priority 1 128
Clock Class subjective
Grandmaster Clock Accuracy usually within 100ns
Clock Variance subjective
Priority 2 128 ptp/ntp/gps/atomicClock/ Time Source terrestiralRadio/handSet/ other/internalOscillator (default)
UTC Offset (s) -32768 to 32767 (default is 35) Local Time Properties UTC Offset Valid marked/unmarked (default)
Time Traceable marked/unmarked (default)
Frequency Traceable marked/unmarked (default)
PTP Timescale marked/unmarked (default)
Table 7.3: BC Configuration Parameter Values for AFS665 L2 switch
83 PTP BCv2 port settings
Parameters Possible Values
Port Number 1 to 11
PTP Enable marked (default)/unmarked initializing/faulty/disabled/ PTP Status listening/pre-master/master passive/uncalibrated/slave
Sync Interval 0.25/0.5/1 (default)/ 2
Delay Mechanism disabled/E2E (default)/P2P
P2P Delay subjective
P2P Delay Interval 1 (default)/2/4/8/16/32
Network Protocol IEEE 802.3 (default)/(UDP/IPv4)
Announce Interval(s) 1/2 (default)/4/8/16
Announce Timeout 2 to 10 (default is 3)
E2E Delay Interval(s) subjective
V1 Hardware Compatibility auto (default)/on/off
Asymmetry -2000000000 to 2000000000 (default is 0)
VLAN none (default)/0 to 4042
VLAN Priority 0 to 7 (default is 4)
Table 7.4: BCv2 port configuration parameters
PTPv2 TC settings
Parameters Possible Values (some are default) E2E (default)/P2P/ Delay Mechanism E2E-optimized/disabled
Primary Domain 0 to 255 (default is 0)
Operation 84 Network Protocol IEEE 802.3 (default)/(UDP/IPv4)
Multi Domain Mode marked/unmarked (default)
VLAN ID none/(0 to 4042)
VLAN Priority 0 to 7 (default is 4)
Syntonize marked (default)/unmarked
Synchronize Local Clock marked/unmarked (default) Local Current Master MAC address of Master Synchronization Offset to Master (ns) subjective
Delay to Master (ns) subjective
Status Clock Identity MAC address of switch
Table 7.5: TC Configuration Parameter Values for AFS665 L2 switch
PTP TCv2 port settings
Parameters Possible Values
Port Number 1 to 11
PTP Enable marked (default)/unmarked
P2P Delay Interval(s) 1 (default)/2/4/8/16/32
P2P Delay subjective
Asymmetry -2000000000 to 2000000000 (default is 0)
Table 7.6: TCv2 port configuration parameters
85 E.3) GPSClock Configuration
PTPv2 settings for Interface MRI1 Possible Values Parameters (some are default)
TCP/IP Address subjective
Status Netmask subjective Network Local MAC Address subjective
UUID subjective
PTP Mode one of the possible values
Port State one of the possible values
Grandmaster MAC subjective
Clock Accuracy <100ns
PTP Seconds subjective
UTC Offset subjective
Global TSU Time subjective
Domain Number 0 to 255 (default is 44)
Port Link Up YES/NO
Delay Asymmetry subjective
Clock Class subjective
Time Source one of the possible values
Leapsecond subjective
SyncE Status Enabled/Disabled(default)
Hostname PTPv2
Config Domainname subjective
Nameserver1 subjective
Nameserver2 subjective
86
Network Enable DHCP Client YES/NO(default)
TCP/IP Address subjective
Netmask subjective
Default Gateway subjective Static(default)/Router IPv6 Mode Advertisement/DHCP
IPv6 Address subjective FF01(default)/FF02/FF03/ IPv6 Multicast Scope FF04/FF05/FF08/FF0E
Enable VLAN Option YES/NO (default)
VLAN Tag 0 to 4094 (default is 0)
Priority 0 to 7 (default is 4)
Disable SSH Service YES (default)/NO
PTP Classification CUSTOM 06
Multicast TTL 0 TO 255 (default is 0)
Operating Mode PTPv2/NTP Custom (default)/ E2E IEEE 1588-2008/ P2P IEEE 1588-2008/ Power IEEE C37.238/ Telecom ITU-T G.8265.1/ Select Profile Telecome ITU-T G.8275.1/ SMPTE ST 2059-2/ AES67 Media Profile/ IEEE 802. 1AS/ Telecom ITU-T G.8275.2/ DOCSIS 3.1 Global Unicast Master/ Slave/ PTP Mode Multicast Master/Slave/ Multicast Master(auto)/
Hybrid Mode YES/NO (default) Unicast Master subjective Address 1
87 Unicast Master subjective Address 2
Delay Mechanism E2E/P2P
Domain Number 0 to 255 (default is 44)
Network Protocol UDP/IPv4orv6(L3)/IEEE 802.3(l2)
Priority 1 0 to 255 (default is 128)
Priority 2 0 to 255 (default is 128)
Time Scale PTP Standard(TAI)/Arbitrary
Announce Interval 1/2/4/8 per sec
Sync Interval 1/2/4/8/16/32/64 per sec Delay Request 1/2/4/8/16/32/64 per sec Interval
Interval Duration(s) 10 to 300 (default is 60) Announce Receipt 2 to 10 (default is 2) Timeout
Activate PTP One Step YES/NO (default) Misc Disable PTP YES (default) /NO Management Messages
Table 7.7: PTP Configuration for Meinberg LANTIME M3000
E.4) Oscilloscope Settings
Oscilloscope Vertical and Horizontal settings
Parameters Possible Values
Number of Channels 4 10x (except for the signal Attenuation from GPS which was 1x)
Volts/division 1V (on all channels)
Time/division 200ns
88 Trigger Type Edge (default)/Pulse/Video
Trigger 2V
Slope Rising (default)/Falling
Pulse Width 33ns to 10s
Table 7.8: Oscilloscope configuration parameters
89 TRITA EE 2017:102 ISSN 1653-5146
www.kth.se