applied sciences

Article Design and Application of a Distribution Network Phasor Data Concentrator

Weiqing Tao 1 , Mengyu Ma 1,*, Chen Fang 2, Wei Xie 2, Ming Ding 1, Dachao Xu 3 and Yangqing Shi 3

1 School of Electrical Engineering and Automation, Hefei University of Technology, Hefei 230009, China; [email protected] (W.T.); [email protected] (M.D.) 2 Electric Power Research Institute, State Grid Shanghai Municipal Electric Power Company, Shanghai 200437, China; [email protected] (C.F.); [email protected] (W.X.) 3 CSG Smart Grid Electrical Technology Co., Ltd., Hefei 230080, China; [email protected] (D.X.); [email protected] (Y.S.) * Correspondence: [email protected]



Received: 8 January 2020; Accepted: 20 April 2020; Published: 24 April 2020


Abstract: The wide area measurement system (WAMS) based on synchronous phasor measurement technology has been widely used in power transmission grids to achieve dynamic monitoring and control of the power grid. At present, to better realize real-time situational awareness and control of the distribution network, synchronous phasor measurement technology has been gradually applied to the distribution network, such as the application of micro multifunctional phasor measurement units (µMPMUs). The distribution network phasor data concentrator (DPDC), as a connection node between the µMPMUs and the main station, is also gaining more attraction. This paper first analyzes the communication network structure of DPDCs and µMPMUs and compares and analyzes the differences in the installation locations, functions, communication access methods and communication protocols of the phasor technology devices of the distribution network and the transmission network. It is pointed out that DPDCs not only need the functions of data collection, storage, and forwarding like transmission network PDCs, but also should be able to access more µMPMUs, and can aggregate the phasor data of the same time scale from µMPMUs by different communication methods. The communication protocol selected by DPDC should be expanded to support remote control, telemetry, fault diagnosis and other functions of distribution automation. The application requirements of DPDCs are clarified, and the key indicators of DPDCs are given as a method to evaluate the basic performance of DPDCs. Then, to address the problems of more µMPMU access, abnormal communication, and data collection with different delays that DPDC encountered, a DPDC that considers multiple communication methods is designed. Based on the Linux system and the libuv library, the DPDC is designed with event-driven mechanism and structured programming, runs multiple threads to implement multitasking, and invokes callbacks to perform asynchronous non-blocking operations. The DPDC test system and test methods are designed. The performance of the designed DPDC is evaluated through the test and the test results are analyzed. Lastly, its real-world application is disclosed, which further confirmed the value of our DPDC.

Keywords: distribution network; PDC; key performance indicators; test

1. Introduction With the distributed energy such as photovoltaics and wind power, flexible load and electric vehicles connected to the distribution network on a large scale, the stable operation of the distribution network faces new challenges in the aspects of the difficulties to quickly and accurately carry out

Appl. Sci. 2020, 10, 2942; doi:10.3390/app10082942 www.mdpi.com/journal/applsci Appl. Sci. 2020, 10, 2942 2 of 18 the state estimation, fault diagnosis and active control using monitoring and controlling methods of the traditional distribution network. To cope with these challenges, the synchronous phasor measurement technology is introduced to the distribution network [1,2] with reference to the wide area measurement system (WAMS) widely used in transmission networks. The micro multifunctional phasor measurement unit (µMPMU) is a miniaturized phasor measurement unit (PMU) that combines the functions of power distribution monitoring terminals [3], which can also be called a distribution network phasor measurement unit (DPMU) or micro phase measurement unit (µPMU) [4]. µMPMUs provide global positioning system (GPS)/BeiDou navigation satellite system (BDS)-based measurement data so that power signals at different locations can be put together for comparison and analysis, such as phase angle. µMPMUs are mainly used to solve the problems of sensing and controlling in the distribution network, including fault information extraction and fault diagnosis, state estimation, voltage control, reactive power optimization and active control [4,5]. In addition to PMUs, the transmission network WAMS also includes components such as phasor data concentrators (PDCs) and communication networks [6,7]. According to the depiction of IEEE C37.244-2013 [8], the primary and most important role of the PDC is to aggregate and forward data of multiple PMUs according to timestamps. Based on this guidance, IEEE introduced the PDC standard IEEE Std C37.247-2019 [9] in 2019, which specifies the requirements for PDC. With the development of technology in this area, PDC further has the ability to process and store data and configure more functions according to different requirements. Although the current application of µMPMUs in the distribution network is increasing, the corresponding distribution network phasor data concentrator (DPDC) has not received sufficient attention. In terms of PDC design, the PDC designed in [10] has a variety of data push logics and has a good performance in ensuring data integrity and latency but there is no targeted design for the PDC for the application scenario of the distribution network. Reference [11] proposed a software PDC that used a reliable communication protocol to establish a database system for efficient data storage and access. It is used as a low-cost alternative to large PDCs, while its application scenario is still in the transmission grid. Reference [12] discussed the design of PDC and the handling of anomalies. The anomalies handling is very meaningful, because DPDC may encounter these problems more frequently in the distribution network, but the article does not elaborate on the details of the anomalies handling scheme. Reference [13] proposed a software PDC for monitoring DER in a smart microgrid, which uses an adaptive compensation scheme to achieve an effective estimation of missing data elements, and proposed a monitoring unit to detect DER. Reference [14] considered the situation that more and more PDCs and PMUs were being installed in the distribution network, and proposed an active PDC with advanced functions to manage the delay according to the requirements of possible target applications. Reference [15] proposed a low-cost active PDC suitable for power distribution networks. The prototype of this PDC is based on the low-cost hardware platform SBC Raspberry Pi. Controlling costs is very important for the promotion of PDCs in the distribution network. At the same time, in order to meet the potential for increased equipment access, resources of a PDC should not be too limited. Reference [16] introduced an FNET/GridEye example of WAMS deployed in the distribution network. According to reports, the frequency monitoring network FNET/GridEye was originally developed in 2003 and was the first WAMS designed for distribution networks. Its sensors have the advantages of low cost and easy installation. Since FNET/GridEye relies on public networks for communication, its latency and other communication problems are more serious than WAMS for transmission networks. The existing PDC solution is mainly for connecting to the PMUs in the transmission network WAMS. Although there are some discussions on PDCs suitable for distribution networks, there is a lack of consideration of the working environment and communication environment of µMPMU and DPDC in distribution networks and the lack of analysis of DPDC application requirements. Various software and hardware PDCs have been introduced in the above solutions but they may face problems such as fewer application scenarios, single functions, higher costs, and limited equipment platform Appl. Sci. 2020, 10, 2942 3 of 18 resources. To better implement the promotion of µMPMU in the distribution network, discussions on DPDC design and testing. The main contributions of this paper are: Explain the µMPMU-DPDC networking structure, analyze the application requirements of DPDC, • and give the key performance indicators of DPDC; A hardware DPDC design method is proposed. Both software design and hardware selection take • into account the need to meet the key performance indicators of DPDC; Proposed a DPDC key performance indicators test method and verified the performance of the • designed DPDC through testing. The structure of this paper is as follows: Section2 introduces the networking structure of µMPMU-DPDC, analyzes the application requirements of DPDC, and provides key performance indicators of DPDC; Section3 describes the design of DPDC, including software structure and hardware selection, and discloses software implementation details; Section4 introduces the DPDC test environment and test tools, evaluates the DPDC’s key performance indicators; Section5 describes the DPDC’s field installation and operation; Section6 summarizes the article.

2. DPDC Application Requirements Analysis

2.1. µMPMU-DPDC Networking Structures The PMUs in the transmission network WAMS are mainly installed in important substations and power plants. In actual engineering, the PDC is generally installed in the same screen cabinet as the PMU. Each device gets timing information from a unified synchronous clock source. PMUs can be directly connected to the PDC or connected to the PDC via a switch. If the PDC is not equipped, PMUs are directly connected to the switch in the plant. The WAMS main station connects to the power dispatch data network through the pre-communication subsystem and receives data from PDCs or PMUs. It can be seen that the working environment of the PDCs and the PMUs in the transmission network WAMS is not bad, the communication connection between the devices is relatively stable, and PMU’s clock signal source is also reliable. Limited by the number of important components monitored in the plant, the number of PMUs connected to the PDC in the station will not be much. In contrast to the PDCs and PMUs in the above-mentioned transmission network WAMS, µMPMUs are installed in the substation, as well as the feeder, ring network cabinet, DER and other points [3]. The µMPMU outside the substation obtains the time signal through the external GPS antenna of the device. In addition to fiber optic Ethernet and Ethernet passive optical network (EPON), the communication method between the µMPMUs and the DPDCs or the main station also uses communication methods with a higher delay such as power line communication (PLC) or a wireless private network. Considering the installation location and communication mode of the µMPMUs and DPDCs, the networking modes of the µMPMU-DPDC can be divided into three types, as shown in Figure1. Figure1a shows a centralized networking structure in which all the µMPMUs and the DPDC are located in the station. This structure is consistent with the PMU-PDC networking structures in the transmission network. Figure1b1,b2 indicates that the µMPMUs are dispersedly installed on the ring main unit and each section of the feeder. The DPDCs are installed inside the station, and they communicate with µMPMUs via wired communication such as EPON or PLC. Figure1c indicates that the µMPMUs are dispersedly installed at feeder line and DER connection points. The µMPMU communicates with the upper device through a built-in wireless communication module or an external customer premise equipment (CPE). The DPDC or the main station is connected to the core network to receive data from the µMPMU. In this case, the DPDC can be placed in a substation, core network or inside of a main station. If only a few scattered µMPMUs use wireless communication, such µMPMUs can communicate directly with the main station without going through the DPDC. This figure does not mean that the µMPMU monitoring DER must use wireless communication. Under the conditions of proper installation cost, µMPMU should choose low-latency fiber-optic communication. Appl. Sci. 2020, 10, 2942 4 of 18 Appl. Sci. 2020, 10, x FOR PEER REVIEW 4 of 18

GPS

Main station

Backbone network

a b1 b2

DPDC DPDC DPDC OLT PLC

POS ONU μMPMU μMPMU μMPMU μMPMU PLC c POS ONU μMPMU 4G LTE/5G μMPMU μMPMU CPE PLC POS ONU μMPMU Base Station μMPMU μMPMU CPE

FigureFigure 1. µ MPMU-DPDC1. μMPMU-DPDC networking networking structures. structures. (a) centralized (a) centralized in the in station; the station; (b1,b2 )(b1,b2 wired) distributed;wired (c) wireless distributed. distributed; (c) wireless distributed. 2.2. Application Requirements 2.2. Application Requirements In addition to the different networking structures compared with the PMU-PDC in the transmission In addition to the different networking structures compared with the PMU-PDC in the network, the µMPMU connected to the DPDC is different from the traditional PMU in terms of function, transmission network, the μMPMU connected to the DPDC is different from the traditional PMU in quantity, and communication method [3]. To meet the application requirements in the active distribution terms of function, quantity, and communication method [3]. To meet the application requirements in network, the µMPMUs have the functions of power distribution terminals, and the corresponding the active distribution network, the μMPMUs have the functions of power distribution terminals, DPDCs also need to have certain functions of the distribution station. The DPDC can be used for data and the corresponding DPDCs also need to have certain functions of the distribution station. The collection, and the collected µMPMUs data is sent to the main station, and the control commands DPDC can be used for data collection, and the collected μMPMUs data is sent to the main station, from the main station are forwarded to the µMPMU. The DPDC can also directly have the monitoring and the control commands from the main station are forwarded to the μMPMU. The DPDC can also function. By analyzing the data of µMPMUs, the DPDC controls µMPMUs to complete fault location, directly have the monitoring function. By analyzing the data of μMPMUs, the DPDC controls isolation, restore power to non-faulty areas, the DPDC then reports the processing results to the main μMPMUs to complete fault location, isolation, restore power to non-faulty areas, the DPDC then station. Thereby the feeder automation is implemented. reports the processing results to the main station. Thereby the feeder automation is implemented. To make the problem that DPDC needs to solve more clear, PMU and µMPMU, PDC and DPDC To make the problem that DPDC needs to solve more clear, PMU and μMPMU, PDC and DPDC are compared from various aspects, and the comparison contents are shown in Table1. are compared from various aspects, and the comparison contents are shown in Table 1. Through the comparison in the table, the following conclusions can be drawn regarding the Through the comparison in the table, the following conclusions can be drawn regarding the application requirements of DPDC: application requirements of DPDC: µ (1)(1) TheThe number number of of μMPMUsMPMUs is is large large and and the the number number of of points points may may increase increase as as the the number number of of importantimportant local local nodes nodes increases. increases. A A DPDC DPDC as as the the substation/node substation/node PDC PDC should should be be able able to to connect connect moremore than 2020µ μMPMUsMPMUs [17 [17],], and and the the dynamic dynamic data data files file cans be can stored be stored for 14 days,for 14 and days, the transientand the transientdata files data more file thans more 1000 than pieces. 1000 Commercial pieces. Commercial PDCs can PDCs connect can hundredsconnect hundreds of PMUs, of but PMUs, their butprices their are prices thousands are thousands of dollars of or dollars more, andor more, there isand no there price is advantage no price advantage in promoting in thempromoting in the themdistribution in the distribution network; network; (2)(2) TheThe installation installation locations locations of of μMPMUsµMPMUs are are scattered scattered and and the the working working environment environment is is diverse. diverse. ComparedCompared with with the the traditional traditional PMU, PMU, the the μMPMUµMPMU may may face face the the problem of communication interruptioninterruption and and clock clock loss loss [11]. [11]. The The DPDC DPDC needs needs to to be be able able to to correctly correctly assemble assemble the the data data and and notnot discard discard the the clock clock a abnormalbnormal data, data, and and also also has has the the function function of of reconnecting reconnecting the the μMPMUµMPMU;; (3)(3) TheThe communication communication modes modes adopted adopted by by μMPMUµMPMU are are various, various, and and the the delay delay of of different different communicationcommunication methods methods is is different. different. Wireless Wireless communication communication may may have have an an unstable unstable connection connection under different meteorological or spatial conditions, so in the case of using a synchronous time signal, the data frames of the same timestamp produced by different μMPMU may arrive at the Appl. Sci. 2020, 10, x; doi: FOR PEER REVIEW www.mdpi.com/journal/applsci Appl. Sci. 2020, 10, 2942 5 of 18

under different meteorological or spatial conditions, so in the case of using a synchronous time signal, the data frames of the same timestamp produced by different µMPMU may arrive at the DPDC at different times, after experiencing a significant delay. The DPDC should be able to tolerate a large arrival time difference of the same timestamp data between the µMPMUs, that is, to correctly collect data of different delays. Reference [18] summarized the delay of wireless communication, and reference [9] also introduced the delay of PMU data transmission using 4G LTE, at an average of 70 ms, and even 1 s in certain cases. This paper defines DPDC’s ability to tolerate data time differences as time difference tolerance, with a reference value of 150 ms; (4) Considering the versatility of the µMPMU, the main tasks performed by different µMPMUs may be different, and different applications have different data delay requirements, ranging from 100 ms to 5 s [19]. The classification and priority transmission of data are helpful for increasing DPDC ability and improving communication efficiency. Traditional phasor transmission such as protocol IEEE C37.118.2-2011 [20] and GB/T 26865.2-2011 [21] cannot meet the requirements, and DPDC needs to support the extended protocol. IEEE C37.118.2-2011 is an internationally popular synchronous phasor transmission protocol. GB/T 26865.2-2011 is a synchronous phasor transmission protocol used by China’s power system, and the contents are mostly the same. The table takes GB/T 26865.2-2011 as an example. The extended protocol is based on the GB/T 26865.2-2011 protocol, adding data priority, remote control commands, etc. The method is also applicable to IEEE C37.118.2-2011, more details can be seen in the paper [3]; (5) A large number of µMPMUs accessed by DPDC will send phasor data at high frequency and will also send a large amount of file data and commands. To avoid affecting the real-time monitoring and active control capabilities of the entire system, DPDC should be able to efficiently process, aggregate and forward data. This requires the DPDC to have a lower processing delay [8], where the DPDC processing delay is defined as the DPDC data delay, indicating the arrival time of the last complete data message with a given timestamp and its leaving time difference from the DPDC, in the case where the data aggregation is normal. DPDC data delay should be within 3–10 ms [17]; (6) Since the increases in the number of µMPMUs also leads to an increase in communication traffic, which further leads to an increase in storage footprint. A more directed way to reduce flow and storage space is via data compression. To avoid any data accuracy loss and data latency increase, it is beneficial to use a suitable compression method.

Table 1. Device comparison.

Device PMU µMPMU PDC DPDC Substation; power Substation; power distribution room; Substation; Substation; power Locations plant transformer; feeder power plant distribution room line; DER Aggregate and Synchronous Synchronous phasor Aggregate and forward data; Main Functions phasor measurement; forward data power distributing measurement distribution terminal substation Optica fiber; twisted Optical fiber; twisted Communication Optica fiber; pair; wireless private Optical fiber; pair; wireless private Modes Twisted pair network; power line twisted pair network; power line carrier carrier GB/T 26865.2-2011; Communication GB/T 26865.2-2011; GB/T GB/T 26865.2-2011; IEC/TR Protocols extended protocol 26865.2-2011 extended protocol 61850-90-2-2016 Appl. Sci. 2020, 10, 2942 6 of 18

Based on the above analysis, it can be concluded that DPDC needs to have expanded functions and features in order to cope with various situations. It summarizes the main problems faced by DPDC and their corresponding extended functions and features, as shown in Table2.

Table 2. DPDC extended functions and features.

No. Situation Faced DPDC Extended Functions and Features 1 Accessing more µMPMUs DPDC should have strong access capabilities and keep costs low The DPDC should be able to aggregate data frames that have the µMPMUs adopt different same timestamp but arrive at the DPDC with a large time 2 communication methods difference; when some devices are disconnected, the normally arriving data is not discarded In the communication protocol, power distribution automation µMPMUs expand distribution 3 functions such as remote control, telemetry, and fault diagnosis terminal functions should be added. Processing of large amounts of DPDC should improve data processing capabilities and keep 4 data of different kinds data delay low.

This paper proposes the key performance indicators of DPDC, which are also the most basic requirements for DPDC, namely access capability, time difference tolerance, and data delay. The performance of DPDC consists of software performance and hardware performance. The hardware performance depends on the CPU, network port, running memory and hard disk. The software performance is affected by the program frame and operation mechanism. By testing the key performance indicators of DPDC, the performance of DPDC is judged [22].

3. DPDC Design

3.1. Software Architecture In the software design of DPDC, to improve the high concurrency and asynchronous processing capability of the software, we adopt an event-driven mechanism and structured program design, run multiple threads to realize multi-tasking processing and use callbacks to complete asynchronous non-blocking operations. Figure2 is a schematic diagram of the proposed DPDC considering multi-communication mode, which shows some major software functional modules inside the DPDC. The figure simplifies the µMPMUs, the communication network, and the main station. O1 Indicates that the µMPMU communicates directly with the DPDC over a twisted pair. O2 Indicates that the µMPMU communicates with the DPDC over the power line carrier. O3 Indicates that the µMPMU communicates with the DPDC over the wireless network. O4 Indicates that the µMPMU communicates with the DPDC via EPON. O5 Indicates that the µMPMU communicates with the DPDC over possibly other communication networks. The communication network from the DPDC to the main station is simplified to a dotted line. The green port on the DPDC boundary indicates the network port. The DPDC has more network ports. First, the communication is initialized in the main function of the DPDC, including establishing a client to complete the connection with the µMPMU and establishing a server to listen for the connection from the main station. The open-source iPDC [23] establishes a thread for each connected PMU and the main station and has only one connection to each PMU or main station, and all types of data are transmitted over one connection. The difference is that DPDC only creates two child threads in addition to the main thread. In the main thread, DPDC establishes two TCP connections for each µMPMU and each main station, which serve as data pipes and command pipes. One sub-thread is responsible for establishing a TCP connection as a file pipe with each µMPMU, and the other is responsible for establishing a file pipe with the main station. This can avoid the impact of file transfer on real-time Appl. Sci. 2020, 10, 2942 7 of 18 data transmission, save the resources consumed by creating a large number of threads, reduce the time overhead of thread switching, and reduce the internal delay of DPDC. We use libuv to establish a TCP connection because libuv is a multi-platform support library that focuses on asynchronous I/O. It uses the high-concurrency asynchronous model provided by the operatingAppl. Sci. 20 system.20, 10, x FOR InLinux, PEER REVIEW epoll supports full-featured event loops [24]. Due to libuv’s event-loop7 of and 18 callback mechanisms, callback functions can be invoked when the event occurs. When the connection isconnection established, is theestablished, uv_read_start the uv_read_start function defined function by libuv defined will by automatically libuv will automatical read the dataly read from the the streamdata from and triggerthe stream the callback and trigger function. the callback Since the function. DPDC Since and each the DPDCµMPMU and have each dedicated μMPMU pipes, have thededicated data received pipes, the from data a pipereceived is the from corresponding a pipe is the µcorrespondingMPMU data. μMPMU Inside the data. DPDC Inside has the shown DPDC in Figurehas shown2, the bluein Figure lines indicate2, the blue data lines pipes indicate for transmitting data pipes data for transmitting frames, the red data lines frames, indicate the command red lines pipesindicate for transmittingcommand pipes command for transmitting frames (CMD) command and configuration frames (CMD) frames and configuration (CFG), and the frames orange (CFG) lines, and the orange lines indicate file pipes for transmitting file command frames and file frames. The indicate file pipes for transmitting file command frames and file frames. The data pipes and the data pipes and the command pipes will remain connected after the communication is started. The file command pipes will remain connected after the communication is started. The file pipes will only pipes will only be established when there is a file sending task and will be disconnected after the task be established when there is a file sending task and will be disconnected after the task is completed. is completed. In order to ensure the stability of the data pipes and command pipes between the DPDC In order to ensure the stability of the data pipes and command pipes between the DPDC and µMPMUs, and μMPMUs, the connection check module is used to check whether each connection is valid. Here the connection check module is used to check whether each connection is valid. Here we set a timer of we set a timer of 3 s to check the continuity of the connection. If it is determined that a connection has 3 s to check the continuity of the connection. If it is determined that a connection has been disconnected, been disconnected, the connection is re-established. After the μMPMU restarts or the communication the connection is re-established. After the µMPMU restarts or the communication link is temporarily link is temporarily faulty, etc., the communication can be resumed through the connection check faulty, etc., the communication can be resumed through the connection check module. module.

DPDC μMPMU1 ① D_data_pipe data U_data_pipe data processing module processing module Data ② PLC aggregation Transmissi on check μMPMU2 Connection Write the check RT files module Main station Read D_cmd_pipe data U_cmd_pipe data CPE processing module processing module Read ③ data D

D master ata ata check

module Command ata ata check Command module cmd μMPMU3 Read reply cmd ④ Write/read Write/read Read ONU Read the CFG files the CFG files file master file μMPMU4 D_file_pipe data U_file_pipe data processing module processing module File File reply ⑤ Command Read the μMPMUn Write the HD HD/RT files files

File storage module

CFG RT HD

FigureFigure 2. 2.DPDC DPDC architecturearchitecture consideringconsidering multiple communication methods methods..

WhenWhen thethe datadata fromfrom thethe µμMPMUMPMU entersenters thethe DPDCDPDC via the data pipe, the Read Read data data module module completescompletes the the reading reading of of the the data data in in response response to theto the event. event. Similarly, Similarly, the the Read Read cmd cmd module module reads reads data fromdata the from command the command pipe, and pipe, the and Read the file Read module file module reads data reads from data the from file pipe.the file All pipe. kinds All of kinds data will of bedata sent will to thebe sent data to check the data module check first, module and then first, checked and then and checked processed and by processed the data by check the moduledata check and sentmodule to the and corresponding sent to the corresponding pipe data processing pipe data module. processing The module. data check The data module check checks module if the checks data hasif the sticky data or has incomplete sticky orpackets. incomplete Sticky packets. packets Sticky mean packets that multiple mean that small multiple packets small are placedpackets in are the sameplaced TCP in packet.the same An TCP incomplete packet. An packet incomplete means packet that a largemeans packet that a islarge separated packet into is separated multiple into TCP packets.multiple The TCP worst packets. case The is thatworst a TCPcase packetis that a has TCP a completepacket has packet a complete and anpacket incomplete and an packet,incomplete and thepacket, remaining and the data remaining is placed data in the is subsequentplaced in the TCP subsequent packet for TCP transmission. packet for transmission. The data check The module data check module decomposes and combines the data in which the above situation occurs and sends the processed data to each pipe data processing module according to the type. In general, the PMU and μMPMU disable the Nagle algorithm of TCP in order to ensure the uniform transmission of data frames in time, so that data frames do not have sticky packets and incomplete packets. The data check module checks the data frames with very little time and does not need to process them. Because configuration frames and file frames are longer than data frames, they are most likely to have sticky

Appl. Sci. 2020, 10, x; doi: FOR PEER REVIEW www.mdpi.com/journal/applsci Appl. Sci. 2020, 10, 2942 8 of 18 decomposes and combines the data in which the above situation occurs and sends the processed data to each pipe data processing module according to the type. In general, the PMU and µMPMU disable the Nagle algorithm of TCP in order to ensure the uniform transmission of data frames in time, so that dataAppl. frames Sci. 2020 do, 10 not, x FOR have PEER sticky REVIEW packets and incomplete packets. The data check module checks the8 of 18 data frames with very little time and does not need to process them. Because configuration frames and fileand frames incomplete are longer packets, than data but frames, their requirements they are most for likely time to are have far sticky less andthan incomplete the data frame, packets, and but the theirprocessing requirements time is for acceptable. time are far less than the data frame, and the processing time is acceptable. AfterAfter the the data data check check module, module, the the data data frame frame is sentis sent to to the the data data processing processing module module of of the theµMPMU μMPMU datadata pipe pipe (D_data_pipe). (D_data_pipe). The The main main function function of of this this module module is is to to aggregate aggregate the the data data frames frames of of the the µMPMU.μMPMU. First, First, it is it judged is judged whether whether the datathe data frame frame is a compressed is a compressed frame, frame, and the and decompressed the decompressed data framedata is frame sent tois thesent data to the aggregation data aggregation stage for stage processing. for processing. If no decompression If no decompression is required, is required, the data the framedata directlyframe directly enters theenters data the aggregation data aggregation stage. Thestage. data The aggregation data aggregation stage produces stage produces DPDC dataDPDC framesdata frames in the standardin the standard format, format, which which are written are written to real-time to real-time (RT) (RT) data data files files (the (the file file format format refers refers toto [21 [21]) and]) and sent sent to to the the data data processing processing module module of of the the main main station station data data pipe pipe (U_data_pipe). (U_data_pipe). The The data data of ofµMPMU μMPMU and and DPDC DPDC are are stored stored by by using using structure structure arrays.arrays. Figure 33 showsshows the cache structure structure of of the theDPDC. DPDC.

Buffer head

pdcData[0] haspmu soc fracsec pmusData[0] pmusData[1] pmusData[2] pmusData[n]

pdcData[1] haspmu soc fracsec pmusData[0] pmusData[1] pmusData[2] pmusData[n] Time pdcData[2] haspmu soc fracsec pmusData[0] pmusData[1] pmusData[2] pmusData[n]

pdcData[max] haspmu soc fracsec pmusData[0] pmusData[1] pmusData[2] pmusData[n]

Buffer end FigureFigure 3. The3. The cache cache structure structure of of the the DPDC. DPDC.

TheThe structure structure array array pdcData[ pdcData[ ] ] is is used used asas thethe bubufferffer of the DPDC, DPDC, and and the the depth depth is is max max + 1.+ The1. Thevalue value of of max max will will affect affect the the time time difference difference tolerance ofof thethe DPDC.DPDC. InIn theory, theory, the the time time di ffdifferenceerence tolerancetolerance is proportionalis proportional to the to max.the max. This valueThis value is not is recommended not recommended to be set to to be a fixedset to value. a fixed It shouldvalue. It beshould set according be set according to the actual to the network actual environmentnetwork envir whereonment the where DPDC the is DPDC located. is Itlocated. is included It is included in the configurationin the configuration file sent by file the sent local by or the the local main or station the main and station is assigned and when is assigned the device when is the initialized. device is Theinitialized. data structure The data members structure have members integer haspmu, have integer long integerhaspmu, soc, long and integer fracsec, soc, and and array fracsec, pmusData[ and array ]. Similarly,pmusDa pmusData[ta[ ]. Similarly, ] is a pmusData[ structure array ] is a with structure a length array ofn with+ 1. a The length value of isn the+ 1. sameThe value as the is number the same of asµMPMUs. the number It is of used μMPMUs. to store It the is dataused framesto store of the the dataµMPMU. frames The of the order μMPMU. of the array The order elements of the is thearray sameelements as that is of the the sameµMPMU. as that The of haspmuthe μMPMU. indicates The the haspmu number indicates of data the frames number of the ofµ MPMUdata frames that thisof the pdcDataμMPMU array that element this pdcData has stored. array When element haspmu has stored.= n + 1,When this buhaspmuffer is full= n and+ 1, thethis databuffer aggregation is full and is the completed.data aggregation Proceeding is completed. to the next Proceed step, involveding to the forming next step, the involved collected form dataing into the a standardcollected data DPDC into dataa standard frame, and DPDC then data sending frame, it toand the then master send stationing it to and the write master the station file. The and soc write and the fracsec file. The represent soc and time,fracsec and theirrepresent source time, can and be local their time source or the canµ MPMUbe local time-synchronized time or the μMPMU data time frame.-synchronized Which source data toframe. choose Which depends source on whether to choose there depends is a time-synchronized on whether there data is a frame time-synchr in thisonized buffer. Ifdata an elementframe in ofthis pdcData[buffer. If ] hasan element not received of pdcData[ a time-synchronized ] has not received data a time frame-synchronized when it isabout data frame to be when sent to it theis about next to stage,be sent the to DPDC’s the next local stage, time the is DPDC assigned’s local to it.time When is assigned the cache to isit. initialized,When the cache haspmu, is initialized, soc, and fracsechaspmu, aresoc, all cleared,and fracsec and theare statusall cleared, word statand ofthe each status element word of stat pmusData[ of each ]element is set to of 0x8000, pmusData[ indicating ] is thatset to the0x data8000, is indicating not available that and the thedata rest is not of the available data is and cleared. the rest of the data is cleared. TheThe program program flow flow of of the the data data aggregation aggregation stage stage is is shown shown inin FigureFigure4 .4. TheThe problemproblem thatthat willwill be besolved solved during during the the data data aggregation aggregation stage stage is is also also the the abnormal abnormal communication communication situation situation that that may may be faced: (i) when the communication connection is normal, some or all of the μMPMUs are out of synchronization and receive their data frames; (ii) some μMPMUs are disconnected from the DPDC and cannot receive their data frames; (iii) some μMPMUs are disconnected, and some μMPMUs with normal communication connections are out of synchronization. In the data aggregation stage, the thread lock mechanism is used to prevent data in the DPDC buffer from being overwritten by other threads.

Appl. Sci. 2020, 10, x; doi: FOR PEER REVIEW www.mdpi.com/journal/applsci Appl. Sci. 2020, 10, 2942 9 of 18 be faced: (i) when the communication connection is normal, some or all of the µMPMUs are out of synchronization and receive their data frames; (ii) some µMPMUs are disconnected from the DPDC and cannot receive their data frames; (iii) some µMPMUs are disconnected, and some µMPMUs with normal communication connections are out of synchronization. In the data aggregation stage, the thread lock mechanism is used to prevent data in the DPDC buffer from being overwritten by otherAppl. Sci. threads. 2020, 10, x FOR PEER REVIEW 9 of 18

Get pmusData[i]

Get DPDC time

Y Judge whether the data N is time-synchronized F1 according to the stat F2

Y There is pdcData [x] There is pdcData[x] satisfying N with the same time of pdcData[x].haspmu>0 && pmusData [i] pdcData[x].pmusData[i]==0 F23

There is pdcData[x] with time 0 N F11 N Y and pdcData[x].pmusData[i]==0 F13 F21 Put pmusData[i] into There is N Put pmusData[i] into Y pdcData[j]; pdcData[x] with pdcData[p]; F22 pdcData[k].haspmu++ time 0 pdcData[p].haspmu++ Y F12 Put pmusData[i] into N pdcData[q]; N Put pmusData[i] into pdcData[p].haspmu==n pdcData[q].haspmu++ pdcData[j].haspmu==n pdcData[k]; assign pmusData[i] time to Y pdcData[k]; N Y N pdcData[k].haspmu==n pdcData[j].haspmu++ The time of pdcData[p] is 0 Process pdcData[j] Y Y pdcData[k].haspmu==n Assign DPDC time N Assign DPDC time to pdcData[q] to pdcData[p] Y Process pdcData[k] Process pdcData[q] Process pdcData[p]

Process pdcData[m] with the earliest time Assign the PDC time to the topmost pdcData[r]; Process pdcData[r] Put pmusData[i] into pdcData[m]; pdcData[m].haspmu++ Put pmusData[i] into pdcData[m]; pdcData[m].haspmu++

return

FigureFigure 4.4. DataData aggregationaggregation programprogram flow.flow.

InIn FigureFigure4 ,4 the, the data data frame frame of of the the i-th i-thµ μMPMUMPMU (0 (0 ≤ ii ≤ n) is firstfirst acquired, storedstored inin pmusData[i],pmusData[i], ≤ ≤ andand thenthen thethe locallocaltime timeis isacquired. acquired. AccordingAccording toto IEEEIEEE C37.118.2-2011C37.118.2-2011 andand GBGB/T/T 26865.2-2011,26865.2-2011, inin thethe stat,stat, bitbit 1313= = 00 meansmeans datadata frameframe timetime synchronization,synchronization, soso itit isis judgedjudged accordingaccording toto statstat whetherwhether thethe datadata frameframe timetime isis synchronized.synchronized. IfIf thethe datadata frameframe timetime isis synchronized,synchronized, thethe processprocess flowflow F1F1 willwill bebe entered,entered, otherwise,otherwise, the the process process flow flow F2 F2 will will be be entered. entered.

(1)(1) TheThe firstfirst stepstep ofof F1F1 isis toto findfind thethe pdcData[pdcData[] ]element elementwith withthe the same same time time as as pmusData[i]. pmusData[i]. IfIf thethe elementelement isis found,found, itit isis assumedassumed toto bebe pdcData[j]pdcData[j] andand proceedproceed toto flowflow F11. F11. PutPut pmusData[i]pmusData[i] intointo thethe corresponding pdcData[j].pmusData[i] and and increase increase the the value value of of pdcData[j].haspmu pdcData[j].haspmu by by 1. 1.If Ifthe the value value of of pdcData[j].haspmu pdcData[j].haspmu is is equal equal to to n, n, it it indicates indicates that that pdcData[j] pdcData[j] has already aggregatedaggregated thethe datadata framesframes of of all allµ μMPMUs.MPMUs. Process Process pdcData[j] pdcData[j] means means that that the the data data contained contained in in pdcData[j] pdcData[j] is composedis composed into into a standard a standard format format DPDC DPDC data data frame, frame, and and then then pdcData[j] pdcData[j] is cleared, is cleared, and these and these data framesdata frames are all are written all written to the file to and the sent file and to the sent U_data_pipe to the U_data_pipe data processing data module. processing Because module. 50 Because 50 or 100 writes per second affect the life of the hard disk, the write operation does not have the same frequency as the framing. The program uses another large buffer to hold the data to be written and writes at regular intervals. (2) If pdcData[j] is not found, continue to look for elements with zero time. If an element that meets the criteria is found, the process flow F12 is performed, otherwise, the process flow F13 is performed. There may be two cases when the time is 0. The first is that the element has not yet obtained the data frame of the μMPMU, and the second is that the element has not yet obtained the time-synchronized data frame. In flow F12, assume that pdcData[k] is the element that meets the criteria and is closest to the cache header, store pmusData[i] into pdcData[k], and assign the

Appl. Sci. 2020, 10, x; doi: FOR PEER REVIEW www.mdpi.com/journal/applsci Appl. Sci. 2020, 10, 2942 10 of 18

or 100 writes per second affect the life of the hard disk, the write operation does not have the same frequency as the framing. The program uses another large buffer to hold the data to be written and writes at regular intervals. (2) If pdcData[j] is not found, continue to look for elements with zero time. If an element that meets the criteria is found, the process flow F12 is performed, otherwise, the process flow F13 is performed. There may be two cases when the time is 0. The first is that the element has not yet obtained the data frame of the µMPMU, and the second is that the element has not yet obtained the time-synchronized data frame. In flow F12, assume that pdcData[k] is the element that meets the criteria and is closest to the cache header, store pmusData[i] into pdcData[k], and assign the synchronization time to it, and then process and judge pdcData[k].haspmu. When there is µMPMU disconnected communication, it will enter flow F13 because pmusData[i] with synchronization time cannot find the elements of the same time and elements without time in the whole buffer, indicating that all elements have data at this time. But they have not been sent yet, they are waiting for some µMPMUs data that has not been received.. In order to control the delay of the data and avoid the overflow of the buffer, process the element with the earliest time, assuming it is pdcData[m], then pmusData[i] can be placed in the cache normally. Because pdcData[m] has not collected all the data, the empty pmusData[ ] element will keep the original value, except stat is 0x8000, the rest of the fields are all 0. According to the stat, the main station can know which µMPMU data is missing. (3) The first step in F2 is to look for the element that already has data but no i-th µMPMU data from the buffer header. If the eligible element is found, assume it is pdcData[p], ends the query, goes to flow F21, and puts pmusData[i] into pdcData[p]. If the pdcData[p] data has already aggregated all the data, it will be judged whether its time is 0 before processing. If it is 0, the local time obtained in the previous step is used. This is in order to deal with the extreme situation where all µMPMUs are out of sync. (4) If pdcData[p] is not found, continue to look for elements with time 0 and no i-th µMPMU data. If an element that meets the criteria is found, assume it is pdcData[q], go to flow F22, otherwise go to flow F23. Put pmusData[i] into pdcData[q]. If the pdcData[q] data has already aggregated all the data, assign the local time to pdcData[q] and process it. Similar to F13, F23 is also dealing with extreme situations. Some µMPMUs are disconnected, and some µMPMUs with normal communication connections are out of synchronization, the data will go to the F23 process. In order to control the delay of the data and avoid the overflow of the buffer, when the cache is full for the first time, the local time is assigned to the pdcData[m] at the top of the buffer, and then it is processed. Each time F23 is entered, the program will move down one element for processing.

The standard format DPDC data frame enters the U_data_pipe data processing module and is ready for transmission. Because the communication rate requirement of the main station to the DPDC may be different from that between the DPDC and the µMPMU, it is necessary to perform a transmission check on the data frame and select a data frame that satisfies the timestamp interval requirement for transmission. The U_data_pipe data processing module can realize the control of the data transmission rate and can also solve the problem that different main stations require different transmission rates when the DPDC communicates with the multi-main station. The µMPMU command pipe (D_cmd_pipe) data processing module is responsible for command interaction with the µMPMU to complete the transmission of commands and configuration information, and also write configuration frames to the CFG files. DPDCs with the extended protocol can also send remote commands. The data processing module of the µMPMU file pipe (D_file_pipe) is responsible for file command interaction with the µMPMU, sending file commands to the µMPMU, and then receiving and storing the transient data files, which may also be referred to as high dynamic (HD) (the file format refers to [25]). The data processing module of the main station command pipe (U_cmd_pipe) is responsible for command interaction with the main station and can read and write CFG files. The data processing module of the main station file pipe (U_file_pipe) is responsible for processing the file Appl. Sci. 2020, 10, 2942 11 of 18 command of the main station and transmitting the locally stored RT and HD files according to the requirements of the main station. When the DPDC does not have the file requested by the main station, DPDC will send a file command to each µMPMU to find it. If the corresponding file can be found, the file will be transparently transmitted, otherwise, the negative message will be replied to the main station. The file storage module is implemented by a shell script, and the RT files and the HD files are compressed and stored using the gzip compression program of the Linux system. When the disk

Appl.capacity Sci. 20 reaches20, 10, x FOR the PEER set upper REVIEW limit, the file storage module deletes the RT file stored for more11 thanof 18 14 days, and also deletes the old HD file and controls the number of HD files to 1000. In addition to the thefile file storage storage module, module, DPDC DPDC also also has has some some functional function modulesal modules not not shown shown in Figurein Figure2 implemented 2 implemented by byshell shell scripts, scripts, such such as watchdog,as watchdog, self-starting, self-starting, system system log log and and other other modules. modules.

3.2. Hardware Hardware P Platformlatform DPDC hardware hardware adopts platform platform hardware hardware architecture architecture design, with with high high-level-level electromagnetic electromagnetic compatibility (EMC) (EMC) protection, protection, its operating system adopts Linux real real-time-time multitasking operating system with independent cutting and packaging. The The DPDC DPDC front front and and r rearear panels panels are are shown in Figure 55.. TheThe DPDCDPDC modelmodel isis PDC-2018,PDC-2018, with a width of 1919 inchesinches andand aa heightheight ofof 22 unitsunits (U)(U) andand aa lengthlength of of 12 12 inches. inches. U U is is a a unit unit representing representing the the external external dimensions of the server, 11 U = 1.75 inches.

Equipment model VGA RS485 Power×2

LED RJ45×6

(a) (b)

Figure 5. DPDCDPDC panel. panel. ( (aa)) F Frontront panel; panel; ( (bb)) rear rear panel. panel.

InIn order order to to meet meet DPDC DPDC’s’s multi multi-tasking-tasking processing processing capabilities, capabilities, the device the device uses a uses dual a-core dual-core dual- threadeddual-threaded CPU, CPU,model model Intel(R) Intel(R) Celeron(R) Celeron(R) 2980 U, 2980 with U, 2 M with cache, 2 M clocked cache, clocked at 1.60 GHz. at 1.60 On GHz. the storage, On the DPDCstorage, needs DPDC to needsstore more to store than more 14 days than 14of daysRT files of RTand files more and than more 1000 than HD 1000 files. HD The files. device The deviceuses a 480uses GB a 480 solid GB-state solid-state drive drive to meet to meet the the requirements requirements of offast fast reading reading and and writing, writing, low low power consumption, shock shock resistance, resistance, an andd anti anti-fall.-fall. In In addition addition to to the the space space occupied occupied by by the the system and software, the the remaining remaining space space of of the the solid solid-state-state drive drive is is greater than 420 GB. The running memory typetype is is DDR3L DDR3L and and the the size size is is 4 4 GB, GB, which which can can meet meet the the need need to to set set a a large large buffer buffer when when DPDC DPDC a accessesccesses a large number of μMPMU.µMPMU. In order to meet the communication with with a a large large number number of of μMPMU,µMPMU, thethe DPDC DPDC is equipped with six Gigabit adaptive Ethernet ports. The The timing timing interface interface is is RS RS-485,-485, and InterRangeInterRange Instrumentation GroupGroup TimeTime Code Code Format Format B B (IRIG-B) (IRIG-B) is is used. used. The The DPDC DPDC is is connected connected to to a asynchronous synchronous clock clock source source through through this this interface interface and and receives receives aa synchronoussynchronous clockclock signal.signal. The DPDC supports dual dual power power supply supply operation operation and and can can be be powered powered by by direct direct current current or or alternating alternating current. current. Maintenance of the DPD DPDCC can be be remotely remotely logged logged in in through through the network network port, port, or or directly directly connected connected toto the the display display and and keyboard. keyboard. There There are are multiple multiple LED LED indicators indicators on the on thefront front panel panel of the of device. the device. The indicationThe indication information information includes includes the the status status of of the the system system operation, operation, alarms, alarms, tim timinging anomalies, anomalies, communication anomalies, and data link status.

4. The The Test Test of the Key Performance Indicators of DPDC InIn this study, equipment including μMPMU,µMPMU, time source, p powerower signal source, etc etc.,., and some software (e.g. (e.g.,, Main station Emulator and PMU Emu Emulator)lator) are are deployed to build the distribution network WAMS systemsystem forfor DPDCDPDC test.test. TheThe laboratorylaboratory testtest environment environment for for DPDC DPDC is is shown shown in in Figure Figure6. 6. The µMPMUs used in the test are produced by XUJI Electrics Company. The model is µM-PMU-851, whichThe can μMPMUs collect three-phase used in the voltages,test are produced three-phase by XUJI currents Electrics and Company. zero-sequence The model current is at μM the-PMU same- 851, which can collect three-phase voltages, three-phase currents and zero-sequence current at the same time and make μMPMUs achieve “three remote” function as the distribution network terminals. The time source is synchronous with GPS and provides time service to μMPMUs and DPDC, manufactured by Xu Ji Group, model DDS100-S, time accuracy ≤ 1μS. A relay protection tester works as a power signals source. It is manufactured by Guangdong onlly Electrical Automation Co., Ltd. and its model is ONLLY-AQ430. PC1 has two network ports, Intel(R) Xeon (R) E5-2603 v4 processor, 8 G RAM with 1.7 GHz working frequency. PC2 is a laptop with an Intel (R) Core (TM) i7 6700HQ processor, 4 G RAM with 3.5 GHz working frequency. PMU Emulator and Main station Emulator software are developed for DPDC testing. PMU Emulator is based on Visual C ++ development, supports the standard GB/T 26865.2-2011 protocol, and adopts a multi-thread design. The software can simulate 30 PMUs at the same time, each PMU has three output streams, the data Appl. Sci. 2020, 10, x; doi: FOR PEER REVIEW www.mdpi.com/journal/applsci Appl. Sci. 2020, 10, 2942 12 of 18 time and make µMPMUs achieve “three remote” function as the distribution network terminals. The time source is synchronous with GPS and provides time service to µMPMUs and DPDC, manufactured by Xu Ji Group, model DDS100-S, time accuracy 1µS. A relay protection tester works as a power ≤ signals source. It is manufactured by Guangdong onlly Electrical Automation Co., Ltd. and its model is ONLLY-AQ430. PC1 has two network ports, Intel(R) Xeon (R) E5-2603 v4 processor, 8 G RAM with 1.7 GHz working frequency. PC2 is a laptop with an Intel (R) Core (TM) i7 6700HQ processor, 4 G RAM with 3.5 GHz working frequency. PMU Emulator and Main station Emulator software are developed for DPDC testing. PMU Emulator is based on Visual C ++ development, supports the standard Appl. Sci. 2020, 10, x FOR PEER REVIEW 12 of 18 GB/T 26865.2-2011 protocol, and adopts a multi-thread design. The software can simulate 30 PMUs lengthat the samecan be time, mod eachified; PMUsupport has 1~200 three frames/s output streams,report rate. the The data data length time can of beeach modified; PMU is supportinitially generated1~200 frames by/ s the report computer rate. The time. data The time timestamp of each PMU can is be initially consistent generated or a by fixed the delay computer can time. be set The to simulatetimestamp the can time be consistentdifference orbetween a fixed different delay can PMUs be set arriving to simulate at the the PDC time through difference different between channels. different MainPMUs station arriving Emulator at the PDC is also through developed different based channels. on Visual Main C ++. station It can Emulator connect multiple is also developed PMU and based PDC deviceson Visual and C ++adopt. It a can multi connect-thread multiple design. PMU The simulated and PDC devicesmaster station and adopt displays a multi-thread the received design. data basedThe simulated on a table. master The packet station capture displays sof thetware received Wireshark data based is also on ainstalled table. The on packetthe DPDC. capture Its function software isWireshark to capture is alsonetwork installed packets on theand DPDC. display Its the function most detailed is to capture network network packet packets data as and possible, display with the timemost accurate detailed to network microseconds. packet data In the as following possible,with key indicator time accurate tests, to PDC microseconds. Emulator and In theMain following station Emulatorkey indicator are mainly tests, PDC used Emulator to test DPDC. and Main station Emulator are mainly used to test DPDC.

Relay protection tester

Main station Emulator Database device

μMPMU

DPDC Time source

Figure 6. DPDCDPDC test test environment environment..

4.1. Access Capacity 4.1. Access Capacity µ The number of connected μMPMUsMPMUs is is the the basic basic parameter parameter to to the the access access capacity capacity of of DPDC. DPDC. µ Meanwhile, thethe amountamount ofof data data sent sent by by MPMUμMPMU should should be be considered. considered. The The connection connection diagram diagram of this of thistest istest shown is shown in Figure in Figure7. The 7. twoThe networktwo network ports ports of the of PC the 1 arePC connected1 are connected to the to two the network two network ports portsof the of DPDC the DPDC via the via network the network cable cable 1 and 1 theand network the network cable cable 2, respectively. 2, respectively. The PMUThe PMU Emulator Emulator and andthe Mainthe Ma stationin station Emulator Emulator are run are on run the on PC1, the andPC1, the and simulated the simulated PMUs PMUs transmit transmit data to data the DPDCto the DPDCthrough through the network the network cable 1. cable After 1. the After DPDC the collects DPDC thecollects data, the the data, data isthe transmitted data is transmitted to the simulated to the simulatedmain station main through station the through network the cable network 2. cable 2.

PC 1 DPDC Cable 1 PMU Emulator

Main station Emulator Cable 2

Figure 7. DPDC access capability test.

The single line measurement configuration of the μMPMU is referenced in the test. Configure each simulated PMU data volume into 12 phasors (three-phase voltage, three-phase current, voltage and current symmetrical components), 20 analog quantities (active power P, reactive power Q, 3rd,

Appl. Sci. 2020, 10, x; doi: FOR PEER REVIEW www.mdpi.com/journal/applsci Appl. Sci. 2020, 10, x FOR PEER REVIEW 12 of 18 length can be modified; support 1~200 frames/s report rate. The data time of each PMU is initially generated by the computer time. The timestamp can be consistent or a fixed delay can be set to simulate the time difference between different PMUs arriving at the PDC through different channels. Main station Emulator is also developed based on Visual C ++. It can connect multiple PMU and PDC devices and adopt a multi-thread design. The simulated master station displays the received data based on a table. The packet capture software Wireshark is also installed on the DPDC. Its function is to capture network packets and display the most detailed network packet data as possible, with time accurate to microseconds. In the following key indicator tests, PDC Emulator and Main station Emulator are mainly used to test DPDC.

Relay protection tester

Main station Emulator Database device

μMPMU

DPDC Time source

Figure 6. DPDC test environment.

4.1. Access Capacity The number of connected μMPMUs is the basic parameter to the access capacity of DPDC. Meanwhile, the amount of data sent by μMPMU should be considered. The connection diagram of this test is shown in Figure 7. The two network ports of the PC 1 are connected to the two network ports of the DPDC via the network cable 1 and the network cable 2, respectively. The PMU Emulator and the Main station Emulator are run on the PC1, and the simulated PMUs transmit data to the DPDC through the network cable 1. After the DPDC collects the data, the data is transmitted to the Appl.simulated Sci. 2020 main, 10, 2942 station through the network cable 2. 13 of 18

PC 1 DPDC Cable 1 PMU Emulator

Main station Emulator Cable 2

FigureFigure 7.7. DPDCDPDC accessaccess capabilitycapability test.test.

Appl. Sci.TheThe 20 singlesingle20, 10, x line lineFOR measurementPEERmeasurement REVIEW configurationconfiguration ofof thethe µμMPMUMPMU isis referencedreferenced inin thethe test.test. ConfigureConfigure13 of 18 eacheach simulatedsimulated PMUPMU datadata volumevolume intointo 1212 phasorsphasors (three-phase(three-phase voltage,voltage, three-phasethree-phase current,current, voltagevoltage and5th,and current7thcurrent harmonic symmetrical symmetr voltageical components), components), and current) 20 and20 analog analog 4 digital quantities quantities quantities (active (active (external power power P,signalreactive P, reactive lights, power relays).power Q, 3rd, Q,A data3rd, 5th, 7thframe harmonic length voltageis 114 bytes. and current)Test at 10, and 25, 4 digital50, and quantities 100 frame/s (external reporting signal rates. lights, Each relays). test increments A data frame the lengthnumberAppl. Sci. is 20 114of20 , simulated10 bytes., x; doi: Test FOR PMUs, at PEER 10, 25,REVIEW observes 50, and whether100 frame the/s reporting data is correct rates. Each through testwww increments the.mdpi.com/journal/ simulated the number masterapplsci ofstation, simulated and monitors PMUs, observes the CPU whether and RAM the utilization data is correct of the through PDC process the simulated through master the top station, command and. monitorsThe test results the CPU are and shown RAM in utilization Table 3. of the PDC process through the top command. The test results are shown in Table3. Table 3. DPDC access capability test results. Table 3. DPDC access capability test results. CPU Utilization (%) at Different Reporting Rates (fps) Number of Connected PMUs RAM Utilization (%) CPU10 Utilization (%)25 at Different50 Reporting Rates100 (fps) Number of Connected PMUs RAM Utilization (%) 3 102 5 25 9 50 10018 0.3 5 3 7 15 29 0.3 3 2 5 9 18 0.3 510 36 15 7 22 15 50 29 0.30.5 1015 69 20 15 42 22 50 50 0.50.6 1520 11 9 26 20 50 42 50 50 0.60.8 20 11 26 50 50 0.8 As can be seen from Table 2, as the number of PMUs or the reporting rate increases, the CPU UtilizAsation can also be seenincreases, from Tableand the2, asRAM the numberUtilization of PMUsis only orrelated the reporting to the number rate increases, of PMUs. the In CPUterms Utilizationof CPU Utili alsozation, increases, the network and the RAMcard driver Utilization calls isthe only DMA related engine to the to number copy the of data PMUs. packet In terms to the of CPUkernel Utilization, buffer. After the the network copy is card successful, driver calls it initiates the DMA an engineinterrupt to copynotification the data interrupt packet to handler. the kernel As butheff er. number After andthe copy frequency is successful, of input it initiates streams an increase, interrupt message notification queues interrupt trigger handler. the CPU As more the numberfrequently and through frequency interrupts, of input so streams CPU Utili increase,zation will message rise. The queues DPDC trigger program the CPU’s internal more mechanism frequently throughalso affects interrupts, CPU Utili soz CPUation. Utilization In terms of will RAM rise. Utili Thezation, DPDC a program’sbuffer of a internal fixed depth mechanism has been also allocated affects CPUaccording Utilization. to the configuration In terms of RAM file Utilization,when the DPDC a buff waser of started. a fixed depthAfter PMUs has been are allocatedconnected, according the length to theof the configuration buffer is also file when determined the DPDC by was the started.data length After of PMUs each are PMU. connected, In the the absence length of of other the bu dataffer isprocessing also determined tasks, the by RAM the data Utili lengthzation of of each the PMU.DPDC Inprogram the absence does ofnot other change. data The processing data aggregation tasks, the RAMof DPDC Utilization in this oftest the is DPDCnormal, program and the does test result not change. shows The that data the aggregationDPDC meets of the DPDC access in thiscapability test is normal,requirement. and the test result shows that the DPDC meets the access capability requirement.

4.2.4.2. Time DiDifferencefference ToleranceTolerance TestTest EssentiallyEssentially speaking, speaking, the time didifferencefference tolerance tolerance is is equivalent equivalent to to the the multi-channel multi-channel access access capability.capability. InIn thethe distributiondistribution network,network, thethe µμMPMUsMPMUs connectedconnected toto the same DPDC may use didifferentfferent communicationcommunication mediamedia andand routes.routes. The didifferentfferent datadata packetspackets withwith thethe samesame timestamptimestamp wouldwould reachreach DPDCDPDC atat didifferentfferent times.times. The laboratory cannot directly reproducereproduce thethe realreal situationsituation ofof thethe powerpower lineline networknetwork andand thethe wirelesswireless network,network, soso wewe useuse thethe PMUPMU EmulatorEmulator forfor furtherfurther testing.testing. TheThe connectionconnection diagramdiagram of of this this test test is isshown shown in Figure in Figure8; the 8; DPDC the DPDC directly directly connects connects to the PC1 to the and PC1 PC2 and via a PC2 network via a cable.network The cable. Main The station Main Emulator station Emulator runs on the runs PC1, on andthe PC1, the PMU and the Emulator PMU Emulator runs on the run PC2.s on the PC2.

PMU Emulator Main station Emulator

PC 2 PC 1

FigureFigure 8. 8.DPDC DPDC time time didifferencefference tolerance test. test.

Eight PMUs were simulated on the PMU Emulator, and different delays for the simulated PMUs to test were set. The eight simulated PMUs are numbered sequentially and then divided into two groups. The odd-numbered PMU does not set the delay, and the even-numbered PMU sets the same delay to simulate the delay in the actual channel. After DPDC establishes communication connections with each simulated PMU, the simulated main station summons the DPDC data and observe whether the DPDC can correctly aggregate and forward PMUs data frames of different delays. Delays of 50, 100, 150, 200, and 300 ms were set respectively to test, and PMUs’ reporting rate was 100 fps. Table 4 shows the results.

Appl. Sci. 2020, 10, x; doi: FOR PEER REVIEW www.mdpi.com/journal/applsci Appl. Sci. 2020, 10, 2942 14 of 18

Eight PMUs were simulated on the PMU Emulator, and different delays for the simulated PMUs to test were set. The eight simulated PMUs are numbered sequentially and then divided into two groups. The odd-numbered PMU does not set the delay, and the even-numbered PMU sets the same delay to simulate the delay in the actual channel. After DPDC establishes communication connections with each simulated PMU, the simulated main station summons the DPDC data and observe whether the DPDC can correctly aggregate and forward PMUs data frames of different delays. Delays of 50, 100, 150, 200, and 300 ms were set respectively to test, and PMUs’ reporting rate was 100 fps. Table4 shows the results. The test results showed that when the time difference between the simulated PMU data is within 200 ms, the DPDC can correctly aggregate the data, and when the time difference reaches 300 ms, the DPDC aggregate data is abnormal. When data aggregation is abnormal, the DPDC aggregates data arrives at different times separately and outputs two data streams, one of which contains PMU data arriving earlier and the other containing PMU data arriving later. It can be observed from the Main station Emulator that the reporting rate of DPDC is increased to 200 fps (twice the normal value), and each data frame contains only half the number of PMU data after parsing. The cause of the abnormality is that the time difference between the PMU data exceeds the depth of the DPDC buffer. When the data that arrives earlier fill each layer of the buffer, the continued arrival of data prompted DPDC to push the data at the top of the buffer away, and the data that arrives later has no opportunity to aggregate with the data that has been pushed away. When the data aggregation is normal, the increase in PMU data delay is positively related to the delay of PMU data arriving later. When the data aggregation is abnormal, the increase in PMU data delay is the depth of the DPDC buffer. The reason why there is no change in CPU and RAM Utilization is the same as the analysis in Section 4.1. With a fixed number of PMUs and reporting rate, if there are no other data processing tasks, CPU and RAM Utilization will not change significantly. According to the test results, the time difference tolerance of DPDC reaches the requirement. When the time difference between µMPMUs connected through different channels is within 200 ms, DPDC can correctly aggregate data.

Table 4. DPDC time difference tolerance test results.

PMU Delay (ms) CPU Utilization (%) RAM Utilization (%) Aggregation 50 45 0.4 Normal 100 45 0.4 Normal 150 45 0.4 Normal 200 45 0.4 Normal 300 45 0.4 Abnormal

4.3. Data Delay Test Measuring the PDC data delay in the transmission network WAMS requires the network tester, but the network tester is expensive and complicated to operate. In this study, a simple way to measure the data delay of the DPDC is proposed. In order to ensure that the data transmission time and the data reception time have the same benchmark, the packet capture software Wireshark runs on the DPDC. Wireshark records the time when the last frame of the data frame of the same timestamp of each simulated PMU arrives at the DPDC and the time when the DPDC sends the data frame with the timestamp and calculates the difference between the two times to obtain the corresponding data delay. The test connection diagram can choose one of the above two schemes. During the test, the time differences over a period are averaged as the DPDC data delay. The number of PMUs is initially set to one and then set to two, followed by two increments each time. PMUs report rate is set to 100 fps. Figure9 shows the test results. Appl. Sci. 2020, 10, 2942 15 of 18 Appl. Sci. 2020, 10, x FOR PEER REVIEW 15 of 18 Appl. Sci. 2020, 10, x FOR PEER REVIEW 15 of 18

0.8 0.8

0.7 0.7

y

a 0.6 y l a

e 0.6 l d e

d a )

t s a a ) t 0.5 s d m a

( 0.5 d m

C ( C D P D P D 0.4

D 0.4

0.3 0.3

0.2 0.2 0 5 10 15 20 0 5 10 15 20 Number of PMUs Number of PMUs

Figure 9. DPDC data delay test. FigureFigure 9.9. DPDC data delay test.test. The test results show that the data delay of the designed DPDC increases with the number of The test resultsresults showshow thatthat thethe datadata delaydelay ofof thethe designeddesigned DPDCDPDC increasesincreases with the numbernumber of accessing PMUs. When 20 PMUs are connected, the data delay remains within 1 ms, which meets the accessing PMUs.PMUs. When 20 PMUs are connected, the data delay remains within 1 ms, which meets the test requirement. testtest requirement.requirement. 5. Actual Actual Installation Installation and and Operation Operation 5. Actual Installation and Operation The designed DPDC DPDC is is currently currently located located in in the the Lingang Lingang area area of of Pudong Pudong New New Area, Area, Shanghai, Shanghai, The designed DPDC is currently located in the Lingang area of Pudong New Area, Shanghai, China. The The area area has has a a full full-voltage-voltage distribution distribution network of 220 kV or less. The The distribution distribution network network is China. The area has a full-voltage distribution network of 220 kV or less. The distribution network is characterized by typical urban power grids. The The feeder feeder lines lines are are cable cable lines lines or or cable/overhead cable/overhead hybrid characterized by typical urban power grids. The feeder lines are cable lines or cable/overhead hybrid lines.lines. It It includes includes substations substations of of various various voltage voltage levels, levels, multiple multiple electric electric vehicles vehicles charging charging stations, stations, lines. It includes substations of various voltage levels, multiple electric vehicles charging stations, multiple photovoltaic power stations, and wind power plants. The The DPDCs DPDCs and and μMPMUsµMPMUs have been multiple photovoltaic power stations, and wind power plants. The DPDCs and μMPMUs have been installedinstalled in in some some 220 220 kV substations, 35 kV substations, 10 kV switchyards, and photovoltaic grid installed in some 220 kV substations, 35 kV substations, 10 kV switchyards, and photovoltaic grid points. The The distribution distribution network network WAMS WAMS constructed by the project has functions such as as distribution distribution points. The distribution network WAMS constructed by the project has functions such as distribution network state state estimation, estimation, fault fault diagnosis diagnosis and and location, location, and and island island control. control. Figure Figure 10 shows 10 shows some actual some network state estimation, fault diagnosis and location, and island control. Figure 10 shows some actualinstallation installation scenes. scenes. actual installation scenes.

μMPMUs μMPMUs

DPDC DPDC Switch Switch

(a) (b) (a) (b) Figure 10. Field installation. (a) The front layout of the cabinet; (b) a μMPMUµ mounted on the column. Figure 10. Field installation. ( a)) T Thehe front layout of the cabinet; ( b) a μMPMUMPMU mounted on the column. (a) Is the installation of equipment in a 35 kV substation, 7 μMPMUs and one DPDC are (a) Is the installation of equipment in a 35 kV substation, 7 μMPMUs and one DPDC are assembled in two cabinets. (b) Shows a μMPMU installed on the feeder, which is placed in a assembled in two cabinets. (b) Shows a μMPMU installed on the feeder, which is placed in a Appl. Sci. 2020, 10, x; doi: FOR PEER REVIEW www.mdpi.com/journal/applsci Appl. Sci. 2020, 10, x; doi: FOR PEER REVIEW www.mdpi.com/journal/applsci Appl. Sci. 2020, 10, 2942 16 of 18

(a) Is the installation of equipment in a 35 kV substation, 7 µMPMUs and one DPDC are assembled in two cabinets. (b) Shows a µMPMU installed on the feeder, which is placed in a dedicated box. The information of the line is measured by the voltage and current transformers and the data of the µMPMU is sent to the substation. The WAMS main station of the distribution network is deployed in the Shanghai Pudong Power Supply Company of the State Grid. The D5000-WAMS front-end machine developed by the NARI Group can view real-time data. The designed DPDC has been running for several months. During the time, DPDC works stably and the data transmission is normal. The project is still in progress and more DPDCs and µMPMUs will be deployed in the future.

6. Conclusions With the distributed energy, flexible load and electric vehicle charging station (pile) connected to the distribution network on a large scale, the monitoring and control methods of the traditional distribution network have difficulty coping with the new problems faced by the distribution network. Therefore, the synchronous phasor measurement technology in the transmission network WAMS is introduced into the distribution network, and the µMPMUs with the power distribution terminal functions are gradually deployed to each key node in the distribution network. The DPDC for the distribution network communication environment has become more important as a connection node between the µMPMUs and the main station. In view of the current lack of discussion on the design and application of the DPDC, this paper analyzes and compares the DPDC and the traditional PDC in terms of functions, communication methods, and application requirements. The key indicators for evaluating the performance of DPDC are proposed, and a DPDC is designed. The test environment is further designed to perform various evaluations on the DPDC, and the evaluation results are analyzed. The contribution of this paper is summarized below: This paper introduces three basic networking structures of µMPMU-DPDC and illustrates the • various communication methods that may exist in the networking. Then, by comparing the PMU and µMPMU, PDC and DPDC from the installation location and main functions, the application requirements of DPDC are obtained. Further to judge the basic performance of DPDC, the key performance indicators of DPDC are provided, namely, the access capability is no less than 20 µMPMU, the time difference tolerance is no less than 150 ms and the data delay is within 3 ms; A design method of hardware DPDC is proposed, which uses event-driven mechanism • and structured program design, uses libuv to establish TCP connection, and completes the asynchronous non-blocking operation for multi-task through the multi-thread and callback mechanisms. The functions of each software module inside DPDC are introduced, and the data buffer structure and data aggregation mechanism are described in detail. The data aggregation mechanism and data structure ensure the efficiency of data processing and reduce the demand for hardware resources. The hardware configuration and selection of the DPDC are also described; The test methods for key performance indicators of DPDC are proposed. Firstly, the development • of the test environment and the test software are described. Then test method for key performance indicators and conduct various tests on the DPDC was designed. The test results show that the designed DPDC access capability can reach 20 µMPMUs, the time difference tolerance is not less than 200 ms, and the data delay is less than 1 ms, which meets the requirements of all key performance indicators. Proven that the performance of the designed DPDC meets its application in the distribution network. Further analysis of the test results shows that the main impact on the performance of DPDC is the program mechanism and hardware performance. The designed DPDC uses a multi-threaded and event-driven mechanism. DPDC will call the corresponding callback function for processing when it receives the data. When the data aggregation is completed, it will be sent to the master station immediately to achieve a small delay and the CPU utilization rate is not high. If the time-driven mechanism is used to periodically query the status of each socket in the receiving thread, the DPDC data delay and CPU utilization rate will increase accordingly. Appl. Sci. 2020, 10, 2942 17 of 18

The field application of the designed DPDC works well. The further work of the author’s team is to expand more distribution sub-station functions and islanded control functions on the DPDC, enabling the DPDC to act as a distributed control sub-station for distribution automation and DER grid-connected control tasks.

Author Contributions: Conceptualization, W.T. and M.M.; methodology, W.T. and M.M.; software, validation, M.M., D.X. and Y.S.; writing—original draft preparation, M.M.; writing—review and editing, M.M., W.T. and M.D.; resources, W.X. and C.F. All authors have read and agreed to the published version of the manuscript. Funding: This research was funded by The National Key R & D Plan of China, grant number 2017YFB0902800; State Grid Corporation Science and Technology Project, grant number 52094017003D; the Science and Technology Research Project of Anhui Province, grant number 1704a0902004. Conflicts of Interest: The authors declare no conflict of interest.

References

1. Cruz, M.A.R.S.; Rocha, H.R.O.; Paiva, M.H.M. An algorithm for cost optimization of pmu and communication infrastructure in WAMS. Int. J. Electr. Power Energy Syst. 2019, 106, 96–104. [CrossRef] 2. Jamil, E.; Rihan, M.; Anees, M.A. Towards optimal placement of phasor measurement units for smart distribution systems. In Proceedings of the 2014 6th IEEE Power India International Conference (PIICON), Delhi, India, 5–7 December 2014; pp. 1–6. 3. Tao, W.; Ma, M.; Ding, M.; Xie, W.; Fang, C. A Priority-Based Synchronous Phasor Transmission Protocol Extension Method for the Active Distribution Network. Appl. Sci. 2019, 9, 2135. [CrossRef] 4. Von Meier, A.; Stewart, E.; Mceachern, A.; Andersen, M.; Mehrmanesh, L. Precision micro-synchrophasors for distribution systems: A summary of applications. IEEE Trans. Smart Grid 2017, 8, 2926–2936. [CrossRef] 5. Yang, X.; Zhang, X.P.; Zhou, S. Coordinated algorithms for distributed state estimation with synchronized phasor measurements. Appl. Energy 2012, 96, 253–260. [CrossRef] 6. De, L.R.J.; Centeno, V.; Thorp, J.S. Synchronized phasor measurement applications in power systems. IEEE Trans. Smart Grid 2010, 1, 20–27. 7. Aminifar, F.; Fotuhi-Firuzabad, M.; Safdarian, A.; Davoudi, A. Synchrophasor measurement technology in power systems: Panorama and state-of-the-art. IEEE Access 2014, 2, 1607–1628. [CrossRef] 8. IEEE Standard C37.244–2013. IEEE Guide for Phasor Data Concentrator Requirements for Power System Protection, Control, and Monitoring; IEEE: Piscataway, NJ, USA, 2013. 9. IEEE C37.247-2019. IEEE Standard for Phasor Data Concentrators for Power Systems; IEEE: Piscataway, NJ, USA, 2019. 10. Derviskadic, A.; Romano, P.; Pignati, M.; Paolone, M. Architecture and experimental validation of a low-latency phasor data concentrator. IEEE Trans. Smart Grid 2016, 8, 2885–2893. 11. Armenia, A.; Chow, J.H. A flexible phasor data concentrator design leveraging existing software technologies. Ieee Trans. Smart Grid 2010, 1, 73–81. [CrossRef] 12. Adamiak, M.G.; Kanabar, M.; Rodriquez, J.; Zadeh, M.D. Design and implementation of a synchrophasor data concentrator. In Proceedings of the 2011 IEEE PES Conference on Innovative Smart Grid—Middle East, Jeddah, Saudi Arabia, 17–20 December 2011; pp. 1–5. 13. Pourramezan, R.; Seyedi, Y.; Karimi, H.; Zhu, G.; Mont-Briant, M. Design of an advanced phasor data concentrator for monitoring of distributed energy resources in smart microgrids. Ieee Trans. Ind. Inform. 2017, 13, 3027–3036. [CrossRef] 14. Castello, P.; Muscas, C.; Pegoraro, P.A.; Sulis, S. Active Phasor Data Concentrator performing adaptive management of latency. Sustain. Energygrids Netw. 2018, 16, 270–277. [CrossRef] 15. Castello, P.; Muscas, C.; Pegoraro, P.A.; Sulis, S. Low-Cost Implementation of an Active Phasor Data Concentrator for Smart Grid. In Proceedings of the 2018 Workshop on Metrology for Industry 4.0 and IoT, Brescia, Italy, 16–18 April 2018; pp. 78–82. 16. Liu, Y.; Zhan, L.; Zhang, Y.; Markham, P.N.; Zhou, D.; Guo, J.; Lei, Y.; Kou, G.; Yao, W.; Chai, J.; et al. Wide-Area Measurement System Development at the Distribution Level: An FNET/GridEye Example. IEEE Trans. Power Delivery 2015, 31, 721–731. [CrossRef] Appl. Sci. 2020, 10, 2942 18 of 18

17. Kanabar, M.; Adamiak, M.G.; Rodrigues, J. Optimizing Wide Area Measurement System architectures with advancements in Phasor Data Concentrators (PDCs). In Proceedings of the 2013 IEEE Power & Energy Society General Meeting, Vancouver, BC, Canada, 21–25 July 2013; pp. 1–5. 18. Gore, R.N.; Valsan, S.P. Wireless communication technologies for smart grid (WAMS) deployment. In Proceedings of the 2018 IEEE International Conference on Industrial Technology (ICIT), Lyon, France, 20–22 February 2018; pp. 1326–1331. 19. Kansal, P.; Bose, A. Bandwidth and Latency Requirements for Smart Transmission Grid Applications. IEEE Trans. Smart Grid 2012, 3, 1344–1352. [CrossRef] 20. IEEE Standard C37.118-2011.2. IEEE Standard for Synchrophasors Data Transfer for Power System; IEEE: Piscataway, NJ, USA, 2011. 21. Field Test Asia Pte Ltd. GB/T26865.2-2011 Real-Time Dynamic Monitoring Systems of Power System-Part 2: Protocols for Data Transferring; Field Test Asia Pte Ltd.: Singapore, 2011. 22. Tao, W.; Zhu, X.; Fang, C.; Liu, J.S. Research on Main Indicators and Test Methods of Distribution Network Phasor Concentrator. Power Syst. Technol. 2019, 43, 801–809. 23. iPDC—Free Phasor Data Concentrator—CodePlex Archive. Available online: https://archive.codeplex.com/ ?p=ipdc (accessed on 4 January 2018). 24. Welcome to the Libuv Documentation. Available online: http://docs.libuv.org/en/v1.x/index.html# (accessed on 1 May 2018). 25. IEEE Standard C37.111-1999. IEEE Standard for Common Format for Transient Data Exchange (COMTRADE) for Power Systems; IEEE: Piscataway, NJ, USA, 1999.

© 2020 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).