Dynamic Performance Evaluation of Various GNSS Receivers and Positioning Modes with Only One Flight Test

Home , Differential GPS, GNSS Augmentation, Racelogic, RINEX

electronics

Technical Note Dynamic Performance Evaluation of Various GNSS Receivers and Positioning Modes with Only One Flight Test

Cheolsoon Lim 1, Hyojung Yoon 1, Am Cho 2, Chang-Sun Yoo 2 and Byungwoon Park 1,*

1 School of Aerospace Engineering, Sejong University, 209 Neungdong-ro, Gwangjin-gu, Seoul 05006, Korea; [email protected] (C.L.); [email protected] (H.Y.) 2 Future Aircraft Research Division, Korea Aerospace Research Institute, Daejeon 34133, Korea; [email protected] (A.C.); [email protected] (C.-S.Y.) * Correspondence: [email protected]; Tel.: +82-02-3408-4385

Received: 20 November 2019; Accepted: 7 December 2019; Published: 11 December 2019

Abstract: The performance of global navigation satellite system (GNSS) receivers in dynamic modes is mostly assessed using results obtained from independent maneuvering of vehicles along similar trajectories at different times due to limitations of receivers, payload, space, and power of moving vehicles. However, such assessments do not ensure valid evaluation because the same GNSS signal environment cannot be ensured in a different test session irrespective of how accurately it mimics the original session. In this study, we propose a valid methodology that can evaluate the dynamic performance of multiple GNSS receivers in various positioning modes with only one dynamic test. We used the record-and-replay function of RACELOGIC’s LabSat3 Wideband and developed a software that can log and re-broadcast Radio Technical Commission for Maritime Services (RTCM) messages for the augmented systems. A preliminary static test and a drone test were performed to verify proper operation of the system. The results show that the system could efficiently evaluate the performances of stand-alone, differential GNSS, and real time kinematics positioning for three GNSS receivers in two different positioning modes by repeatedly re-radiating the recorded signals acquired through only one flight. Our proposed system is expected to be useful in evaluating dynamic navigation performance accurately and conveniently in a valid manner.

Keywords: GNSS; GNSS augmentation system; GNSS signal recording and replaying; RTCM; GNSS performance evaluation

1. Introduction GNSS signals transmitted by medium Earth orbit (MEO) satellites [1] are easily affected by noise and errors during propagation through atmospheric layers [2]. In addition to errors and noise, the GNSS positioning performance completely depends on various factors, such as the positioning mode, receiver location, signal reception time, satellite geometry, and constellation [3]. Because of these vulnerabilities and dependency to signal environments, field testing with real signals before applying the high-accurate applications is very important to analyze whether their navigation and position will work as planned in the real world. The performance of GNSS is generally assessed through long-term static positioning error analysis for several hours or even days [4]. In such analyses, although all the signal environments are not strictly controlled, the effects of uncontrolled environments in a statistical sense are not so overwhelming that the long-term processing and analysis distort the original performance of GNSS receivers. Static test environments allow sufficient power, time, and space for the comparison of the performance of various

Electronics 2019, 8, 1518; doi:10.3390/electronics8121518 www.mdpi.com/journal/electronics Electronics 2019, 8, 1518 2 of 21

GNSS receivers simultaneously under the same conditions (e.g., satellite signal environment). Static tests also enable direct analysis of errors in a position domain because they are generally performed based on precisely surveyed known positions. On the other hand, performance assessments during dynamic tests are very difficult and non-standardized. Unlike the static test, the space and power are very limited in dynamic vehicles, and thus, simultaneous test sessions or exactly same trajectories cannot be guaranteed. In particular, the maximum operating time of an unmanned aerial vehicle (UAV) is very short (less than 30 min), and its space and payload are not sufficient to carry multiple GNSS receivers. Satellite visibility, ionospheric [5,6] and tropospheric effects, and noise levels generally change with time. Even if the time correlation of GNSS signals between multiple test sessions is high, different trajectories or attitudes might induce large differences in GNSS signal quality. Post-processing RINEX (receiver independent exchange format) data from various receivers can be one of the good options for evaluating the characteristics of receiver measurements in a dynamic environment [7,8]. This method is suitable for evaluating the measurement quality of each receiver and for analyzing the performance of known or newly developed algorithms, but it has limitations in evaluating the performance of commercial receivers whose detailed positioning algorithms are not disclosed. In order to analyze the GNSS navigation performances of dynamic systems such as UAV applications, it is very important to ensure validity of evaluation by providing the exact same dynamic environments for all the receivers or modes. To overcome the limitations of the dynamic test, several radio frequency (RF) record-and-replay approaches have been suggested and applied to a test driving [9,10]. This testing approach represents an effective and efficient method of testing stand-alone hardware receivers in a dynamic mode, but it cannot be applied for high-performance positioning modes, which require additional correction information to the GNSS RF signals. In this study, we propose a practical methodology that provides the same signal environments to different receivers and various augmented navigation modes to meet emerging and increasingly high expectations of users. For system implementation, we used RACELOGIC LabSat 3 Wideband [11], recording and replaying GNSS signals (e.g., GPS L1, L2, and L5) as suggested by previous research. In addition, our proposed “RTCM Logger and Broadcaster” software stores Radio Technical Commission for Maritime Services (RTCM) [12,13] messages and feeds them to multiple GNSS receivers to evaluate the differential GNSS (DGNSS) [14,15] or real-time kinematic (RTK) [16] performance as well as stand-alone GNSS performance. We also analyze whether pseudorange and carrier phase measurements from live and replayed signals are both equivalent, which can explain the repeatability of the position solutions. The remainder of this paper is structured as follows. In Section2, the suggested methodology and the developed system are introduced. Section3 shows the results of preliminary static tests to verify the GNSS performance evaluation system. The evaluation system was used for a drone flight test, and its performance assessments for multiple GNSS receivers and various positioning modes are described in Section4. The discussions and conclusions are given in Section5.

2. Method and System Implementation

2.1. GNSS Signal Recording and Replaying Because repeatable and controllable signals cannot be ensured for every test session in dynamic testing under live-sky conditions, the ability to exhaustively field-test products in real-world conditions is necessary for developers of location-aware chipsets, devices, and systems before launching them. When budgets and time are both under pressure, they inevitably select simulators, but simulators cannot anticipate the full richness and variety of real-world conditions. In this sense, the record-and-replay method, which is derived from software-defined radio (SDR) paradigm [17], is one of the easy and efficient means of characterizing new high-performing product performance in the real world by providing a combination of realism and repeatability, which is ideal for performance verification. Electronics 2019, 8, 1518 3 of 21

In this study, we used RACELOGIC’s LabSat 3 Wideband, which can record and replay GNSS signals. LabSat 3 supports multi-frequency GNSS signals, such as GPS L1/L2/L5, GLONASS G1/G2/G3, Galileo E1/E5/E6, BeiDou B1/B2/B3, QZSS L1/L2/L5, and SBAS L1/L5 anywhere and anytime without the need for connecting to a desktop or laptop computer [11]. This device is suitable for the purpose of our veriﬁcation methodology, i.e., providing the same signal environment to all GNSS receivers for which the performance is to be evaluated. The simulation or validation test is based on the replay functionality of similar devices, but most are for representing the resilience of their algorithms or devices to increasing GNSS vulnerabilities, such as jamming, spooﬁng [18–20], multipath [21], and interferences.

2.2. Software for Logging and Broadcasting RTCM Messages Most GNSS receivers support various types of positioning modes, such as precise point positioning (PPP) [22], differential GNSS (DGNSS) [14], satellite-based augmentation system (SBAS) based on wide-area DGPS (WADGPS) [23], real-time kinematic (RTK) [16], and network RTK [24–26] in addition to the stand-alone mode. Among these positioning modes, the stand-alone and SBAS modes do not require an additional communication channel for receiving correction data. Therefore, these modes are easy and simple to configure in UAVs, and recorded GNSS signals are easily provided using LabSat 3. For other modes, correction data generated from ground reference station facilities must be fed to the GNSS receivers through additional communication channels. Even though a pair of two high-end signal recorders enables the reproduction of the working process of an RTK base-rover system [27], GNSS RF signals should be logged at one as a rover and the other as a base station, and they should be synchronized to replay the RTK process. In particular, since various services that provide GNSS correction messages via the internet without any physical base station have recently become common, applications using the pair of two devices are inevitably becoming very limited. Therefore, it is not only complicated to configure a real-time system, but also impossible to perform the test using a single device. To solve this problem practically, we propose a method that enables a rover GNSS receiver to operate in DGNSS, RTK, or network RTK modes using the RF signal replayed at only one device. We developed a software that logs the correction data of the augmentation system during the recording of real-time GNSS RF signals. The software can also re-broadcast the logged corrections to the GNSS receivers when replaying the recorded signals. Using the software, it is possible to evaluate and compare the performances of various positioning modes for multiple GNSS receivers. The developed software consists of ‘RTCM Logger’ and ‘Broadcaster’ parts. The RTCM Logger writes RTCM version 2 and 3 messages in a file and the RTCM Broadcaster re-broadcasts the logged RTCM messages to each GNSS receiver. While LabSat 3 is recording GNSS RF signals, the RTCM Logger can receive RTCM messages through the Network Transport of RTCM via Internet Protocol (NTRIP) [28], Transmission Control Protocol/Internet Protocol (TCP/IP) or a serial port. Then, the received RTCM messages are converted and written in a format starting with a header ($RTCMV2 or $RTCMV3) to facilitate the distinguishing of the augmentation mode. Subsequently, the RTCM messages are written in the binary format with its coordinated universal time (UTC) [29], as shown in ElectronicsFigure1. 2019, 8, x FOR PEER REVIEW 4 of 22

Figure 1. Scheme of logging Radio Tec Technicalhnical Commission for Maritime Services (RTCM) messages (RTCM Logger).

Figure 2. Scheme of re-broadcasting RTCM messages (RTCM Broadcaster).

2.3. Performance Evaluation Method of Various GNSS Receivers Figure 3. illustrates the flow of the methodology for evaluating and analyzing the performance of GNSS receivers for a moving vehicle. First, the received live GNSS signals are recorded while a vehicle with LabSat 3 is moving along a specific trajectory. At the same time, RTCM messages for DGPS or RTK are also written by the RTCM Logger. Then, the recorded signals are re-radiated to GNSS receivers, and the RTCM message(s) corresponding to the UTC time for the replayed signal is/are fed to the receivers by the RTCM Broadcaster. Through these procedures, the trajectory of the drone flight for each positioning mode (e.g., stand-alone, DGPS, RTK, SBAS) of the receivers can be obtained. We processed receiver independent exchange format (RINEX) [31] files (one for the trajectory of the vehicle, and another from reference stations located near the rover) using GNSS post- processing software Trimble Business Center (TBC) [32] to obtain 5 mm accurate true trajectories of the moving vehicle. Finally, we evaluated the performance of the receivers by comparing the calculated trajectory from multiple receivers with the true trajectory obtained using TBC.

Electronics 2019, 8, x FOR PEER REVIEW 4 of 22

Electronics 2019, 8, 1518 4 of 21

During signal replay, the RTCM Broadcaster can obtain the UTC time of the replayed signal through the National Marine Electronics Association (NMEA) [30] GGA sentence from the GNSS rover receiver. The RTCM Broadcaster searches for the UTC time from the logged ﬁle, and then feeds the timely corresponding message(s) to the receivers, as shown in Figure2. The correction age of the NEMA GGA sentence represents the duration since the correction was fed to the receiver, which is typically 1 s for normal DGPS and RTK conditions. We can ensure that the correction has been Figure 1. Scheme of logging Radio Technical Commission for Maritime Services (RTCM) messages delivered to the receiver on time by monitoring the correction age, and this methodology also allows (RTCM Logger). us to simulate a long-latency case due to a bad connection to the correction server.

Figure 2. Scheme of re-broadcasting RTCM messages (RTCM Broadcaster). Figure 2. Scheme of re-broadcasting RTCM messages (RTCM Broadcaster). 2.3. Performance Evaluation Method of Various GNSS Receivers 2.3. PerformanceFigure3 illustrates Evaluation the Method flow of of the Various methodology GNSS Receivers for evaluating and analyzing the performance of GNSSFigure receivers 3. illustrates for a movingthe flow vehicle. of the methodology First, the received for evaluating live GNSS and signals analyzing are recorded the performance while a vehicleof GNSS with receivers LabSat for 3 isa moving alongvehicle. a specificFirst, the trajectory. received live At the GNSS same signals time, are RTCM recorded messages while for a DGPSvehicle or with RTK LabSat are also 3 is written moving by along the RTCM a specific Logger. trajectory. Then, theAt the recorded same time, signals RTCM are re-radiated messages for to GNSSDGPS receivers,or RTK are and also the written RTCM by message(s) the RTCM corresponding Logger. Then, to the the recorded UTC time signals for the are replayed re-radiated signal to isGNSS/are fed receivers, to the receivers and the byRTCM the RTCMmessage(s) Broadcaster. correspon Throughding to thesethe UTC procedures, time for thethe trajectoryreplayed ofsignal the droneis/are fed flight to forthe eachreceivers positioning by the RTCM mode (e.g.,Broadcaste stand-alone,r. Through DGPS, these RTK, procedures, SBAS) of thethe receiverstrajectory can of the be obtained.drone flight We for processed each positioning receiver independent mode (e.g., exchangestand-alone, format DGPS, (RINEX) RTK, [SBAS)31] files of (one the forreceivers the trajectory can be ofobtained. the vehicle, We andprocessed another receiver from reference independent stations exch locatedange near format the rover) (RINEX) using [31] GNSS files post-processing (one for the softwaretrajectory Trimble of the vehicle, Business and Center another (TBC) from [32 referenc] to obtaine stations 5 mm located accurate near true the trajectories rover) using of theGNSS moving post- vehicle.processing Finally, software we evaluated Trimble Business the performance Center (TBC) of the [3 receivers2] to obtain by comparing 5 mm accurate the calculated true trajectories trajectory of fromthe moving multiple vehicle. receivers Finally, with the we true evaluated trajectory the obtained performance using TBC. of the receivers by comparing the calculated trajectory from multiple receivers with the true trajectory obtained using TBC.

Electronics 2019, 8, 1518 5 of 21 Electronics 2019, 8, x FOR PEER REVIEW 5 of 22

Electronics 2019, 8, x FOR PEER REVIEW 5 of 22

FigureFigure 3. 3.Scheme Scheme of of feeding feeding RTCM RTCM messages messages (RTCM (RTCM Broadcaster). Broadcaster).

3. Verification of the Performance Evaluation System 3. Verification of the PerformanceFigure 3. Scheme Evaluation of feeding RTCM System messages (RTCM Broadcaster). 3.1. Static Test Configuration 3.1. 3.Static Verification Test Configuration of the Performance Evaluation System To verify the suitability of our developed system, we conducted static tests using LabSat 3, 3.1.To Static verify Test the Configuration suitability of our developed system, we conducted static tests using LabSat 3, and and the RTCM message logging/broadcasting software. First, we performed range-domain and the RTCM message logging/broadcasting software. First, we performed range-domain and positionposition-domainTo verify analyses the suitability to examine of our developed whether the system, use of we LabSat conducted 3 degraded static tests the using performance. LabSat 3, and For this domain analyses to examine whether the use of LabSat 3 degraded the performance. For this analysis, analysis,the RTCM we collected messageGNSS logging/broadcasting satellite signals software. received First, by we a GNSS performed antenna range-domain (NovAtel’s and GPS-703-GGG position- we collected GNSS satellite signals received by a GNSS antenna (NovAtel’s GPS-703-GGG Triple- Triple-Frequencydomain analyses Pinwheel to examine GNSS whether Antenna) the use installed of LabSat on 3 thedegraded roof of the the performance. Chung-moo For building this analysis, at Sejong Frequencywe collected Pinwheel GNSS GNSSsatellite Antenna) signals received installed by aon GNSS the antennaroof of (NovAtel’sthe Chung-moo GPS-703-GGG building Triple- at Sejong University in South Korea, as shown in Figure4. The signals were processed using NovAtel FlexPak6 UniversityFrequency in SouthPinwheel Korea, GNSS as shownAntenna) in installedFigure 4. on The the signals roof of were the Chung-mooprocessed using building NovAtel at Sejong FlexPak6 (high-end receiver), u-blox M8P (low-cost receiver), and LabSat 3, which are connected to a signal (high-endUniversity receiver), in South u-blox Korea, M8Pas shown (low-cost in Figure receiver 4. The ),signals and LabSatwere processed 3, which using are NovAtel connected FlexPak6 to a signal splitter. After recording the live signals on LabSat 3, the recorded signals were re-radiated to both splitter.(high-end After receiver), recording u-blox the liveM8P si(low-costgnals on receiver LabSat), 3,and the LabSat recorded 3, which signals are connected were re-radiated to a signal to both FlexPak6 and M8P receivers. We obtained the RINEX file for each receiver, for use in the range domain FlexPak6splitter. and After M8P recording receivers. the live We si obtainedgnals on LabSat the RINEX 3, the recordedfile for eachsignals receiver, were re-radiated for use toin boththe range analysis.FlexPak6 Position and resultsM8P receivers. were also We obtainedobtained fromthe RINEX both receivers,file for each as receiver, described for in use Section in the2 ,range and then domain analysis. Position results were also obtained from both receivers, as described in Section 2, performancedomain analysis. analysis Position in the position-domain results were also obtained was performed. from both receivers, as described in Section 2, andand then then performance performance analysis analysis in in thethe position-domain was was performed. performed.

FigureFigure 4. Test 4. Test conﬁguration configuration for for verifying verifying performanceperformance degradation degradation due due to the to the use useof LabSat of LabSat 3. 3.

Figure 4. Test configuration for verifying performance degradation due to the use of LabSat 3.

Electronics 2019, 8, x FOR PEER REVIEW 6 of 22

3.2. Measurement-Domain Test To compare the real time measurements with those from the recorded RF signals, we analyzed double difference (DD) measurements using code and phase measurements. Even in a static test, code (𝑃) and phase (𝛷) measurements obtained from both signals are mathematically different to each other due to their own receiver clock bias (𝐵 and cycle ambiguity (𝑁 terms, as described in Equations (1) and (2) [22]. 𝑃 𝜌 𝐵 𝑏 𝑇 𝐼 𝑀 𝜀 , , , , (1)

𝛷 𝜌 𝐵 𝑏 𝑇 𝐼 𝑀 𝑁 𝜆 𝜀 , , , , , (2) The subscript r denotes the source of signals (real time (RT) or LabSat 3 (LS)), and the subscripts s and i denote the s-th satellite and type of carrier frequency, respectively. In addition, 𝜌, 𝐵, 𝑏, 𝑇, 𝐼, 𝑀, 𝑁, 𝜆,and𝜀 denote the geometric range from satellite to ground receiver, receiver clock bias, satellite clock bias, tropospheric delay, ionospheric delay, multipath error, integer ambiguity, Electronicscarrier wavelength,2019, 8, 1518 and measurement noise, respectively. 6 of 21 As direct one-by-one comparisons of the measurements cannot be performed in a statistical manner (Figure 5) due to different 𝐵 and 𝑁, we calculated DD measurements for the s-th and t-th 3.2. Measurement-Domain Test satellites between RT and LS. DD (𝛥𝛻) between measurements of RT and LS can be expressed as follows.Tocompare the real time measurements with those from the recorded RF signals, we analyzed double difference (DD) measurements using code and phase measurements. Even in a static test, code 𝛥𝛻𝑃 𝛥𝛻𝜌 𝛥𝛻𝐵 𝛥𝛻𝑏 𝛥𝛻𝑇𝛥𝛻𝐼 𝛥𝛻𝑀 𝛥𝛻𝜀 (3) (P) and phase (Φ) measurements obtained from both signals are mathematically diff erent to each other due to their own receiver clock bias (B) and cycle ambiguity (N) terms, as described in Equations (1) 𝛥𝛻Φ 𝛥𝛻𝜌 𝛥𝛻𝐵 𝛥𝛻𝑏 𝛥𝛻𝑇𝛥𝛻𝐼 𝛥𝛻𝑀 𝛥𝛻𝑁𝜆 𝛥𝛻𝜀 (4) and (2) [22]. s s s s s In the above equations, the geometric range (𝜌), multipaths errors (M), and ionospheric (I) and Pr,i = ρr + Br b + Tr + Ir,i + MP + εP (1) tropospheric (T) delay are also canceled− out because bothr,i signalsr,i are from a zero-baseline s s s s s s s configuration. Receiver (𝐵Φ) and= ρsatelliter + Br (𝑏b + clockTr biasesI + M areΦs also+ N removedλ + εΦ bys the differences between(2) r,i − − r,i r,i r,i i r,i satellitesThe and subscript receivers,r denotes respectively. the source If the code of signals measurements (real time from (RT) RT orand LabSat LS are 3exactly (LS)), the and same, the only noise (𝛥𝛻𝜀 should remain in DD of the code measurements (𝛥𝛻𝑃 . In the same manner, the subscripts s andi denote the s-th satellite and type of carrier frequency, respectively. In addition, ρphase, B, b DD, T, (I𝛥𝛻Φ, M, N ,wouldλ, and notε denote contain the all geometric the terms range other from than satellite the ambiguity to ground (𝛥𝛻𝑁 receiver,𝜆 and receiver noise 𝛥𝛻𝜀 clock( bias, . satelliteAssuming clock that bias, any tropospheric cycle slip of delay,both measur ionosphericements delay, does multipath not occur error, in Equation integer ambiguity,(4), DD of 𝛥𝛻Φ carrierinteger wavelength,ambiguity (𝛥𝛻𝑁 and measurement can be estimated noise, and respectively. removed by the mean integer value of from 𝜆 EquationAs direct (4). Figure one-by-one 6a,b show comparisons the DD residual of the measurementserrors of carrier cannot phase measurements be performed inobtained a statistical from mannerreal-time (Figure and LabSat5) due 3 signals to di ff erentfor allB theand visibleN, we sa calculatedtellites, respectively. DD measurements The maximum for the and s-th root and mean t-th satellites between RT and LS. DD (∆ ) between measurements of RT and LS can be expressed as follows. square (RMS) values of DD residuals∇ of both receivers are less than 0.02 m, as shown in Table 1.

Figure 5. Direct comparison comparison of of GPS GPS L1 L1 pseudorange pseudorange meas measurementsurements from from real real time time (RT) (RT) and and LabSat LabSat 3 3(LS). (LS).

∆ Pi = ∆ ρ + ∆ B ∆ b + ∆ T + ∆ Ii + ∆ M∆ P + ∆ ε∆ P (3) ∇ ∇ ∇ − ∇ ∇ ∇ ∇ ∇ i ∇ ∇ i ∆ Φi = ∆ ρ + ∆ B ∆ b + ∆ T + ∆ Ii + ∆ M∆ Φ + ∆ Niλi + ∆ ε∆ Φ (4) ∇ ∇ ∇ − ∇ ∇ ∇ ∇ ∇ i ∇ ∇ ∇ i In the above equations, the geometric range (ρ), multipath error (M), and ionospheric (I) and tropospheric (T) delay are also canceled out because both signals are from a zero-baseline conﬁguration. Receiver (B) and satellite (b) clock biases are also removed by the diﬀerences between satellites and receivers, respectively. If the code measurements from RT and LS are exactly the same, only noise (∆ ε∆ P ) should remain in DD of the code measurements (∆ Pi). In the same manner, the phase DD ∇ ∇ i ∇ ( would not contain all the terms other than the ambiguity (∆ Niλi) and noise (∆ ε∆ Φ ). Assuming ∇ ∇ ∇ i that any cycle slip of both measurements does not occur in Equation (4), DD of integer ambiguity ∆ Φi (∆ Ni) can be estimated and removed by the mean integer value of ∇ from Equation (4). Figure6a,b ∇ λi show the DD residual errors of carrier phase measurements obtained from real-time and LabSat 3 signals for all the visible satellites, respectively. The maximum and root mean square (RMS) values of DD residuals of both receivers are less than 0.02 m, as shown in Table1. Electronics 2019, 8, 1518 7 of 21 Electronics 2019, 8, x FOR PEER REVIEW 7 of 22

(a) (b)

FigureFigure 6.6. DoubleDouble didifferenceﬀerence (DD)(DD) residualsresiduals ofof carrier carrier phase phase measurements; measurements; ( a(a)) NovAtel, NovAtel, ( b(b)) u-blox. u-blox.

Table 1. Root mean square (RMS) and maximum of double diﬀerence residuals of carrier Table 1. Root mean square (RMS) and maximum of double difference residuals of carrier phase phase measurements. measurements.

GNSSGNSS Receiver Receiver RMS RMS Max. Max. NovAtelNovAtel FlexPak6 FlexPak6 0.0011 0.0011 m m 0.0089 0.0089 m m u-blox M8P 0.0014 m 0.0157 m u-blox M8P 0.0014 m 0.0157 m

3.3.3.3. Position-DomainPosition-Domain TestTest WeWe also also examined examined whether whether performance performance degradation degradation occurred occurred due due to theto the use use of LabSat of LabSat 3 and 3 ourand developedour developed software software through through position position domain domain analysis. analysis. We conducted We conducted real-time real-time and semi-real-timeand semi-real- statictime testsstaticusing tests using the live the and live recorded and recorded signals. signals. As shown As shown in Figure in Figure7, 3 receivers 7, 3 receivers and LabSat and LabSat 3 were 3 connectedwere connected to the sameto the antenna same antenna to receive to thereceive same the live same satellite live signals. satellite RTCM signals. v2 RTCM and v3 v2 messages and v3 generatedmessages bygenerated Trimble by NetR9 Trimble were NetR9 respectively were respectively fed to M8P fed for to RTK M8P positioning for RTK positioning and EVK-6 and for DGPSEVK-6 positioning.for DGPS positioning. These messages These were messages also logged were by also our logged developed by our RTCM developed logger. AfterRTCM the logger. real-time After static the test,real-time we performed static test, the we semi-real-time performed the static semi-real-tim test by re-radiatinge static test the by recorded re-radiating signals the torecorded both receivers. signals Into addition,both receivers. for DGPS In addition, and RTK for positioning, DGPS and RTK RTCM positioning, v2 and v3 RTCM messages v2 and at the v3 timemessages corresponding at the time tocorresponding the UTC time to in the the UTC NMEA time GGA in the sentence NMEA obtained GGA se forntence each obtained receiver for were each fed receiver to each receiver,were fed as to showneachElectronics receiver, in Figure 2019 as,8 8 .shown, x Finally, FOR PEER in we FigureREVIEW obtained 8. Finally, the NMEA we obtained ﬁles (GGA) the NMEA from each files receiver. (GGA) from each8 ofreceiver. 22

FigureFigure 7. Real-time 7. Real-time static static test test conﬁguration configuration forfor verifyingverifying performance performance degradation degradation due dueto the to use the of use of our developedour developed software. software.

Figure 8. Semi-real-time static test configuration for verifying performance degradation due to the use of our developed software.

Figures 9 and 10 show the differences between east-north-up (ENU) errors of the real-time and semi-real time tests in DGPS and RTK modes. In case of the DGPS mode, there were discrepancies due to noise differences between real-time and recorded RF signals, but biases were not observed. Furthermore, the position fix type in the NMEA GGA sentence turned to 2 (DGPS mode) at the exact same time that the real-time process turned to the DGPS mode, and the fix type stayed at 2 for the remainder of the test. For the RTK case, although the discrepancies increased to about 0.6 m during the initial convergence period of RTK (float mode, fix type = 5), the RMS value of the discrepancies was only 0.001 m during the RTK fixed mode (fix type = 4). Except for the initial convergence period, the RTK fix type (Fixed or Float) was the same overall. These results imply that the use of recorded signals and our developed software does not lead to position performance degradation.

Electronics 2019, 8, x FOR PEER REVIEW 8 of 22

Figure 7. Real-time static test configuration for verifying performance degradation due to the use of our developed software. Electronics 2019, 8, 1518 8 of 21

FigureFigure 8.8. Semi-real-timeSemi-real-time static static test test conﬁguration configuration for for verifying verifying performance performance degradation degradation due due to the to usethe ofuse our of developedour developed software. software.

FiguresFigures9 9 and and 10 10 show show the the di differencesfferences betweenbetween east-north-upeast-north-up (ENU)(ENU) errorserrors ofof thethe real-timereal-time andand semi-realsemi-real timetime teststests inin DGPSDGPS andand RTKRTK modes.modes. InIn casecase ofof thethe DGPSDGPS mode,mode, therethere werewere discrepanciesdiscrepancies duedue toto noisenoise didifferencesfferences betweenbetween real-timereal-time andand recordedrecorded RFRF signals,signals, but biases were not observed. Furthermore,Furthermore, thethe positionposition fixfix typetype inin thethe NMEANMEA GGAGGA sentencesentence turnedturned toto 22 (DGPS(DGPS mode)mode) atat thethe exactexact samesame timetime thatthat thethe real-timereal-time processprocess turnedturned toto thethe DGPSDGPS mode,mode, andand thethe fixfix typetype stayedstayed atat 22 forfor thethe remainderremainder ofof the the test. test. For For the the RTK RTK case, case, although although the the discrepancies discrepancies increased increased to aboutto about 0.6 m0.6 during m during the initialthe initial convergence convergence period period of RTK of RTK (float (float mode, mode, fix type fix type= 5), = the 5), RMS the RMS value value of the of discrepancies the discrepancies was onlywas only 0.001 0.001 m during m during the RTK the RTK fixed fixed mode mode (fix type (fix type= 4). = Except 4). Except forthe for initialthe initial convergence convergence period, period, the RTKthe RTK fix type fix type (Fixed (Fixed or Float) or Float) was the was same the overall.same over Theseall. These results results imply thatimply the that use the of recordeduse of recorded signals Electronics 2019, 8, x FOR PEER REVIEW 9 of 22 andsignals our and developed our developed software software does not does lead not toposition lead to position performance performance degradation. degradation.

(a) (b)

FigureFigure 9. 9.( a()a) Discrepancies Discrepancies in in ENU ENU errors errors between between both both signals signals in thein the DGPS DGPS mode; mode; (b) NMEA(b) NMEA GGA GGA ﬁx typefix type in the in DGPSthe DGPS mode. mode.

(a) (b)

Figure 10. (a) Discrepancies in ENU errors between both signals in the RTK mode; (b) NMEA GGA fix type in the RTK mode.

4. Flight Test Performance Evaluation

4.1. Drone Flight Test Configurations For the dynamic test, we selected a drone capable of mounting the LabSat 3 antenna and configured a flight scenario for the GNSS signal acquisition. We used DJI’s Phantom 3 Professional. Table 2 shows the specifications of Phantom 3.

Electronics 2019, 8, x FOR PEER REVIEW 9 of 22

(a) (b)

Figure 9. (a) Discrepancies in ENU errors between both signals in the DGPS mode; (b) NMEA GGA Electronicsfix 2019type, 8in, 1518 the DGPS mode. 9 of 21

(a) (b)

FigureFigure 10. 10.(a ()a Discrepancies) Discrepancies in in ENU ENU errors errors between between both both signals signals in thein the RTK RTK mode; mode; (b) NMEA(b) NMEA GGA GGA ﬁx typefix type in the in RTK the RTK mode. mode.

4.4. Flight Flight Test Test Performance Performance Evaluation Evaluation 4.1. Drone Flight Test Configurations 4.1. Drone Flight Test Configurations For the dynamic test, we selected a drone capable of mounting the LabSat 3 antenna and configured For the dynamic test, we selected a drone capable of mounting the LabSat 3 antenna and a flight scenario for the GNSS signal acquisition. We used DJI’s Phantom 3 Professional. Table2 shows configured a flight scenario for the GNSS signal acquisition. We used DJI’s Phantom 3 Professional. the specifications of Phantom 3. Table 2 shows the specifications of Phantom 3.

Table 2. Speciﬁcations of DJI’s Phantom 3 Professional.

Diagonal Size 350 mm Weight 1.28 kg Max Ascent Speed 5 m/s Max Descent Speed 3 m/s Max Speed 57.6 km/h Max Flight Time Approx. 23 min Max Tilt Angle 35◦ Max Transmission 3.5–5 km Distance

We divided the drone’s total flight time of 10 min into 3 min of remaining stationary on the ground and 7 min of dynamic maneuvering. The drone flight scenario is as follows. First, the drone was allowed to remain stationary on the ground for 3 min after switching it on. Then the drone was flown along a square trajectory, as shown in Figure 11, after taking off. There are four flight path segments, and each segment includes 1 min of hovering at its waypoint and 30 s of maneuvering to another waypoint. We carried out the drone flight test on the rooftop of Gwanggaeto building in Sejong University on November 6, 2018, as shown in Figure 12. GNSS signals collected by the drone were recorded in LabSat 3, and RTCM messages generated by Trimble NetR9 (baseline length <1 km) were logged at the same time, as shown in Figure 13. The developed RTCM logger stored the real-time streams of RTCM v 2.3 message to feed the receivers Type (MT) 1 for differential corrections and MT 3 for reference station parameters. It also logged RTCM v3.1 messages of MT 1004 and 1012 with 1005 to re-broadcast L1 and L2 GPS and GLONASS RTK observables in order to replay GPS/GLONASS RTK. Electronics 2019, 8, x FOR PEER REVIEW 10 of 22

Table 2. Specifications of DJI’s Phantom 3 Professional.

Diagonal Size 350 mm Weight 1.28 kg Max Ascent Speed 5 m/s Max Descent Speed 3 m/s Max Speed 57.6 km/h Max Flight Time Approx. 23 min Max Tilt Angle 35° Max Transmission Distance 3.5–5 km

We divided the drone’s total flight time of 10 min into 3 min of remaining stationary on the ground and 7 min of dynamic maneuvering. The drone flight scenario is as follows. First, the drone was allowed to remain stationary on the ground for 3 min after switching it on. Then the drone was flown along a square trajectory, as shown in Figure 11, after taking off. There are four flight path segments,Electronics 2019 and, 8 each, 1518 segment includes 1 min of hovering at its waypoint and 30 s of maneuvering10 ofto 21 another waypoint.

Electronics 2019, 8, x FOR PEER REVIEW 11 of 22 Electronics 2019, 8, x FOR PEER REVIEW Figure 11. Drone ﬂight scenario. 11 of 22 Figure 11. Drone flight scenario.

We carried out the drone flight test on the rooftop of Gwanggaeto building in Sejong University on November 6, 2018, as shown in Figure 12. GNSS signals collected by the drone were recorded in LabSat 3, and RTCM messages generated by Trimble NetR9 (baseline length <1 km) were logged at the same time, as shown in Figure 13. The developed RTCM logger stored the real-time streams of RTCM v 2.3 message to feed the receivers Type (MT) 1 for differential corrections and MT 3 for reference station parameters. It also logged RTCM v3.1 messages of MT 1004 and 1012 with 1005 to re-broadcast L1 and L2 GPS and GLONASS RTK observables in order to replay GPS/GLONASS RTK. After recording the real time signals through the drone flight test, we performed a total of three re-radiation tests to evaluate the performance of the stand-alone, DGPS, and RTK modes. The recorded signals and RTCM messages were processed by three types of GNSS receivers: Novatel flexpak6, SwiftNav’s Piksi Multi, u-blox M8P, and u-blox EVK-6 (Figure 14). Flexpak6 and Piksi Multi receivers among the them support Beidou and Galileo as well as GPS and GLONASS, however DGPS and RTK of GPS and GLONASS have been analyzed because the correction messages for only GPS and GLONASS have been stored. Even though Phantom 3 does not provide enough payload and Figure 12. Drone flight test location. space to carry three receivers, the performancesFigureFigure 12. 12. DroneDrone of ﬂightflightall the test receivers location. location. with various positioning modes could be evaluated with only one flight test.

FigureFigure 13.13. Drone flight ﬂight test test configurations conﬁgurations (real-time). (real-time). Figure 13. Drone flight test configurations (real-time).

Electronics 2019, 8, 1518 11 of 21

After recording the real time signals through the drone flight test, we performed a total of three re-radiation tests to evaluate the performance of the stand-alone, DGPS, and RTK modes. The recorded signals and RTCM messages were processed by three types of GNSS receivers: Novatel flexpak6, SwiftNav’s Piksi Multi, u-blox M8P, and u-blox EVK-6 (Figure 14). Flexpak6 and Piksi Multi receivers among the them support Beidou and Galileo as well as GPS and GLONASS, however DGPS and RTK of GPS and GLONASS have been analyzed because the correction messages for only GPS and GLONASS have been stored. Even though Phantom 3 does not provide enough payload and space to carry three receivers, the performances of all the receivers with various positioning modes could be evaluatedElectronics with 2019, only8, x FOR one PEER flight REVIEW test. 12 of 22

FigureFigure 14. 14.Drone Drone ﬂight flight test test conﬁgurationsconfigurations (re-radiation).

4.2. Stand-Alone Mode In the first test (stand-alone mode), only the recorded signals were input to three receivers, and the4.2. performance Stand-Alone was Mode analyzed using NMEA files obtained from each receiver. Figures 15 and 16 show the estimatedIn the first trajectories test (stand-alone of the drone mode), and only ENU the errors recorded in the signals stand-alone were input mode, to three respectively. receivers, Table and 3 showsthe RMSperformance and maximum was analyzed values using of errors NMEA for eachfiles direction.obtained from Our each proposed receiver. system Figures allows 15 intuitiveand 16 interpretationshow the estimated and easy trajectories comparison of the of thedrone performance and ENU errors of the in three the stand-alone different receivers mode, respectively. with a single flightTable test. 3 shows RMS and maximum values of errors for each direction. Our proposed system allows intuitive interpretation and easy comparison of the performance of the three different receivers with a single flight test.Table 3. RMS and maximum of ENU errors in the stand-alone mode.

SwiftNav u-blox u-blox Piksi Multi EVK-6 M8P East 0.8947 m 1.0870 m 0.8400 m RMS North 0.7828 m 1.0204 m 0.7163 m Up 4.3090 m 1.6958 m 2.4837 m East 2.5083 m 4.4078 m 1.1333 m Max. North 2.6784 m 5.7743 m 1.6973 m Up 9.0660 m 5.3500 m 3.3260 m

Figure 15. Horizontal and vertical flight trajectories of the drone in the stand-alone mode.

Electronics 2019, 8, x FOR PEER REVIEW 12 of 22

Figure 14. Drone flight test configurations (re-radiation).

4.2. Stand-Alone Mode In the first test (stand-alone mode), only the recorded signals were input to three receivers, and the performance was analyzed using NMEA files obtained from each receiver. Figures 15 and 16 show the estimated trajectories of the drone and ENU errors in the stand-alone mode, respectively. Table 3 shows RMS and maximum values of errors for each direction. Our proposed system allows Electronicsintuitive2019 interpretation, 8, 1518 and easy comparison of the performance of the three different receivers12 with of 21 a single flight test.

Electronics 2019, 8, x FOR PEER REVIEW 13 of 22 Figure 15. Horizontal and vertical ﬂight trajectories of the drone in the stand-alone mode. Figure 15. Horizontal and vertical flight trajectories of the drone in the stand-alone mode.

Figure 16. ENU errors in the stand-alone mode. Figure 16. ENU errors in the stand-alone mode. 4.3. DGPS Mode Table 3. RMS and maximum of ENU errors in the stand-alone mode. Because only u-blox EVK-6 has the DGPS functionality among the three types of receivers, the recorded signals and RTCM v2 messages wereSwiftNav input to u-bloxu-blox EVK-6 u-blox for the second test (DGPS mode).

The performance was analyzed using the NMEAPiksi Multi ﬁle obtained EVK-6 from thisM8P receiver, referenced to the TBC true trajectory. Figures 17 and 18 Eastshow the 0.8947 estimated m trajectories 1.0870 m of 0.8400 the drone m and ENU errors in the DGPS mode, respectively.RMS Table 4North shows that 0.7828 the RMS m and 1.0204 maximum m 0.7163 values m of errors are about 1–2 m and 3–5 m for each direction. Up 4.3090 m 1.6958 m 2.4837 m East 2.5083 m 4.4078 m 1.1333 m Max. North 2.6784 m 5.7743 m 1.6973 m Up 9.0660 m 5.3500 m 3.3260 m

4.3. DGPS Mode Because only u-blox EVK-6 has the DGPS functionality among the three types of receivers, the recorded signals and RTCM v2 messages were input to u-blox EVK-6 for the second test (DGPS mode). The performance was analyzed using the NMEA file obtained from this receiver, referenced to the TBC true trajectory. Figures 17 and 18 show the estimated trajectories of the drone and ENU errors in the DGPS mode, respectively. Table 4 shows that the RMS and maximum values of errors are about 1–2 m and 3–5 m for each direction.

Electronics 2019, 8, x FOR PEER REVIEW 14 of 22

ElectronicsElectronics2019 2019,,8 8,, 1518 x FOR PEER REVIEW 1413 of 2122

Figure 17. Horizontal and vertical flight trajectories of the drone in the DGPS mode. Figure 17. Horizontal and vertical ﬂight trajectories of the drone in the DGPS mode. Figure 17. Horizontal and vertical flight trajectories of the drone in the DGPS mode.

Figure 18. ENU errors in DGPS mode. Figure 18. ENU errors in DGPS mode. Table 4. RMS and maximum of ENU errors in the DGPS mode. Table 4. RMSFigure and maximum 18. ENU errorsof ENU in errors DGPS in mode. the DGPS mode. u-blox u-blox Table 4. RMS and maximum of ENU errorsEVK-6 in the DGPS mode. EVK-6 Eastu-blox 0.8139 m EastNorth 0.8139 0.8039 m m RMS EVK-6 RMS NorthUp 0.8039 0.9259 m m East 0.8139 m UpEast 0.9259 4.8190 m m RMS North 0.8039 m Max. EastNorth 4.8190 5.2008 m m Max. NorthUpUp 5.20080.9259 3.7500 mm m EastUp 3.75004.8190 mm 4.4. RTK Mode Max. North 5.2008 m 4.4. RTK Mode Up 3.7500 m 4.4.1. Scenario I: RTK Performance of Different Receivers 4.4. RTK Mode 4.4.1.In Scenario the third I: RTK test (RTKPerformance mode), theof Different recorded Receivers signals and RTCM v3 messages were input to u-blox M8P4.4.1.In and Scenario the SwiftNav third I: testRTK Piksi (RTK Performance Multi, mode), and the of the recordedDifferent performance signReceiversals was and analyzed RTCM v3 using messages NMEA were files input obtained to u-blox from M8Pboth and receivers. SwiftNav Figures Piksi 19 Multi, and 20 and show the performanc the estimatede was trajectories analyzed of using the drone NMEA and files ENU obtained errors from with boththe RTK receivers.In the fix third type Figures intest the (RTK 19 RTK and mode), mode, 20 show the respectively. recorded the estimated sign Tableals trajectories5 andshows RTCM RMS of v3 the and messages drone maximum and were ENU values input errors ofto errorsu-blox with M8P and SwiftNav Piksi Multi, and the performance was analyzed using NMEA files obtained from both receivers. Figures 19 and 20 show the estimated trajectories of the drone and ENU errors with

Electronics 2019, 8, x FOR PEER REVIEW 15 of 22 Electronics 2019,, 8,, 1518x FOR PEER REVIEW 1514 of of 22 21 the RTK fix type in the RTK mode, respectively. Table 5 shows RMS and maximum values of errors thefor eachRTK directionfix type in as the well RTK as Time-to-first-fixmode, respectively. (TTFF) Table and 5 showsRTK fix RMS rate. and According maximum to these values results, of errors the forpositioning each direction performance as well ofas theTime-to-first-fix Time-to-first-fix two receivers (TTFF)in the RTK and RTKmode fixfix are rate.rate. similar According at the 1 to cm these level, results, and Piksi the positioningMulti using performancedual frequency of thesignals two is receivers superior in to the M8P RTK in modeTTFF areand similar RTK fix at rate. the 1 cm level, and Piksi Multi using dual frequency signals is superior to M8P in TTFF andand RTKRTK fixfix rate.rate.

Figure 19. Horizontal and vertical flight ﬂight trajec trajectoriestories of the drone in the RTK mode. Figure 19. Horizontal and vertical flight trajectories of the drone in the RTK mode.

Figure 20. ENU errors and RTK ﬁxfix type in RTK mode. Table 5. RMS andFigure maximum 20. ENU of ENU errors errors, and andRTK TTFF fix type and in RTK RTK ﬁx mode. rates in the RTK mode. Table 5. RMS and maximum of ENU errors, and TTFF and RTK fix rates in the RTK mode. Table 5. RMS and maximum of ENU errors, and SwiftNavTTFF and RTK fixu-blox rates in the RTK mode. SwiftNav u-blox Piksi Multi M8P PiksiSwiftNav Multi u-bloxM8P East 0.0046 m 0.0047 m RMS NorthEast Piksi 0.0046 0.0042 Multi m m 0.0047 0.0034M8P m m RMS NorthEastUp 0.00460.0042 0.0566 m m 0.00470.0034 0.0479 m m RMS NorthUpEast 0.0042 0.0566 0.0140 m m 0.0034 0.0479 0.0122 m m Max. NorthEastUp 0.05660.0140 0.0140 m m 0.04790.0122 0.0112 m m Max. NorthEastUp 0.0140 0.0900 m m 0.01220.0112 0.0730 m m RTKMax. Fix Rate NorthUp 0.0140 0.0900>99.99% m 0.0112 0.0730 98.35% m RTKTTFF Fix Rate Up 0.0900 >99.99% 10 sm 0.0730 98.35% 14 sm RTKTTFF Fix Rate >99.99% 10 s 98.35% 14 s TTFF 10 s 14 s

Electronics 2019, 8, x FOR PEER REVIEW 16 of 22 Electronics 2019, 8, 1518 15 of 21 4.4.2.Electronics Scenario 2019, 8 ,II: x FOR RTK PEER Performa REVIEWnce of Different Position Modes 16 of 22 4.4.2.The Scenario u-blox II: M8P RTK Performancereceiver used of for Diff theerent previous Position RTK Modes scenario supports different dynamic 4.4.2. Scenario II: RTK Performance of Different Position Modes platform modes, such as portable, automotive, and airborne modes, and it adjusts the navigation The u-blox M8P receiver used for the previous RTK scenario supports different dynamic platform filteringThe engine u-blox adaptively M8P receiver to the used expected for the applicat previousion environment. RTK scenario To supports investigate different the effect dynamic of the modes, such as portable, automotive, and airborne modes, and it adjusts the navigation filtering engine differentplatform position modes, modessuch as on portable, the RTK automotive,accuracy, we andcompared airborne the modes, positioning and resultsit adjusts of the the automotive navigation adaptively to the expected application environment. To investigate the effect of the different position andfiltering airborne engine modes adaptively using the to therecorded expected signal applicat and loggedion environment. RTCM message, To investigate although the the effect data ofwere the modes on the RTK accuracy, we compared the positioning results of the automotive and airborne acquireddifferent fromposition an airbornemodes on system. the RTK accuracy, we compared the positioning results of the automotive modes using the recorded signal and logged RTCM message, although the data were acquired from an and Evenairborne though modes both using positioning the recorded modes signal calculated and loggedRTK positions RTCM usingmessage, the samealthough GNSS the signals data were and airborne system. correctionacquired from messages an airborne obtained system. from the flight test, the wrong dynamic model of automotive mode Even though both positioning modes calculated RTK positions using the same GNSS signals and cannotEven provide though position both positioning results as accurately modes calculated as the right RTK model positions of airborne using the mode same as GNSSshown signals in Figures and correction messages obtained from the flight test, the wrong dynamic model of automotive mode 21correction and 22 and messages Table 6. obtained Since the from automotive the flight mode test, assumesthe wrong the dynamic vehicle dynamics model of toautomotive be plane-like, mode it cannot provide position results as accurately as the right model of airborne mode as shown in Figures 21 couldcannot not provide have solved position the results RTK fix as accuratelysolutions with as the the right errors model of several of airborne meters mode until as the shown drone in flew Figures up and 22 and Table6. Since the automotive mode assumes the vehicle dynamics to be plane-like, it to21 the and altitude 22 and ofTable 116 m,6. Since while the the automotive airborne mode mode acquired assumes its the RTK vehicle fix positions dynamics in tothe be middle plane-like, of the it could not have solved the RTK fix solutions with the errors of several meters until the drone flew up ascendingcould not havephase. solved the RTK fix solutions with the errors of several meters until the drone flew up to the altitude of 116 m, while the airborne mode acquired its RTK fix positions in the middle of the to the altitude of 116 m, while the airborne mode acquired its RTK fix positions in the middle of the ascending phase. ascending phase.

Figure 21. Horizontal and vertical flight trajectories of the drone in the automotive and airborne modes.Figure 21. Horizontal and vertical ﬂight trajectories of the drone in the automotive and airborne modes. Figure 21. Horizontal and vertical flight trajectories of the drone in the automotive and airborne modes.

FigureFigure 22. 22. ENUENU errors errors and and RTK RTK fix ﬁx type in th thee automotive and airborne modes.

Figure 22. ENU errors and RTK fix type in the automotive and airborne modes.

Electronics 2019, 8, 1518 16 of 21 Electronics 2019, 8, x FOR PEER REVIEW 17 of 22

Table 6.6. RMS and maximum of ENU errors, and TTFF and RTK ﬁxfix rates in thethe automotiveautomotive and airborne modes.

AutomotiveAutomotive Airborne Airborne EastEast 0.0052 0.0052 m m 0.0047 0.0047 m m RMSRMS NorthNorth 0.0042 0.0042 m m 0.0035 0.0035 m m Up 0.0458 m 0.0473 m Up 0.0458 m 0.0473 m EastEast 0.0153 0.0153 m m 0.0130 0.0130 m m Max. North 0.0132 m 0.0110 m Max. North 0.0132 m 0.0110 m Up 0.0780 m 0.0730 m Up 0.0780 m 0.0730 m RTK Fix Rate 93.32% 98.49% RTK Fix Rate 93.32% 98.49% TTFFTTFF 40 40 s s 13 13 s s

The results of this scenario show that the valid dynamic model enables the solution to follow the true trajectory, whichwhich impliesimplies thisthis methodmethod isis helpfulhelpful forfor evaluatingevaluating ifif thethe platformplatform oror model has been designed appropriately forfor eacheach kinematickinematic system.system.

4.4.3. Scenario III: RTK Performance of DiDifferfferentent ConstellationsConstellations withwith TroposphericTropospheric EEffectsffects This methodology can easily compare the RTK performance of GPS-only with that of GPSGPS/GLONASS/GLONASS using using the the RF RF signal signal with with the the correction correction dataset dataset acquired acquired from from the drone the drone flight flight test. test.Figure Figure 23 and 23 andTable Table 7 show7 show that that multi-constellat multi-constellationion of of GPS GPS and and GLONASS GLONASS contributes contributes toto reducingreducing the initializationinitialization time and improving thethe accuracyaccuracy ofof thethe RTKRTK fixedfixed solutions.solutions. The NovatelNovatel flexpak6flexpak6 receiver can also also be be configured configured to to apply apply the the tropos troposphericpheric correction correction to tothe the RTK RTK positioning. positioning. Since Since the thealtitude altitude difference difference between between the thereference reference stat stationion and and the the UAV’s UAV’s maneuver maneuver was was only only 3 3 m, the positioning results didifferencefference isis notnot noticeable whilewhile the tropospheric correction obviously shortened the initialization time for the ambiguity fix.fix. Howeve However,r, the proposed method makes it easy to see how the receiver’s RTK performanceperformance ofof thethe dynamicdynamic vehiclevehicle variesvaries accordingaccording toto eacheach configuration.configuration.

Figure 23. ENUENU errors errors and RTK fix ﬁx type of the mult multii GNSS and Tropospheric correctioncorrection test.test.

ElectronicsElectronics 20192019,, 88,, 1518x FOR PEER REVIEW 1817 of 2122

Table 7. RMS and maximum of ENU errors, and TTFF and RTK fix rates of the multi GNSS and TableTropospheric 7. RMS correction and maximum test. of ENU errors, and TTFF and RTK ﬁx rates of the multi GNSS and Tropospheric correction test. GPS Only GPS Only GPS+GLO GPS+GLO

(Tropo.GPS on) Only (Tropo.GPS off) Only (Tropo.GPS+GLO on) GPS(Tropo.+GLO off) East 0.0053(Tropo. m on) 0.0056(Tropo. m oﬀ) (Tropo. 0.0065 m on) (Tropo. 0.0065 oﬀ) m RMS North East 0.0040 0.0053m m 0.0040 0.0056 m m 0.0038 0.0065 m m 0.0065 0.0037 m m RMSUp North 0.0599 0.0040m m 0.0611 0.0040 m m 0.0594 0.0038 m m 0.0037 0.0581 m m Up 0.0599 m 0.0611 m 0.0594 m 0.0581 m East 0.0191 m 0.0135 m 0.0189 m 0.0139 m Max. North East 0.0116 0.0191m m 0.0123 0.0135 m m 0.0110 0.0189 m m 0.0139 0.0129 m m North 0.0116 m 0.0123 m 0.0110 m 0.0129 m Max.Up 0.0970 m 0.0890 m 0.1110 m 0.0850 m Up 0.0970 m 0.0890 m 0.1110 m 0.0850 m RTK Fix RTK Fix Rate 98.61% 98.61% 96.40% 96.40% 99.54% 99.54% 97.80% 97.80% Rate TTFF TTFF 13 s 13 s 32 s 32 s 5 5s s 20 s 20 s

4.4.4.4.4.4. Scenario IV: Performance ofof RTKRTK FedFed withwith OldOld CorrectionCorrection MessageMessage WeWe alsoalso simulated a correction correction latency latency case case by by intentionally intentionally allowing allowing the the RTCM RTCM Broadcaster Broadcaster to tofeed feed old old RTCM RTCM messages messages into into the the receiver. receiver. Five Five and and ten ten second second latencies latencies were were assigned for thisthis simulationsimulation andand thethe resultsresults werewere comparedcompared withwith thethe normalnormal RTK results,results, whosewhose correctioncorrection agesages werewere typicallytypically 1 or 2 s s old. old. Even Even though though the the ages ages of of co correctionrrection in in the the NMEA NMEA sentences sentences were were printed printed out out as as6 and 6 and 11 s 11 as swe as designed, we designed, the errors the errors were wereless than less 3 than mm 3in mmRMS in for RMS each for direction each direction once the once position the positionhad been had turned been to turned RTK fixed to RTK mode fixed as mode shown as in shown Figures in Figures24, 25, and 24– 26 andand TableTable8 8.. TheThe accuracy accuracy degradationdegradation ofof the the RTK RTK fixed fixed positions positions due due to the to latency the latency under anunder open-sky an open-sky were negligible, were negligible, however, thehowever, general the performance general performance during the during ambiguity the resolutionambiguity processresolution was process affected was by theaffected latency. by Thethe maximumlatency. The error maximum of the RTK error float of the solution RTK float fed by solution 10 s old fed correction by 10 s old message correction was message larger by was 3 cm larger than thatby 3 without cm than latency, that without and it tooklatency, 75 s and for the it took receiver 75 s with for the old receiver messages with to find old themessages solution to while find thethe generalsolution receiver while neededthe general only 14receiver s. Other needed ambiguity only fixing 14 s.performance, Other ambiguity for examples fixing performance, TTFF and overall for ambiguityexamples TTFF fixing and rate, overall were alsoambiguity degraded fixing due ra tote, the were latency. also degraded due to the latency.

FigureFigure 24.24. Horizontal andand verticalvertical ﬂightflight trajectoriestrajectories of of the the drone drone in in the the case case of of RTCM RTCM messages messages with with a delaya delay of of 5 and5 and 10 10 s. s.

Electronics 2019, 8, x FOR PEER REVIEW 19 of 22

Electronics 2019, 8, 1518 18 of 21 Electronics 2019, 8, x FOR PEER REVIEW 19 of 22

Figure 25. ENU errors and RTK fix type in the case of RTCM messages with a delay of 5 and 10 s.

Table 8. RMS and maximum of ENU errors, and TTFF and RTK fix rates in the case of RTCM messages with a delay of 5 and 10 s.

No Latency 5 s 10 s East 0.0049 m 0.0049 m 0.0073 m RMS North 0.0035 m 0.0065 m 0.0100 m Up 0.0468 m 0.0467 m 0.0484 m East 0.0130 m 0.0154 m 0.0217 m Max. North 0.0110 m 0.0223 m 0.0302 m Up 0.0730 m 0.0840 m 0.1090 m RTK Fix Rate 98.49% 96.47% 80.98% Figure 25. ENU errorsTTFF and RTK ﬁx type in the case 13 s of RTCM messages 29 s with 75 a s delay of 5 and 10 s. Figure 25. ENU errors and RTK fix type in the case of RTCM messages with a delay of 5 and 10 s.

Table 8. RMS and maximum of ENU errors, and TTFF and RTK fix rates in the case of RTCM messages with a delay of 5 and 10 s.

Figure 26. Time latency versus RMS values of horizontal and vertical errors in RTK mode. Figure 26. Time latency versus RMS values of horizontal and vertical errors in RTK mode. Table 8. RMS and maximum of ENU errors, and TTFF and RTK ﬁx rates in the case of RTCM messages with a delay of 5 and 10 s.

No Latency 5 s 10 s East 0.0049 m 0.0049 m 0.0073 m RMS North 0.0035 m 0.0065 m 0.0100 m Up 0.0468 m 0.0467 m 0.0484 m East 0.0130 m 0.0154 m 0.0217 m Max. North 0.0110 m 0.0223 m 0.0302 m Up 0.0730 m 0.0840 m 0.1090 m RTK Fix Rate 98.49% 96.47% 80.98% TTFF 13 s 29 s 75 s Figure 26. Time latency versus RMS values of horizontal and vertical errors in RTK mode.

Electronics 2019, 8, 1518 19 of 21

5. Conclusions With more applications of GNSS and more users relying on position, the evaluation of positioning performance is becoming increasingly more important. When evaluating the navigation performance of various GNSS receivers, it is very important to provide the same signal environment to all the receivers. However, it is difficult to perform the performance evaluation by mounting all receivers at once because of the limitation of payload capacity of vehicles such as UAVs. Therefore, the performance of GNSS receivers in dynamic modes has been assessed by comparing positioning results obtained from each receiver loaded on vehicles flying along similar trajectories independently at different times. However, such assessments do not ensure validness of evaluation for all receivers. To ensure evaluation validity of the dynamic test, exhaustively numerous field-tests are essential in real-world conditions, which impose a huge pressure on budgets and time. In this study, we proposed a methodology to guarantee the validity of the dynamic performance evaluation of GNSS receivers with only one flight test. We used RACELOGIC’s LabSat 3 Wideband with the ability to record and replay the GNSS signals to provide the same signal environment to multiple GNSS receivers. Furthermore, we developed a software that can store RTCM messages and re-broadcast them to GNSS receivers using Python. In order to verify the suitability of the system, we performed preliminary static tests and investigated whether performance degradation occurred due to the use of LabSat 3 or the RTCM Logger and Broadcaster. The results of range and position domain tests showed that performance degradation does not occur. After system verification, we performed a drone flight test using DJI’s Phantom 3 Professional. During the dynamic test, GNSS live signals were recorded in LabSat 3, and RTCM messages were also stored simultaneously by our software. Then, we performed three re-radiation tests by applying the recorded signals and RTCM messages to three different GNSS receivers (SwiftNav Piksi Multi, u-blox M8P, and u-blox EVK-6). In this manner, we obtained trajectory estimates of the drone in different positioning modes (stand-alone, DGPS, SBAS, and RTK) of all the GNSS receivers. Based on a total of six trajectories, we compared and analyzed the performance of the GNSS receivers in various positioning modes. The most significant advantage of our system is that performance analyses of unlimited GNSS receivers and modes are possible with only one flight test. It enables positioning performance evaluation of multiple receivers that cannot be loaded on a small vehicle. Moreover, it can save the cost and time of multiple flight testing as only one dynamic test is required for evaluation. We expect our suggested system to be useful as an accurate and convenient tool for evaluating the dynamic navigation performance in the near future.

Author Contributions: All the authors have contributed to the presented work. C.L. designed the evaluation system and implemented the RTCM Logger/Broadcaster software. H.Y. carried out static/drone flight tests and data analysis. A.C. and C.-S.Y. provided valuable comments and critical feedback regarding the algorithm and applications. B.P. suggested the original concept of the system and supervised its development and the direction of the research. All authors participated in formulating the idea and in discussing the proposed approach and results. Funding: This research was supported by the MSIT (Ministry of Science and ICT) of Korea under the ITRC (Information Technology Research Center) support program (IITP-2019-2018-0-01423) supervised by the IITP (Institute for Information and Communications Technology Promotion). This research was also supported by the Unmanned Vehicles Advanced Core Technology Research and Development Program through the National Research Foundation of Korea (NRF), Unmanned Vehicle Advanced Research Center (UVARC) funded by the Ministry of Science and ICT, the Republic of Korea (2017M1B3A2A01049995). Conflicts of Interest: The authors declare no conflict of interest.

References

1. Hofmann-Wellenhof, H.; Lichtenegger, H.; Wasle, E. GNSS—Global Navigation Satellite Systems; Springer: Vienna, Austria, 2008; pp. 55–85. 2. Misra, P.; Enge, P. Global Positioning System: Signals, Measurements, and Performance, 2nd ed.; Ganga-Jamuna Press: Lincoln, MA, USA, 2006; pp. 147–185. Electronics 2019, 8, 1518 20 of 21

3. Kaplan, E.D. Understanding GPS: Principles and Applications; Artech House: Boston, MA, USA, 1996; pp. 301–375. 4. UNAVCO. UNAVCO Campaign GPS/GNSS Handbook; UNAVCO: Boulder, CO, USA, 2018; pp. 1–20. 5. Kim, M.; Seo, J.; Lee, J. A Comprehensive Method for GNSS Data Quality Determination to Improve Ionospheric Data Analysis. Sensors 2014, 14, 14971–14993. [CrossRef][PubMed] 6. Seo, J.; Walter, T.; Chiou, T.-Y.; Blanch, J.; Enge, P. Evaluation of Deep Signal Fading Effects Due to Ionospheric Scintillation on GPS Aviation Receivers. In Proceedings of the Institute of Navigation GNSS, Savannah, GA, USA, 16–19 September 2008. 7. Vinande, E.; Weinstein, B.; Chu, T.; Akos, D. GNSS receiver evaluation record-and-playback test methods. GPS World 2010, 21, 28–34. 8. Cristodaro, C.; Ruotsalainen, L.; Dovis, F. Benefits and Limitations of the Record and Replay Approach for GNSS Receiver Performance Assessment in Harsh Scenarios. Sensors 2018, 18, 2189. [CrossRef][PubMed] 9. Hadas, T.; Kazmierski, K.; So´snica,K. Performance of Galileo-only dual-frequency absolute positioning using the fully serviceable Galileo constellation. GPS Solut. 2019, 23, 108. [CrossRef] 10. Kazmierski, K. Performance of Absolute Real-Time Multi-GNSS Kinematic Positioning. Artif. Satell. 2018, 53, 75–88. [CrossRef] 11. RACELOGIC. LabSat 3 Wideband. Available online: https://www.labsat.co.uk/index.php/en/products/labsat- 3-wideband (accessed on 10 December 2019). 12. RTCM Special Committee No. 104. RTCM Standard 10403.1. Radio Technical Commission for Maritime Services; RTCM Special Committee: Arlington, VA, USA, 2011; pp. 378–383. 13. RTCM Special Committee No. 104. RTCM Standard 10402.3. Radio Technical Commission for Maritime Services; RTCM Special Committee: Arlington, VA, USA, 2010. 14. Parkinson, B.W.; Enge, P.Differential GPS. In Global Positioning System: Theory and Applications; Parkinson, B.W., Spilker, J.J., Axelrad, P., Enge, P., Eds.; AIAA: Washington, DC, USA, 1996; pp. 3–50. 15. Park, B.; Kim, J.; Kee, C.; Cleveland, A.; Parsons, M.; Wolfe, D.; Kalafus, R. RRC Unnecessary for DGPS Messages. IEEE Trans. Aerosp. Electron. Syst. 2006, 42, 1149–1160. [CrossRef] 16. Park, B. A Study on Reducing Temporal and Spatial Decorrelation Effect in GNSS Augmentation System: Consideration of the Correction Message Standardization. Ph.D. Thesis, Seoul National University, Seoul, Korea, 2008. 17. Lo Presti, L.; Falletti, E.; Nicola, M.; Troglia Gamba, M. Software Defined Radio Technology for GNSS Receivers. In Proceedings of the IEEE Metrology for Aerospace (MetroAeroSpace), Benevento, Italy, 29–30 May 2014; pp. 314–319. 18. Dimc, F.; Bažec, M.; Borio, D.; Gioia, C.; Baldini, G.; Basso, M. An Experimental Evaluation of Low-Cost GNSS Jamming Sensors. J. Inst. Navig. 2017, 64, 93–109. [CrossRef] 19. Borio, D.; Gioia, C. A Sum-of-Squares Approach to GNSS Spoofing Detection. IEEE Trans. Aerosp. Electron. Syst. 2016, 52, 1756–1768. [CrossRef] 20. Borio, D.; Gioia, C.; Baldini, G.; Giuliani, R. A Real-time SDR Sanitizer for GNSS Interference Mitigation in Road Transportation. In Proceedings of the 24th International Electrotechnical and Computer Science Conference ERK 2015, Bernardin, Portoroz, Slovenia, 21–23 September 2015. 21. Hünerbein, K.; Lange, W.R. Testing Acquisition of GPS/GNSS Location and Velocity to Improve Safety in Autonomous Driving. In Proceedings of the European Test and Telemetry Conference, Nürnberg, Germany, 26–28 June 2018. 22. Kouba, J.; Héroux, P. Precise Point Positioning Using IGS Orbit and Clock Products. GPS Solut. 2011, 5, 12–28. [CrossRef] 23. Kee, C.; Parkinson, B.W.; Axelrad, P. Wide Area Differential GPS. In Proceedings of the 3rd International Technical Meeting of the Satellite Division of the Institute of Navigation, Colorado Spring, CO, USA, 19–21 September 1990; pp. 587–598. 24. Park, B.; Kee, C. The Compact Network RTK Method: An Effective Solution to Reduce GNSS Temporal and Spatial Decorrelation Error. J. Navig. 2010, 63, 343–362. [CrossRef] 25. Kim, J.; Song, J.; No, H.; Han, D.; Kim, D.; Park, B.; Kee, C. Accuracy Improvement of DGPS for Low-Cost Single-Frequency Receiver Using Modified Flächen Korrektur Parameter Correction. ISPRS Int. J. Geo Inf. 2017, 6, 222. [CrossRef] Electronics 2019, 8, 1518 21 of 21

26. Song, J.; Park, B.; Kee, C. Comparative Analysis of Height-Related Multiple Correction Interpolation Methods with Constraints for Network RTK in Mountainous Areas. J. Navig. 2016, 69, 991–1010. [CrossRef] 27. NTRIP—Networked Transport of RTCM via Internet Protocol. Available online: https://igs.bkg.bund.de/ ntrip/about (accessed on 13 October 2019). 28. Spirent. GNSS Record and Playback: Four Test Needs that Demand Richer Real World Testing, Spirent eBook. Available online: https://www.spirent.com/assets/eb/eb_four-test-needs-that-demand-richer-real- world-testing?submited=Y (accessed on 13 October 2019). 29. European Space Agency. Atomic Time. Available online: https://gssc.esa.int/navipedia/index.php/Atomic_ Time (accessed on 13 October 2019). 30. NMEA Data. Available online: https://www.gpsinformation.org/dale/nmea.htm (accessed on 24 May 2019). 31. RINEX. The Receiver Independent Exchange Format. Available online: ftp://igs.org/pub/data/format/ rinex301.pdf (accessed on 13 October 2019). 32. Trimble Geospatial. Trimble Business Center. Available online: https://geospatial.trimble.com/products-and- solutions/trimble-business-center (accessed on 13 October 2019).

© 2019 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).