S S symmetry
Article Application-Level Packet Loss Rate Measurement Based on Improved L-Rex Model
Haoliang Lan 1,2,3,*, Wei Ding 1,2,3 and Lu Deng 3
1 School of Cyber Science and Engineering, Southeast University, Nanjing 211189, China; [email protected] 2 Key Laboratory of Computer Network and Information Integration, Ministry of Education, Southeast University, Nanjing 211189, China 3 Jiangsu Key Laboratory of Computer Networking Technology, Southeast University, Nanjing 211189, China; [email protected] * Correspondence: [email protected]
Received: 4 March 2019; Accepted: 24 March 2019; Published: 27 March 2019
Abstract: The current network information asymmetry is infringing the interests of end users in non-neutral network environment. To empower users, application-level performance measurements are usually used to rebalance the information asymmetry currently disfavoring users. Application-level packet loss rate, as an important key performance indicator (KPI), receives extensive attention in academia. However, every coin has its two sides. Although the application-level packet loss estimation is simple and nonintrusive, since the information available from the application layer is relatively scarce compared to the lower layers, the estimation accuracy has no guarantee. In view of this, taking the L-Rex model as the cut-in spot, this paper focuses on leveraging the self-clocking mechanism of Transmission Control Protocol (TCP) to improve the accuracy of application-level loss estimation. Meanwhile, owing to the dynamically estimated parameters and the weakened packet reordering impact, the robustness of estimation is also expected to be improved accordingly. Finally, a series of comparative experiments at the practically relevant parameter space show that the enhanced approach and strategy detailed in this paper are effective.
Keywords: network neutrality; information asymmetry; performance evaluation; self-clocking mechanism; packet loss rate
1. Introduction The internet was originally designed to be neutral and primarily provide service for time-insensitive traffic. However, with the development of the network, users put forward higher requirements for network services than just being able to obtain information from the internet. Moreover, due to the increasing amount of data, the diversity of applications (e.g., the rise of time-sensitive applications), and the competitive nature of the internet service providers (ISP) industry, although Internet Protocol (IP) packets are still best-effort forwarding without guaranteeing delivery, the intermediate nodes usually process data by priority aimed at achieving differentiated quality of service (QoS) [1–3], which makes the network no longer neutral [4]. It is no doubt that some stakeholders will benefit from such non-neutrality, such as high-priority users who pay more fees [5]. Of course, some scholars [6] also believed that, if non-neutrality is well implemented, it will benefit all. However, now, the diversity and heterogeneity of applications and services, as well as the complex infrastructure in behind, present new challenges and make it difficult to implement effective non-neutrality that satisfies everyone. In general, the non-neutrality prioritizes privileged traffic, i.e., the high-priority traffic has a specific performance lower bound, while the low-priority traffic has a performance upper bound but no performance lower bound. Moreover,
Symmetry 2019, 11, 442; doi:10.3390/sym11040442 www.mdpi.com/journal/symmetry Symmetry 2019, 11, 442 2 of 17 the current network is information asymmetric, and end users know much less than network operators, internet service providers (ISP), and content providers [7]. Such information asymmetry, especially in a non-neutral network environment, will seriously infringe the interests of end users. This damage is ultimately reflected on the deterioration of the user’s quality of experience (QoE), and, for some ordinary users, the extent of such bad QoE naturally has no lower bound due to the lack of performance lower bound guarantees. Therefore, the quantitative characterization and accurate evaluation of an end user’s QoE are the premise of guaranteeing the interests of end users (when end users realize their interests are infringed, they can fully exercise their market rights by reasonably switching ISPs). Again, since QoE is subjective and abstract, the effective characterization and evaluation of QoE is based on the accurate measurement of QoS. Accordingly, light and user-friendly application-level QoS measurement is conducive to promoting large-scale user installation and, thus, quickly reversing the information asymmetry [8]. Among the numerous key performance indicators (KPI) indicating QoS, application-level packet loss rate, as an important KPI, is widely studied and measured [9–12]. However, as said by Basso et al., “the collected data in the application layer is more coarse-grained than the data collected with kernel-level measurements” [8]. Therefore, the available traffic information related to packet losses is less in application-level packet loss estimation, which poses a challenge to the estimation accuracy. In view of this, taking the L-Rex model [8] as the cut-in spot, this paper focuses on improving the accuracy and robustness of application-level loss estimation. That is, leveraging the TCP’s self-clocking mechanism [13], we mine the information contained in the burst-gap traffic patterns to dynamically estimate the two decisive parameters required by L-Rex. To rebalance the robustness, the effect of packet reordering is also considered. Also, a series of comparative experiments performed with real measured data and testbed measurements eventually show that the improved L-Rex (IM-L-Rex) is effective for the entire measured parameter space. In this paper, we firstly recall the application-level loss rate estimation model L-Rex and specify its limitations. Then, we utilize the TCP’s self-clocking mechanism to dynamically estimate the two decisive parameters required by L-Rex and exclude the effect of packet reordering. After that, to elaborate the model, we describe the implementation of IM-L-Rex. Finally, we detail and discuss the comparative experiments performed with real measured data and testbed measurements. The remainder of this paper is organized as follows. Section2 outlines the L-Rex model and its limitations. Section3 sketches our enhanced version with an eye toward overcoming the deficiencies of L-Rex model. Section4 details the implementation of IM-L-Rex. Section5 presents the comparative experiment results. Section6 concludes the paper.
2. L-Rex Model The difficulty in application-level packet loss estimation is how to use the information available only from the application layer to achieve loss estimation and be as accurate as possible. It is well known that, for TCP, the packet loss detection and repair are controlled by the transport layer, while what the application layer has and can contact with the lower transport layer are just a series of system calls. In this case, the L-Rex is naturally based on the system calls to estimate the application-level packet loss rate. Specially, it is inspired by the empirical observation about the dynamics of data returned by the recv() system call. Under the current TCP/IP architecture, packet loss is repaired with retransmission. Accordingly, it is almost a natural idea to take retransmissions as building blocks to construct loss estimation algorithms [14–19]. Likewise, L-Rex is also built on the retransmission patterns. The difference lies in that L-Rex mines the hidden retransmission events contained in the data sequence returned by the application-layer recv(). As a result, there is little information available and the problem becomes more challenging.
2.1. Model Description In the absence of packet loss, the data sequence returned by recv() has the following features: (a) The recv() will be triggered and return a typical number of bytes as soon as data are available; Symmetry 2019, 11, 442 3 of 17
(b) During a burst, the time interval between two consecutive recv() is very small; (c) The “silence period” between two consecutive bursts is usually smaller than Round Trip Time (RTT).
In the case of packet loss, the data sequence returned by recv() appears as follows:
(A) The recv() is blocked due to packet loss, which makes the silence period longer; in the best case (i.e., fast retransmission), an additional RTT is needed to re-trigger recv(); (B) When the lost packet(s) are repaired, a large and non-typical number of bytes will be returned to userspace.
On the basis of the above observations, Basso et al. [8] found that many losses followed the pattern indicated by (A) and (B), i.e., a longer silence period, and a large, non-typical returned number of bytes. Therefore, L-Rex scans the data sequence returned by recv() and counts the number of following cases to achieve the estimation for packet losses:
(i) The time elapsed since the previous recv() is greater than H · RTT; (ii) The number of received bytes is greater than 1 MSS and less frequent than K;
where H is the smoothing parameter, and K is the ratio of the maximum number of bytes returned by recv() at a time to the total number of bytes received by the receiver during a TCP transfer. Once the number of estimated packet losses is determined, the estimated application-level packet loss rate (APLREST) is calculated as
Losses APLR = , (1) EST Received_bytes/MSS where Losses is the number of estimated packet losses, Received_bytes is the total number of bytes received by the receiver during a TCP transfer, and MSS is the maximum segment size obtained through getsockopt().
2.2. Limitations of L-Rex As a key parameter, the accurate estimation of RTT is crucial to L-Rex. However, in L-Rex implementation, the RTT value is the connect delay, i.e., the time required for the connect() system call to complete. In general, the RTT at the TCP connection establishment is not completely consistent with that after the transmission is stable. For a TCP flow, after the transmission is stable, with the increase of the sending window and the limitation of the bottleneck bandwidth, the RTT may increase due to the queuing problem at the bottleneck. Although Basso et al. used the parameter H to smooth RTT (H · RTT), the value of H is set to 0.7, which is still statically assigned. Similarly, the parameter K is also static in L-Rex implementation with a value of 1%. As we know, due to the packet dynamics along the network path and the diversity of the network that the packets travel through, it is impossible for static parameters to be applicable to all loss patterns. Therefore, the robustness of L-Rex is poor, and the accuracy in practical applications is also difficult to guarantee. In addition, relying solely on the information available from the data receiver side cannot achieve completely accurate RTT estimation, together with the fact that packet reordering [20] can also result in the pattern indicated by (A) and (B), which will eventually fool L-Rex’s detection rules (i) and (ii). For instance, in Figure1, the sender sends four data packets; the first two are successfully received and returned to the user space, but due to the delayed D3 caused by packet reordering, the time difference between the last two packets and the first two packets is greater than or equal to the estimated RTT. This situation is in line with the detection rules (i) and (ii), which fools L-Rex. Therefore, distinguishing the pattern caused by packet reordering from that indicated by (A) and (B) is also beneficial to further improve the estimation accuracy of L-Rex. Symmetry 2019, 11 FOR PEER REVIEW 3 of 17 b) During a burst, the time interval between two consecutive recv() is very small; c) The “silence period” between two consecutive bursts is usually smaller than Round Trip Time (RTT). In the case of packet loss, the data sequence returned by recv() appears as follows:
A) The recv() is blocked due to packet loss, which makes the silence period longer; in the best case (i.e., fast retransmission), an additional RTT is needed to re-trigger recv();
B) When the lost packet(s) are repaired, a large and non-typical number of bytes will be returned to userspace. On the basis of the above observations, Basso et al. [8] found that many losses followed the pattern indicated by A) and B), i.e., a longer silence period, and a large, non-typical returned number of bytes. Therefore, L-Rex scans the data sequence returned by recv() and counts the number of following cases to achieve the estimation for packet losses: (i) The time elapsed since the previous recv() is greater than H · RTT; (ii) The number of received bytes is greater than 1 MSS and less frequent than K; where H is the smoothing parameter, and K is the ratio of the maximum number of bytes returned by recv() at a time to the total number of bytes received by the receiver during a TCP transfer. Once the number of estimated packet losses is determined, the estimated application-level packet loss rate (APLREST) is calculated as Losses APLR = , (1) EST Received_bytes/MSS where Losses is the number of estimated packet losses, Received_bytes is the total number of bytes received by the receiver during a TCP transfer, and MSS is the maximum segment size obtained through getsockopt(). Symmetry 2019, 11, 442 4 of 17 2.2. Limitations of L-Rex
Source Sink
Recv()
RTT
≥
Recv()
Figure 1. Sample of packet reordering reordering.. 3. IM-L-Rex As a key parameter, the accurate estimation of RTT is crucial to L-Rex. However, in L-Rex implementation3.1. Self-Clocking, Mechanismthe RTT value is the connect delay, i.e., the time required for the connect() system call to complete. In general, the RTT at the TCP connection establishment is not completely In the network, the number of active flows carried by a single router is huge, while, in core routers, consistent with that after the transmission is stable. For a TCP flow, after the transmission is stable, it can even reach thousands or even more. All of these flows have no any specific information about with the increase of the sending window and the limitation of the bottleneck bandwidth, the RTT the network state, which is especially true for TCP/IP networks. At the same time, TCP is greedy, but it may increase due to the queuing problem at the bottleneck. Although Basso et al. used the parameter does not know which transmission rate to use to ensure high resource utilization without overloading H to smooth RTT (H · RTT), the value of H is set to 0.7, which is still statically assigned. Similarly, the the network. In this case, it can only slowly increase the size of the congestion window through a parameter K is also static in L-Rex implementation with a value of 1%. As we know, due to the strategy known as slow start until congestion happens; then, it will reduce the sending rate and the congestion window in respond to the congestion. In this way, it will loop back and forth until the data transfer is completed. Such a reciprocating process is called TCP’s self-clocking mechanism and eventually produces a traffic pattern where the packets are sent in bursts and the spacing between the bursts is probably preserved from one round trip to the next. Although the bursts (e.g., split or merged) and spacing may be changed due to the packet losses or competing traffic, the changes happen infrequently, and the bursts are preserved for at least several round trips after each change.
3.2. Methodology By detecting the repeated burst-gap patterns, TCP’s self-clocking mechanism is widely used to dynamically estimate RTT and follow changes in the RTT throughout the lifetime of a TCP session [13,21,22]. Since recv() will be triggered as soon as data are available, the self-clocking (or burst-gap) patterns are also passed to the application layer. This provides us with the possibility to dynamically estimate RTT required by L-Rex utilizing the self-clocking mechanism. Autocorrelation is the correlation (similarity) of a dataset with a delayed copy of itself, as a function of the time lag between the datasets. The discrete autocorrelation analysis is a mathematical tool for finding repeating patterns, such as the periodic burst-gap pattern obscured by noise in TCP’s self-clocking mechanism. Similar to Veal et al.’s work [21], in this paper, combined with the TCP self-clocking mechanism, we leverage discrete autocorrelation analysis to realize RTT dynamic estimation. Specifically, the estimation is repeated once per measurement interval T. During this interval, array P(t) ranging from 1 to T is used to store the number of segments that arrive with timestamp t. Once an interval time T is expired, the discrete autocorrelation R(l) is calculated for each lag l from 1 to l/2, and the estimated RTT is the max(R), where R(l) is calculated as
1 T−l R(l) = (P − µ)(P − µ), (2) ( − )σ2 ∑ t t+l T l t=1 where the measurement interval time T is supplied as a parameter and determines the upper bound on the maximum RTT that can be measured; in our implementation, T is set to 500 ms. Also, to overcome Symmetry 2019, 11, 442 5 of 17 the multiple burst-gap patterns, the limit of fractional lags and the maximum correlation percentage are set to 1/4 and 75%, respectively, while the lower bound RTT is set to min (10 ms, the Nyquist period within T) to exclude the effect of extremely small lags caused by the noises like competing traffic, back-to-back received packets, and other network conditions disrupting the burst-gap patterns. Since RTT is dynamically estimated, the parameter H used for smoothing RTT is no longer needed in our loss detection. Moreover, regarding parameter K, the number of received segments at a time, no matter how non-typical, will not exceed the number of segments sent by the sender in one sending round. In this case, we use the peak value of the strong correlation corresponding to the estimated RTT to infinitely approximate the maximum number of segments sent within T, i.e., the parameter K is dynamically computed as Bytes_peak K = , (3) Received_bytes where Bytes_peak is the maximum number of segments sent in all sending rounds within one measurement interval T. In order to cope with the problem of packet reordering, we explore distinguishing the pattern caused by packet reordering from that indicated by (d) and (e). In fact, just like L-Rex’s full name “Likely Rexmit”, L-Rex estimates packet losses by identifying likely “rexmit” (i.e., retransmissions). For a loss event during a TCP transfer, we noted that the retransmission(s) used for repairing it are either triggered by a fast retransmission or triggered by an overtime retransmission. Therefore, a loss event is either repaired with a group of retransmission(s) that begin with fast retransmission or repaired with a group of retransmission(s) that begin with overtime retransmission. In view of this, we introduce the following definitions:
Definition 1 (Fast loss event): It is repaired with a group of retransmission(s) that begin with fast retransmission.
Definition 2 (Overtime loss event): It is repaired with a group of retransmission(s) that begin with overtime retransmission.
By Definition 1 and Definition 2, the retransmissions used for repairing the loss events during a TCP transfer can be divided into the following two categories: ( beginning with fast retransmission; Retransmissions beginning with overtime retransmission.
As shown in Figure2, to exclude the effect of packet reordering, instead of adopting the L-Rex’s traditional way of identifying retransmissions coarsely, we further refine the loss set determined by (i) and (ii) to screen out the retransmissions belonging to set F∪O. If N and M denote the number of fast loss events and the number of overtime loss events in a TCP connection, respectively, then the number of retransmissions belonging to set F∪O (i.e., the retransmissions used for repairing the loss events in a TCP connection) can be calculated as
N M Retransrepair = ∑ Fasti + ∑ Overtimej, (4) i=1 j=1 where Fasti denotes the number of retransmissions used for repairing the ith fast loss event, and Overtimej denotes the number of retransmissions used for repairing the jth overtime loss event. Referring to Equation (4), to determine Retransrepair, we firstly need to identify each loss event in a TCP connection and then determine the retransmissions used for repairing it. Symmetry 2019, 11 FOR PEER REVIEW 5 of 17 percentage are set to 1/4 and 75%, respectively, while the lower bound RTT is set to min (10 ms, the Nyquist period within T) to exclude the effect of extremely small lags caused by the noises like competing traffic, back-to-back received packets, and other network conditions disrupting the burst-gap patterns. Since RTT is dynamically estimated, the parameter H used for smoothing RTT is no longer needed in our loss detection. Moreover, regarding parameter K, the number of received segments at a time, no matter how non-typical, will not exceed the number of segments sent by the sender in one sending round. In this case, we use the peak value of the strong correlation corresponding to the estimated RTT to infinitely approximate the maximum number of segments sent within T, i.e., the parameter K is dynamically computed as Bytes_peak K = , (3) Received_bytes
Symmetrywhere Bytes_peak2019, 11, 442 is the maximum number of segments sent in all sending rounds within6 of one 17 measurement interval T. In order to cope with the problem of packet reordering, we explore distinguishing the pattern causedFor by fast packet loss event, reordering we identify from that it by indicated detecting by the d) first and fast e). retransmissionIn fact, just like used L-Rex’s for repairing full name it. Without“Likely Rexmit loss of” generality,, L-Rex estimates a fast retransmission packet losses by is causedidentifying by more likely than “rexmit or equal” (i.e., to fourretransmissions). out-of-order arrivalsFor a loss that event appear during at the a TCP receiver transfer, side inwe the noted following that the two retransmission(s) forms: used for repairing it are either triggered by a fast retransmission or triggered by an overtime retransmission. Therefore, a loss Form 1: The out-of-order arrivals are continuous and can be returned by recv() at one time. event is either repaired with a group of retransmission(s) that begin with fast retransmission or For instance, in Figure3a, the four continuous out-of-order arrivals result in recv() returning to repaired with a group of retransmission(s) that begin with overtime retransmission. In view of this, userspace at least five segments at a time (i.e., the fast retransmission used for repairing the lost we introduce the following definitions: packet plus the four continuous out-of-order segments that arrived while recv() was blocked).
DefinitionForm 2: 1The (Fast out-of-order loss event): arrivals It is are repaired intermittent with a group (discontinuous) of retransmission(s) and need that at least begin two with recv()s fast retransmission.to be returned to userspace; the first recv() is triggered by a fast retransmission, while the time interval between the subsequent recv() and the previous recv() is greater than RTT. For instance, Definitionin Figure 2 3 (Overtimeb, the four loss out-of-order event): It arrivals is repaired triggering with a fast group retransmission of retransmission(s) require that calling begin recv() with overtimethree retransmission. times to return to user space. In this case, the recv () triggered by the first fast retransmission returns a fast retransmission and the first two out-of-order arrivals; after RTT, the recv() triggered By Definition 1 and Definition 2, the retransmissions used for repairing the loss events during a by the second retransmission returns a retransmission and the third out-of-order arrival; after TCP transfer can be divided into the following two categories: another RTT, recv() triggered by the third retransmission returns a retransmission and the fourth out-of-order arrival. ⟡ beginning with fast retransmission; Retransmissions { ⟡ beginning with overtime retransmission.
F F⋃O
O S
S: Retransmissions determined by i) and ii) F: Retransmissions beginning with fast retransmission O: Retransmissions beginning with overtime retransmission F⋃O: Misidentified retransmissions due to packet reordering Symmetry 2019, 11 FOR PEER REVIEW 6 of 17 FigureFigure 2. 2.Retransmissions Retransmissions determineddetermined by by (i) i) and (ii).ii).
Lost packet Lost packets
1 2 3 4 ...... 1 2 3 4 ......
Continuous out-of-order arrivals Intermittent out-of-order arrivals
repair repair
Fast retransmission Repair starting with fast retransmission
1 2 3 4 ...... 1 2 3 4 ......
recv() recv() recv() recv() ⩾ RTT ⩾ RTT
(a) (b)
Figure 3. 3. FastFast loss lossevent event scenario scenario.. (a) Continuous (a) Continuous out-of-order out-of-order arrivals; (b arrivals;) Intermittent (b) Intermittentout-of-order out-of-orderarrivals. arrivals.
BasedAs shown on the in Figure discussion 2, to above,exclude assume the effect that of the packet data reordering, sequence returned instead of by adopting recv() during the L a-Rex’s TCP S R T B R T B ∀i ∈ N R T B transfertraditional is described way of identifying with set =retransmissions {< 1, 1, 1>, ... coarsely,, < n, wen, nfurther>}, where refine the, loss < i, seti, determinedi> denotes theby i) and ii) to screen out the retransmissions belonging to set F∪O. If N and M denote the number of fast loss events and the number of overtime loss events in a TCP connection, respectively, then the number of retransmissions belonging to set F∪O (i.e., the retransmissions used for repairing the loss events in a TCP connection) can be calculated as
N M
Retransrepair = ∑ Fasti + ∑ Overtimej, (4) i=1 j=1 where Fasti denotes the number of retransmissions used for repairing the ith fast loss event, and Overtimej denotes the number of retransmissions used for repairing the jth overtime loss event. Referring to Equation (4), to determine Retransrepair, we firstly need to identify each loss event in a TCP connection and then determine the retransmissions used for repairing it. For fast loss event, we identify it by detecting the first fast retransmission used for repairing it. Without loss of generality, a fast retransmission is caused by more than or equal to four out-of-order arrivals that appear at the receiver side in the following two forms: Form 1: The out-of-order arrivals are continuous and can be returned by recv() at one time. For instance, in Figure 3a, the four continuous out-of-order arrivals result in recv() returning to userspace at least five segments at a time (i.e., the fast retransmission used for repairing the lost packet plus the four continuous out-of-order segments that arrived while recv() was blocked). Form 2: The out-of-order arrivals are intermittent (discontinuous) and need at least two recv()s to be returned to userspace; the first recv() is triggered by a fast retransmission, while the time interval between the subsequent recv() and the previous recv() is greater than RTT. For instance, in Figure 3b, the four out-of-order arrivals triggering fast retransmission require calling recv() three times to return to user space. In this case, the recv () triggered by the first fast retransmission returns a fast retransmission and the first two out-of-order arrivals; after RTT, the recv() triggered by the second retransmission returns a retransmission and the third out-of-order arrival; after another RTT, recv() triggered by the third retransmission returns a retransmission and the fourth out-of-order arrival. Based on the discussion above, assume that the data sequence returned by recv() during a TCP transfer is described with set S = {
Symmetry 2019, 11, 442 7 of 17
ith group data returned by recv(), Ri denotes the byte length of
Li = min ∪ k. (6) k≥i Tk+1−Tk From Figure3, we can see that, for an identified fast lost event, the number of retransmissions used for repairing it depends on the number of recv()s corresponding to it. Therefore, after identifying the fast loss event, Fasti can be calculated as Fasti = Li − i + 1. (7) For an overtime loss event, we identify it by detecting the first overtime retransmission used for repairing it. Since the overtime retransmission is caused by an expired retransmission timeout (RTO) timer, Referring to RFC6298 [23], once RTT is dynamically measured, RTO can be calculated as RTO = RTTS + 4 × RTTD, (9) where RTTS is the smoothed RTT and RTTD is the weighted average of the deviation of RTT. They can be calculated as follows: new RTTS = (1 − α) × (old RTT S) + α × (new RTT sample), (10) where α is an empirical constant, and the value of this constant is 1/8. new RTTD = (1 − β) × (old RTT D) + β × |RTTS − new RTT sample|, (11) where β is an empirical constant, and the value of this constant is 1/4. Likewise, as shown in Figure4, for an overtime loss event, the number of retransmissions used for repairing it also depends on a group of recv()s with a time interval greater than RTT. Therefore, Overtimej can be calculated as Overtimej = Lj − j + 1, (12) where Lj indicates the last recv() corresponding to the identified overtime loss event, and it can be calculated as Lj = min ∪ k. (13) k≥j Tk+1−Tk Lost packet Lost packets 1 2 3 ...... 1 2 3 ...... Continuous out-of-order arrivals Intermittent out-of-order arrivals repair repair Repair with overtime retransmission Repair starting with overtime retransmission 1 2 3 ...... 1 2 3 ...... recv() recv() recv() recv() ⩾ RTT ⩾ RTT (a) (b) FigureFigure 4. 4.Overtime Overtime loss loss event event scenario.scenario. (a (a) )C Continuousontinuous out out-of-order-of-order arrivals arrivals;; (b) (Ibntermittent) Intermittent out-of-order arrivals. out-of-order arrivals 3.3. TCP Variants Likewise, as shown in Figure 4, for an overtime loss event, the number of retransmissions used for repairingFollowing it thealso operation depends ofon IM-L-Rex, a group of we recv()s can see with that a it time estimates interval packet greater losses than by RTT retransmissions.. Therefore, AsOvertime we know,j can forbe calculated different TCP as variants, different techniques are used to control TCP retransmissions. Therefore, the TCP variant in use is crucial to the accuracy of IM-L-Rex. Given this, this paper Overtime = L − j + 1, investigates the performance of IM-L-Rex underj the followingj TCP variants: (12) • Reno: According to RFC 2581 [24], it cooperates with TCP’s basic congestion control mechanisms (e.g., slow start, congestion avoidance, fast recovery, etc.) to repair packet losses with overtime where Lj indicates the last recv() corresponding to the identified overtime loss event, and it can be retransmissions and fast retransmissions. calculated as • NewReno: This version, described in RFC 2582 [25], is an extension to the Reno. It mainly improves TCP’s fast recovery algorithmLj = min to⋃ avoid multiple k . retransmission timeouts in Reno’s fast k≥j (13) recovery phase. Tk+1−Tk 3.4. SummaryAfter an overview of IM-L-Rex, in the section, we systematically introduce its packet loss detection process. The parameters involved in IM-L-Rex are shown in Table1. Among them, the first four parametersIn thisare section, used to we dynamically detailed IM determine-L-Rex. SinceRTT, and IM- theL-Rex parameters is an evolution numbered of 5 L to-Rex, 11 are to used better to estimateunderstand the it, application-level we outline the differences loss rate. between the two technologies as follows: Symmetry 2019, 11 FOR PEER REVIEW 9 of 17 1) As the key parameters, RTT, H and K are all static in L‐Rex, i.e., RTT is the time required for the connect() system call to complete, while the parameters H and K were optimized from repeated experiments on Asymmetric Digital Subscriber Lime (ADSL) and fast Ethernet, with values of 0.7 and 1%, respectively. On the contrary, the parameters RTT and K are dynamically estimated in IM‐L‐Rex by leveraging the TCPʹs self‐clocking mechanism. Moreover, the parameter H used for smoothing RTT is no longer needed in IM‐L‐Rex due to the dynamically estimated RTT. 2) Compared with L‐Rex, IM‐L‐Rex is also committed to excluding the adverse effect of packet reordering by refining overtime retransmissions and fast retransmissions. Symmetry 2019, 11, 442 9 of 17 4. Implementation TableTable 1. Description 1. Description of parametersof parameters involved involved in in improved improved L‐Rex L-Rex (IM‐ (IM-L-Rex).L‐Rex). ID Parameter Parameter description 1 Intvltime The time interval of RTT dynamic estimation 2 Nqstperiod The Nyquist period 3 Frctnlag The limit of fractional lags 4 Crltnpercent The maximum correlation percent 5 RTT Dynamic round trip time 6 RTO Retransmission timeout 7 Pkvalue The maximum number of segments sent in all sending rounds within intvltime 8 Timestamp The timestamp of each recv() 9 Recvsingle The returned number of bytes of each recv() 10 Recvtotal The total number of bytes returned by recv() during a TCP transfer 11 MSS Maximum segment size 12 EPLR Estimated application‐level packet loss rate After an overview of IM‐L‐Rex, in the section, we systematically introduce its packet loss Followdetection the process. loss detection The parameters process involved shown in inIM‐ FigureL‐Rex are5, weshown implemented in Table 1. Among IM-L-Rex. them, the Specifically, first within thefour time parameters interval are Intvltime used to dynamically (500 ms), thedetermine self-clocking-based RTT, and the parametersRTT estimation numbered firstly 5 to 11 operates are on used to estimate the application‐level loss rate. the data sequence formed by the arriving recv()s to get the dynamic RTT. To improve the RTT estimation Follow the loss detection process shown in Figure 5, we implemented IM‐L‐Rex. Specifically, accuracy,within the the parameters time interval Nqstperiod, Intvltime (500 Frctnlag, ms), the andself‐clocking Crltnpercent‐based RTT are estimation used to exclude firstly operates the negative effects. Amongon the data them, sequence Nyquist formed is used by the for arriving correcting recv()s the to small get the lags dynamic caused RTT by. To the improve noises likethe RTT competing traffic, back-to-backestimation accuracy, received the parameters packets, and Nqstperiod, other network Frctnlag, conditions and Crltnpercent disrupting are used the to burst-gap exclude the patterns. In our implementation,negative effects. Among the lagsthem, less Nyquist than is min(Nyquist used for correcting period, the small 10 ms) lags arecaused not by considered. the noises like Frctnlag competing traffic, back‐to‐back received packets, and other network conditions disrupting the and Crltnpercent are set to 1/4 and 75% to exclude the effect of strong autocorrelation at multiples burst‐gap patterns. In our implementation, the lags less than min(Nyquist period, 10 ms) are not of the RTTconsidered.. Furthermore, Frctnlag whenand Crltnpercent estimating areRTT set, theto 1/4 parameters and 75% to RTO exclude and Kthededuced effect of fromstrong Pkvalue are alsoautocorrelation obtained synchronously. at multiples of Then, the RTT for. theFurthermore, data returned when estimating by each recv(), RTT, the we parameters determine RTO whether it indicatesand a likelyK deduced retransmission from Pkvalue by are using also obtained the parameters synchronously. Timestamp, Then, forRTT the, anddata Kreturned. If it does, by each we further determinerecv(), whether we determine it indicates whether a packet it indicates loss by meansa likely of theretransmission relevant parameters by using theRTT parameters, Recvsingle, and RTO (i.e.,Timestamp, the parameters RTT, andRTT K. If itand does, Recvsingle we further determine for fast loss whether event, it indicates and the a parameters packet loss byRTT meansand RTO of the relevant parameters RTT, Recvsingle, and RTO (i.e., the parameters RTT and Recvsingle for for overtime loss event). After that, we divide the total number of packets (Recvtotal/MSS) by the fast loss event, and the parameters RTT and RTO for overtime loss event). After that, we divide the estimatedtotal number number of packets packet (Recvtotal/MSS) losses (Lossnum by ++)the estimated to get the number EPLR. of Finally, packet losses the algorithm(Lossnum++) to IM-L-Rex get is shownSymmetry inthe Algorithm EPLR. 2019, 11Finally, FOR 1. PEER the algorithmREVIEW IM‐L‐Rex is shown in Algorithm 1. 10 of 17 Start Arriving recv() Intvltime Nqstperiod Frctnlag Crltnpercent form Data sequence Each recv() Self-clocking Timestamp Recvsingle RTT RTO Pkvalue K yes no no Rexmit? Fast? Overtime? Reordering yes yes no Losses++ no yes Losses++ Last? EPLR = End Recvtotal/MSS FigureFigure 5. 5.Loss Loss detection process process of of IM IM-L-Rex.-L-Rex. Algorithm 1: IM-L-Rex 01. //Preprocessing stage 02. Total_num = Loss_num = Nqst = EPLR = 0 03. Intvl = 500 04. Frctn = 1/4 05. Crltn = 0.75 06. S = null 07. for Recvdata returned by each recv() 08. //Form data sequence 09. Recvdata.addArray(S) 10. Total_num += Recvdata.getPktnum() 11. end for 12. for Recvdata in S 13. //Count application level traffic losses 14. if Recvdata.isRexmit(S, Intvl, Nqst, Frctn, Crltn) 15. if Recvdata.isFast(S, Intvl, Nqst, Frctn, Crltn) 16. Loss_num+=1 17. else if Recvdata.isFast(S, Intvl, Nqst, Frctn, Crltn) 18. Loss_num+=1 19. else 20. continue 21. else 22. continue 23. end for 24. EPLR = Loss_num / Total_num 5. Experiments In this section, we test IM-L-Rex with real measured data and testbed measurements. Among them, the real measured data obtained through controlled TCP transfers are used to verify the effectiveness of dynamic parameters, while the testbed measurements are performed to analyze the impact of packet reordering, TCP variants, and parameters RTT and K. Symmetry 2019, 11, 442 10 of 17 Algorithm 1: IM-L-Rex 01. //Preprocessing stage 02. Total_num = Loss_num = Nqst = EPLR = 0 03. Intvl = 500 04. Frctn = 1/4 05. Crltn = 0.75 06. S = null 07. for Recvdata returned by each recv() 08. //Form data sequence 09. Recvdata.addArray(S) 10. Total_num += Recvdata.getPktnum() 11. end for 12. for Recvdata in S 13. //Count application level traffic losses 14. if Recvdata.isRexmit(S, Intvl, Nqst, Frctn, Crltn) 15. if Recvdata.isFast(S, Intvl, Nqst, Frctn, Crltn) 16. Loss_num+=1 17. else if Recvdata.isFast(S, Intvl, Nqst, Frctn, Crltn) 18. Loss_num+=1 19. else 20. continue 21. else 22. continue 23. end for 24. EPLR = Loss_num / Total_num 5. Experiments In this section, we test IM-L-Rex with real measured data and testbed measurements. Among them, the real measured data obtained through controlled TCP transfers are used to verify the effectiveness of dynamic parameters, while the testbed measurements are performed to analyze the impact of packet reordering, TCP variants, and parameters RTT and K. 5.1. Controlled TCP Transfers In order to focus on the model itself, the semantically complete TCP transfers are used to evaluate IM-L-Rex, where semantically complete TCP transfer is defined as a TCP transfer for which one can see the connection set-up (SYN-ACK) and another connection tear-down (FIN-ACK). Accordingly, the controlled TCP transfer (called “controlled TCP transfer” because we had full control over the two communicating end-hosts) shown in Figure6 is used to produce the wanted semantically complete TCP transfer. In our previous work [12], we implemented controlled TCP transfer for research purposes. Specifically, our experiments involved four servers (see Table2 for configuration information) within JSERNET and ten client hosts in the outside of JSERNET. JSERNET (Jiangsu Education and Research Network) is a regional academic network of China Education and Research Network (CERNET), and it covers more than 100 research units and universities. In January 2006, its backbone bandwidth increased from OC-48 to OC-192. In Table2, hosts No. 1 to No. 3 are communication servers and are used for conducting controlled TCP transmissions with clients to form the required TCP streams, while host No. 4 is a storage server that stores the packet traces captured from the two communicating end-hosts. The controlled TCP transfer at the client side is realized with the Flow_Mirror, the tool that we developed and maintain [27]. Moreover, to fully evaluate the accuracy of IM-L-Rex, we placed some files of varied sizes on each communication server, and irregularly scheduled an Hyper Text Transfer Protocol based (HTTP-based) file transfer (that is, the client tool Flow_Mirror obtains the resource identified by the Uniform Resource Locator (URL) from the server with an HTTP GET request) Symmetry 2019, 11, 442 11 of 17 between randomly chosen communication server and client host to obtain the TCP transfers with different sizes and various loss rates. During transfer, we used the sniffers (TCPDUMP on the sever side and WireShark on the client side) to collect packet traces from two endpoints of a TCP connection andSymmetry compared 2019, 11 theFOR data PEER packets REVIEW to obtain the (Real Packet Loss Rate) RPLR. 11 of 17 Server Client Controlled Controlled transfer transfer Sniffer Sniffer Data aggregation TCP transfer Expired flows Database RPLR FigureFigure 6.6. ControlledControlled TCPTCP transfertransfer architecture.architecture. Table 2. Server configuration information. 5.1. Controlled TCP Transfers Server-ID IP Address Port Number RAM ROM Table 2. Server configuration information. 1 211.65.*.31 1313 512 MB 8 GB Server2-ID IP 211.65.*.32 address Port number 1313 RAM 512 MB ROM 8 GB 3 211.65.*.33 1313 512 MB 8 GB 41 211.65.*.31 211.65.*.34 1313 80 512 512 MB MB 8 8 GB GB 2 211.65.*.32 1313 512 MB 8 GB 3 211.65.*.33 1313 512 MB 8 GB Notably, when validating L-Rex, Basso et al. used retransmissions to approximate RPLR. Although 4 211.65.*.34 80 512 MB 8 GB the retransmitted data pkts is a good approximation of packet losses, the spurious retransmissions caused by flaws in TCP’s retransmission schemes [14] make the retransmissions larger than the actual Table 3. EVLT_SET. packet losses. Given this, in this paper, we get the real packet losses by comparing the packet traces at two communicatingRPLR end-hosts [12]. Again,1 MB for the data10 MB packet performing100 MB IP fragmentation [28], as long as one fragment[10 belonging−3, ∞) to it57 is lost,(19%) the entire68 data (26%) packet is considered57 (35%) lost. In addition, the involved models in this(10−4 paper, 10−3) are all111 based (37%) on application109 (42%) layer data.111 Thus, (44%) it is not enough to just capture the packet traces[0, 10 for−4 calculating) 132 RPLR. (44%) To obtain83 the (32%) required data,132 (21%) during a TCP transfer, we also synchronously record the number of bytes returned by each recv() and the time elapsed since the previousIn order recv(), to as focus wellas on the the values model returned itself, by the other semantically system calls, complete such as TCP getsockopt() transfers and are connect(). used to Finally,evaluate the IM dataset-L-Rex, EVLT_SET where semantically for evaluating complete IM-L-Rex TCP transfer is shown is in defined Table3 .as As a weTCP can transfer see, three for which sizes ofone file can transfers see the (1 Mbytes, connection 10 Mbytes, set-up (SYN and 100-ACK) Mbytes) and were another chosen connection to evaluate tear the-down performance (FIN-ACK). of IM-L-RexAccordingly, when the facing controlled short and TCP long transfer TCP flows.(called Moreover, “controlled for TCP rigorous transfer” evaluation, because the we practically had full relevantcontrol over loss ratethe two parameter communicating space was end also-hosts divided) shown into in three Figure intervals 6 is used ([0,10 to− produce4), (10−4 ,the 10− wanted3), and (10semantically−3, ∞)) to compare complete the TCP performance transfer. In of ourIM-L-Rex previous under work different [12], weloss implemented rates. In Table controlled3, the first row TCP istransfer the information for research of TCP purposes. flows withSpecifically, loss rate our greater experiments than 10− 3i,nvolved totaling four 195 TCPservers transfers, (see Table while 2 thefor secondconfiguration row is the information) information within of the loss JSERNET rate between and ten 10 − client4 and 10 hosts−3, totaling in the 308 outside TCP transfers, of JSERNET. and theJSERNET third row (Jiangsu is theresults Education of the and loss rateResearch lower Network) than 10−4 ,is totaling a regional 257 TCP academic transfers. network The first of column China isEducation the TCP transfers and Research with file Network size of 1 (CERNET), MB, and the and numbers it covers of such more TCP than transfers 100 research corresponding units and to eachuniversities. loss rate In interval January are 2006, 57 (19%), its backbone 111 (37%), bandwidth and 132 (44%), increased respectively, from OC while-48 to the OC second-192. In column Table is2, thehosts TCP No. transfers 1 to No. with 3 file are size communication of 10 MB, and servers the numbers and are of used such TCP for conducting transfers corresponding controlled TCP to eachtransmissions loss rate interval with clients are 68 to (36%), form the 109 required (42%), and TCP 83 streams, (32%), respectively, while host No. and 4 the is a third storage column server is that the TCPstores transfers the packet with filetraces size captured of 100 MB, from and the the numberstwo communicating of such TCP end transfers-hosts. corresponding The controlled to each TCP losstransfer rate intervalat the client are 70side (35%), is realized 88 (44%), with and the 42 Flow_Mirror, (21%), respectively. the tool that we developed and maintain [27]. Moreover, to fully evaluate the accuracy of IM-L-Rex, we placed some files of varied sizes on each communication server, and irregularly scheduled an Hyper Text Transfer Protocol based (HTTP-based) file transfer (that is, the client tool Flow_Mirror obtains the resource identified by the Uniform Resource Locator (URL) from the server with an HTTP GET request) between randomly chosen communication server and client host to obtain the TCP transfers with different sizes and various loss rates. During transfer, we used the sniffers (TCPDUMP on the sever side and WireShark on the client side) to collect packet traces from two endpoints of a TCP connection and compared the data packets to obtain the (Real Packet Loss Rate) RPLR. Symmetry 2019, 11 FOR PEER REVIEW 12 of 17 Notably, when validating L‐Rex, Basso et al. used retransmissions to approximate RPLR. Although the retransmitted data pkts is a good approximation of packet losses, the spurious retransmissions caused by flaws in TCP’s retransmission schemes [14] make the retransmissions larger than the actual packet losses. Given this, in this paper, we get the real packet losses by Symmetrycomparing2019 ,the11, 442packet traces at two communicating end‐hosts [12]. Again, for the data packet12 of 17 performing IP fragmentation [28], as long as one fragment belonging to it is lost, the entire data packet is considered lost. In addition, theTable involved 3. EVLT_SET. models in this paper are all based on application layer data. Thus, it is not enough to just capture the packet traces for calculating RPLR. To obtain the required data, duringRPLR a TCP transfer, 1we MB also synchronously 10 MB record the number 100 of MB bytes returned by each recv() and[10 the− 3time, ∞) elapsed since57 (19%) the previous recv(), 68 (26%) as well as the values 57 (35%) returned by other system calls, such(10−4 ,as 10 −getsockopt()3) 111and (37%) connect(). Finally, 109 (42%) the dataset EVLT_SET 111 (44%) for evaluating −4 IM‐L‐Rex is shown[0, 10 in Table) 3. As we132 can (44%) see, three sizes 83 of (32%) file transfers (1 132Mbytes, (21%) 10 Mbytes, and 100 Mbytes) were chosen to evaluate the performance of IM‐L‐Rex when facing short and long TCP flows.Figure Moreover,7 shows for the rigorous empirical evaluation, cumulative the practically distribution relevant function loss (CDF) rate parameter of the relative space errors was also of L-Rexdivided and into IM-L-Rex, three intervals for different ([0, 10 size−4), files (10− and4, 10 RPLR−3), and ranges. (10−3, ∞ From)) to Figure compare7, we the can performance see that L-Rex of isIM more‐L‐Rex accurate under underdifferent low loss RPLR rates. than In itTable is under 3, the high first RPLR. row is This the can information be attributed of TCP to the flows fact with that L-Rexloss rate was greater designed than and 10−3, optimized totaling 195 for TCP landline transfers, network while with the moderatesecond row losses is the (i.e., information RPLR ≥ 10of −the4). Inloss addition, rate between we can 10 see−4 and that, 10 when−3, totaling the RPLR 308 isTCP on thetransfers, same order and the of magnitude, third row is the the estimation results of accuracy the loss ofrate L-Rex lower becomes than 10 lower−4, totaling as the 257 file TCP size transfers. increases. The This first can column be explained is the by TCP the transfers fact that with L-Rex file takes size the of time1 MB, required and the for numbers the connect() of such system TCP transfers call to complete corresponding as RTT to, while each thelossRTT rateat interval the TCP are connection 57 (19%), establishment111 (37%), and is132 usually (44%), inconsistentrespectively, with while that the aftersecond the column transmission is the TCP is stable. transfers It canwith be file inferred size of that,10 MB, when and the the filenumbers is large, of thesuch fluctuation TCP transfers of RTT correspondingis larger, which to each makes loss the rate estimation interval are accuracy 68 (36%), of L-Rex109 (42%), face moreand 83 severe (32%), challenges. respectively, On and the contrary,the third benefitingcolumn is fromthe TCP dynamic transfers parameters, with file insize Figure of 1007, IM-L-RexMB, and the is more numbers accurate of such than TCP L-Rex transfers and is corresponding basically not affected to each by loss packet rate interval loss rate are and 70 file (35%), size. 88 (44%), and 42 (21%), respectively. Figure 7. Cumulative distribution function (CDF) of the relative errors of L-Rex and IM-L-Rex. L-Rex is designed and optimized based on fast Ethernet and ADSL. As we know, ADSL is rarely used nowadays. Correspondingly, the experimental results in Figure7 are all derived from the TCP Symmetry 2019, 11 FOR PEER REVIEW 13 of 17 Figure 7. Cumulative distribution function (CDF) of the relative errors of L-Rex and IM-L-Rex. Figure 7 shows the empirical cumulative distribution function (CDF) of the relative errors of L-Rex and IM-L-Rex, for different size files and RPLR ranges. From Figure 7, we can see that L-Rex is more accurate under low RPLR than it is under high RPLR. This can be attributed to the fact that L-Rex was designed and optimized for landline network with moderate losses (i.e., RPLR 10−4). In addition, we can see that, when the RPLR is on the same order of magnitude, the estimation accuracy of L-Rex becomes lower as the file size increases. This can be explained by the fact that L-Rex takes the time required for the connect() system call to complete as RTT, while the RTT at the TCP connection establishment is usually inconsistent with that after the transmission is stable. It can be inferred that, when the file is large, the fluctuation of RTT is larger, which makes the estimation accuracy of L-Rex face more severe challenges. On the contrary, benefiting from dynamic parameters, in Figure 7, IM-L-Rex is more accurate than L-Rex and is basically not affected by packet Symmetry 2019, 11, 442 13 of 17 loss rate and file size. L-Rex is designed and optimized based on fast Ethernet and ADSL. As we know, ADSL is rarelytransfers used whose nowadays. clients areCorrespondingly, located in fast Ethernet.the experimental To verify results the performance in Figure 7 ofare IM-L-Rex all derived in otherfrom tcurrenthe TCP mainstreamtransfers whose access clients networks, are located we also in fast compared Ethernet L-Rex. To verify and IM-L-Rexthe performance in gigabit of IM Ethernet.-L-Rex inFor other the sake current of highlighting mainstream the access advantages networks, of L-Rex, we also we comparedchose TCP Ltransfers-Rex and with IM a-L file-Rex size in ofgigabit 1 MB Ethernetand a packet. For lossthe s rateake greaterof highlighting than 10− the3, for advantages experimental of L comparison.-Rex, we chose From TCP the transfers result inwith Figure a file8, wesize can of 1 see MB that and the a packet accuracy loss of rate L-Rex greater in gigabit than 10 Ethernet−3, for experimental is worse than com thatparison. in fast Ethernet. From the The result error in Figuremainly 8 stems, we can from see the that limitations the accuracy of the of L static-Rex parametersin gigabit Ethernet optimized is worse in ADSL than and that fast in fast Ethernet, Ethernet i.e.,. Thein gigabit error Ethernet;mainly stems the differencefrom the inlimitations the number of the of bytesstatic receivedparameters by eachoptimized recv() isin higherADSL thanand thatfast Ethernetassumed, byi.e. L-Rex., in gigabit In this Ethernet case, the; the parameter difference values in the of 1%number and 0.7of maybytes be received too strict. byIn each comparison, recv() is higherin Figure than8, thethat accuracyassumed ofby IM-L-RexL-Rex. In this based case, on the dynamic parameter parameters values of is 1% comparable and 0.7 may to be that too in strict. fast InEthernet comparison, and is in not Figure affected 8, the by theaccuracy specific of levelIM-L- 2Rex connection based on type. dynamic parameters is comparable to that in fast Ethernet and is not affected by the specific level 2 connection type. Figure 8. Result of g gigabitigabit Ethernet Ethernet.. 5.2. Testbed 5.2. Testbed 5.2.1. Impact of Packet Reordering eth0 eth1 In this subsection, we use testbed measurements to investigate the performance of IM-L-Rex in processing packet reordering. The testbed shownRouter R in Figure9 consisted of three fedora8 virtual Host A Host B machines installed on the fedora17 operating system: host A, host B, and router R. We used GNU Zebra to emulate and configure the router,Figure while 9. Testbed the packet scenario reordering. and random packet losses are added to network interface eth1 with Netem. Corresponding to the scenario in Figure9, the specific testbed parameters are shown in TableTable4. In 4. the Testbed previous parameters experiments,. we verified the performance of IM-L-Rex underID fastParameter Ethernet and gigabit Ethernet,Parameter while description here the link bandwidth is set to 10 Mbps to further verify1 the robustnessProtocol of IM-L-Rex underNewReno lower bandwidth. Also, the 10-second random-data TCP download2 is consistentProportion withdelay the fact that10 ms most application level tools run their tests for a fixed number of seconds3 File (typically size 10) [7]. 10-second random-data TCP download Table 4. Testbed parameters. ID Parameter Parameter Description 1 Protocol NewReno 2 Proportion delay 10 ms 3 File size 10-second random-data TCP download 4 Link bandwidth 10 Mbps 5 Packet size uniformly distributed between 0.2 and 1 Kbytes 6 Loss rate gradually from 0 to 5% 7 Reordering proportion 5% and 10% Symmetry 2019, 11 FOR PEER REVIEW 13 of 17 Figure 7. Cumulative distribution function (CDF) of the relative errors of L-Rex and IM-L-Rex. Figure 7 shows the empirical cumulative distribution function (CDF) of the relative errors of L-Rex and IM-L-Rex, for different size files and RPLR ranges. From Figure 7, we can see that L-Rex is more accurate under low RPLR than it is under high RPLR. This can be attributed to the fact that L-Rex was designed and optimized for landline network with moderate losses (i.e., RPLR 10−4). In addition, we can see that, when the RPLR is on the same order of magnitude, the estimation accuracy of L-Rex becomes lower as the file size increases. This can be explained by the fact that L-Rex takes the time required for the connect() system call to complete as RTT, while the RTT at the TCP connection establishment is usually inconsistent with that after the transmission is stable. It can be inferred that, when the file is large, the fluctuation of RTT is larger, which makes the estimation accuracy of L-Rex face more severe challenges. On the contrary, benefiting from dynamic parameters, in Figure 7, IM-L-Rex is more accurate than L-Rex and is basically not affected by packet loss rate and file size. L-Rex is designed and optimized based on fast Ethernet and ADSL. As we know, ADSL is rarely used nowadays. Correspondingly, the experimental results in Figure 7 are all derived from the TCP transfers whose clients are located in fast Ethernet. To verify the performance of IM-L-Rex in other current mainstream access networks, we also compared L-Rex and IM-L-Rex in gigabit Ethernet. For the sake of highlighting the advantages of L-Rex, we chose TCP transfers with a file size of 1 MB and a packet loss rate greater than 10−3, for experimental comparison. From the result in Figure 8, we can see that the accuracy of L-Rex in gigabit Ethernet is worse than that in fast Ethernet. The error mainly stems from the limitations of the static parameters optimized in ADSL and fast Ethernet, i.e., in gigabit Ethernet; the difference in the number of bytes received by each recv() is higher than that assumed by L-Rex. In this case, the parameter values of 1% and 0.7 may be too strict. In comparison, in Figure 8, the accuracy of IM-L-Rex based on dynamic parameters is comparable to that in fast Ethernet and is not affected by the specific level 2 connection type. Figure 8. Result of gigabit Ethernet. Symmetry 2019, 11, 442 14 of 17 5.2. Testbed eth0 eth1 Router R Host A Host B Figure 9. Testbed scenario.scenario. The simulation results of L-Rex andTable IM-L-Rex 4. Testbed under parameters different. reordering rate (the proportion of data packets reordered during a TCP transfer) are shown in Figure 10. The application-level packet loss rate, inID the actualParameter network, typicallyParameter spans several description orders of magnitude. Accordingly, in the final experiment1 results,Protocol five to 10 TCP transferNewReno samples were selected for each order of magnitude Symmetry 2019, 11 FOR PEER REVIEW 14 of 17 to investigate2 the effectivenessProportion delay of our strategy10 msused for excluding packet reordering. In Figure 10, 3 File size 10-second random-data TCP download the dashed bisector4 Link line bandwidth represents the exact10 prediction,Mbps and the closer the points are to the bisection, the more accurate the corresponding model is. From Figure 10, we can see that, on the whole, the 5 Packet size uniformly distributed between 0.2 and 1 Kbytes points of L-Rex6 areLoss scattered rate on both sidesgradually of the bisection, from 0 to while 5% the points of IM-L-Rex are mainly located above7 the bisectionReordering and proportion are closer to5% the and bisection 10% than L-Rex. Moreover, we can see that, as the proportion of packets that suffer reordering increases, L-Rex becomes less accurate, while IM-L-Rex is 5.2.1.relatively Impact accurate. of Packet All R thiseordering indicates that our strategy is effective; in other words, IM-L-Rex is more robust than L-Rex in the face of packet reordering. Meanwhile, the points scattered on both sides of the bisectionIn this imply subsection, that L-Rex we sometimesuse testbed overestimates measurements and to sometimesinvestigate underestimates,the performance while of IM the-L-Rex points in processingabove the bisection packet indicatereordering. that The IM-L-Rex testbed will shown produce in overestimatesFigure 9 consisted in most of cases. three The fedora8 phenomenon virtual machinesbehind L-Rex installed further on illustrates the fedora17 that theoperating static parameters system: host of L-RexA, host are B, unable and router to adapt R. We to the used dynamic GNU Zebraand complex to emulate network and co conditions.nfigure the As router, for the while reason the whypacket IM-L-Rex reordering always and outputsrandom overestimatespacket losses areis that added IM-L-Rex to network also utilizes interface retransmissions eth1 with Netem. to realize Corre losssponding estimation; to the although scenario the in estimationFigure 9, the of specificIM-L-Rex testbed is closer parameters to the actual are number shown in of retransmissions,Table 4. In the previous the spurious experiments, retransmissions we verifi includeded the performancein retransmissions of IM- inevitablyL-Rex under skewed fast Ethernet the estimation and gigabit of IM-L-Rex Ethernet, andwhile produced here the thelink overestimates. bandwidth is setMoreover, to 10 Mbps the results to further in Figure verify 10 also the show robustness that L-Rex of IM performs-L-Rex under poorly lower at low bandwidth. RPLR, which Also, may the be 10owing-second to the random static- parameterdata TCP download values (i.e., is 1% consistent for H and with 0.7 fortheK fact) optimized that most under application moderate level losses. tools run their tests for a fixed number of seconds (typically 10) [7]. Figure 10. TheThe performance performance of IM IM-L-Rex-L-Rex and L L-Rex-Rex in processing packet reordering reordering.. 5.2.2. Impact of TCP Variants The simulation results of L-Rex and IM-L-Rex under different reordering rate (the proportion of data Inpackets order reordered to check how during TCP a variant TCP transfer influences) are theshown proposal, in Figure we carried10. The out application experiments-level with packet the losssame rate, set-up in the shown actual in network, Figure9 totypically compare spans the performanceseveral orders of of IM-L-Rex magnitude. and Accordingly, L-Rex under in different the final TCPexperim variantsent results, (i.e., Reno, five to NewReno, 10 TCP transfer and SACK). samples Unlike were the selected parameters for each in Table order1, of we magnitude did not add to packetinvestigate reordering the effectiveness to the network of our interface strategy to used focus for on excl the effectuding ofpacket the TCP reordering. variant itself. In Figure In Figure 10, the 11, dashedthe experiment bisector resultsline represents can be seen. the exact It can prediction, be seen that, and regardless the closer ofthe IM-L-Rex points are or to L-Rex, the bisection, in terms the of moreoverall accurate estimation the c accuracy,orresponding SACK model is best, is. From NewReno Figure is 10 second,, we can and see Reno that, is on the the worst. whole, Particularly, the points weof L can-Rex see are that, scattered although on both IM-L-Rex sides outperformsof the bisection, L-Rex while under the thepoints same of TCPIM-L variant,-Rex are IM-L-Rexmainly located is not abovealways the superior bisection to L-Rex and are in the closer entire to parameter the bisection space. than That L-Rex. is, the Moreover, L-Rex under we canSACK see is that better, as than the proportion of packets that suffer reordering increases, L-Rex becomes less accurate, while IM-L-Rex is relatively accurate. All this indicates that our strategy is effective; in other words, IM-L-Rex is more robust than L-Rex in the face of packet reordering. Meanwhile, the points scattered on both sides of the bisection imply that L-Rex sometimes overestimates and sometimes underestimates, while the points above the bisection indicate that IM-L-Rex will produce overestimates in most cases. The phenomenon behind L-Rex further illustrates that the static parameters of L-Rex are unable to adapt to the dynamic and complex network conditions. As for the reason why IM-L-Rex always outputs overestimates is that IM-L-Rex also utilizes retransmissions to realize loss estimation; although the estimation of IM-L-Rex is closer to the actual number of retransmissions, the spurious retransmissions included in retransmissions inevitably skewed the estimation of IM-L-Rex and produced the overestimates. Moreover, the results in Figure 10 also show that L-Rex performs poorly at low RPLR, which may be owing to the static parameter values (i.e., 1% for H and 0.7 for K) optimized under moderate losses. Symmetry 2019, 11, 442 15 of 17 Symmetry 2019, 11 FOR PEER REVIEW 15 of 17 theSymmetry IM-L-Rex 2019 under, 11 FOR RenoPEER REVIEW when the relative error is greater than 66.43%. From these, it is inferred15 of 17 that different5.2.2. Impact TCP variants of TCP V haveariants a great impact on the performance of IM-L-Rex. 5.2.2. Impact of TCP Variants Figure 11. Impact of TCP variants. FigureFigure 11.11. ImpactImpact of TCP TCP variants variants.. In order to check how TCP variant influences the proposal, we carried out experiments with theExploring sameIn order set- up theto check shown reasons how in behind FigureTCP variant this 9 to phenomenon, compareinfluences the the performance weproposal, found we that of carried IM it- isL- Rex relatedout andexperiments to L- theRex different under with mechanismsdifferentthe same TCP set via- upvariants which shown different (i.e., in Reno,Figure TCP NewReno, 9 variants to compare and handle SACK). the packet performance Unlike losses. the Specifically,of parameters IM-L-Rex NewRenoin and Table L-Rex 1, andwe under did Reno havendifferentot the add same packet TCP mechanism variants reordering (i.e., in to dealingReno, the network NewReno, with a interface single and packetSACK). to focus loss. Unlike on However, the the effect parameters whenof the losingTCP in variantTable multiple 1 ,itself. we packets did In at aFigurenot time, add 11, NewRenopacket the experiment reordering shows result itsto the advantagess network can be seen. interface by reducingIt can to be focus seen the on occurrence that the, regardlesseffect of the overtime of TCP IM- Lvariant- retransmissionsRex or itself. L-Rex, In withinFigure theterms improved11, of the overall experiment fast-recovery estimation result accuracy, algorithm.s can be SACKseen. Likewise, It is can bes comparedbet, NewReno seen that with, isregardless second, NewReno, and of IM SACKReno-L-Rex is utilizes the or Lworst.-Rex, SACK Particularly, we can see that, although IM-L-Rex outperforms L-Rex under the same TCP variant, blocksin terms to reduce of overall spurious estimation retransmissions, accuracy, SACK thus showingis best, NewReno its advantages is second, in dealing and Reno with is packet the worst. losses. IM-L-Rex is not always superior to L-Rex in the entire parameter space. That is, the L-Rex under Furthermore,Particularly, since we can IM-L-Rex see that and, although L-Rex are IM based-L-Rex on outperforms retransmissions, L-Rex theunder relative the same advantages TCP variant, between SACKIM-L-Rex is better is not than always the IMsuperior-L-Rex tounder L-Rex Reno in the when entire the parameterrelative error space. is greater That is, than the 66.43%. L-Rex underFrom different TCP variants ultimately lead to their different performance. these,SACK itis is better inferred than that the different IM-L-Rex TCP under variants Reno have when a greatthe relative impact error on the is performancegreater than 66.43%.of IM-L -FromRex. 5.2.3.these, ImpactExploring it is inferred of Parameters the that reasons differentRTT behindand TCP K this variants phenomenon, have a great we impact found thaton the it performance is related to ofthe IM different-L-Rex. mechanismsExploring via the which reasons different behind TCP this variants phenomenon, handle we packet found losses. that it Specifically, is related to NewReno the different and RenomechanismsTo fully have evaluate thevia samewhich IM-L-Rex, mechanism different we TCP inalso dealing variants investigated with handle a the single packet effects packet losses. of dynamic loss. Specifically, However,RTT and NewReno when dynamic losing andK on IM-L-Rex.multipleReno have Thepackets the evaluation sameat a time, mechanism data NewReno come in from shows dealing the its TCP withadvantages transfers a single by under packetreducing NewReno loss. the However,occurrence version inwhen of Section overtime losing 5.2.2 , andretransmissionsmultiple the experiment packets withat results a time, the are improved NewReno shown fast inshows Figure-recovery its 12advantages. algorithm. Comparing by Likewise, r L-Rexeducing and comparedthe “IM-L-Rex, occurrence with Dynamicof NewReno, overtimeRTT , StaticSACKretransmissionsK”, utilizes we can SACKsee with that theblocks the improved dynamic to reduce fastRTT spurious-recoveryhas a certain retransmissions, algorithm. improvement Likewise, thus for showing compared the estimation its with advantages NewReno, accuracy in of L-Rex.dealingSACK Similarly, utilizes with packet the SACK comparison losses. blocks Furtherm to of reduce “IM-L-Rex,ore, spurious since Dynamic IM retransmissions,-L-RexRTT and, StaticL-Rex thusK are” and showingbased L-Rex on its alsoretransmissions, advantages illustrates in that thethedealing dynamic relative withK haspacket advantages a certain losses. improvementbetween Furtherm differentore, forsince the TCPIM estimation- L- variantsRex and accuracy L ultimately-Rex are of L-Rex,based lead onbut to retransmissions, their its improvement different effectperformance.the is relative less than advantages that of dynamic betweenRTT different. This indicates TCP variants that, for ultimately IM-L-Rex, leadRTT to is their a more different critical parameterperformance. than K, which has the more important role in the estimation accuracy. 5.2.3. Impact of Parameters RTT and K 5.2.3. Impact of Parameters RTT and K FigureFigure 12. 12.Impact Impact of of parameters parameters RTTRTT andand K (“IM-L-Rex,(“IM-L-Rex, Dynamic Dynamic RTTRTT, , Static Static K”K ”represents represents the the IM-L-RexIMFigure-L-Rex 12. taking taking Impact dynamic dynamic of parametersRTT RTTand and RTT static static and KK,, whilewhileK (“IM “IM-L-Rex,“IM-L-Rex,L-Rex, Dynamic Static Static RTTRTT RTT, ,Dynamic, Dynamic Static K K””K representsrepresents” represents the the IM-L-RexIM-L-Rex taking taking static staticdynamicRTT RTT andRTT and dynamicand dynamic staticK K).), .while “IM-L-Rex, Static RTT, Dynamic K” represents the IM-L-Rex taking static RTT and dynamic K). Symmetry 2019, 11, 442 16 of 17 Nonetheless, it is worth pointing out that, although IM-L-Rex is superior to L-Rex, there is still a gap between the estimation accuracy and the practical application requirement [29]. From our experiments, we can see that some relative errors are even higher than 100% for both IM-L-Rex and L-Rex. Upon tracing its root, for IM-L-Rex, the estimation errors mainly stem from the inevitably spurious retransmissions associated with the flaws in TCP’s retransmission schemes. Accordingly, for our future work, further excluding the effect of spurious retransmissions is necessary and crucial to eliminate the gap. 6. Conclusion and Future Work Faced with the packet dynamics along the network path, dynamic parameters are beneficial to improve the accuracy and robustness of the model L-Rex. To this end, we leveraged the TCP’s self-clocking mechanism to dynamically estimate the parameters required by L-Rex. Meanwhile, the effect of packet reordering was also considered by refining fast retransmissions and overtime retransmissions. Correspondingly, the experiments performed with real measured data and testbed measurements showed that the approach and strategy proposed in this paper are effective. Yet, from the experimental results, it can be also seen that, although IM-L-Rex is superior to L-Rex in terms of estimation accuracy and stability, there is still a gap between the estimation accuracy and the practical application requirement. With our analysis, spurious retransmission is the root of the problem. Also, minimally, PLR would vary randomly in real network; thus, coping with this problem will be beneficial to improve the robustness of IM-L-Rex. To this end, for our future work, we intend to further mine the information available from the application layer to exclude these adverse effects. Moreover, since wireless access networks are now very popular and have noticeable effects on packet losses and RTT, testing IM-L-Rex in wireless networks will be interesting. Author Contributions: Conceptualization, H.L. and W.D.; data curation, H.L.; investigation, H.L.; methodology, H.L.; resources, H.L.; software, H.L. and L.D.; validation, H.L.; visualization, H.L.; writing—original draft, H.L.; writing—review and editing, W.D.; project administration, W.D. Funding: This research was funded by the National Nature Science Foundation of China (No. 61602114), the National Nature Science Foundation of China (No. 60973123), and CERNET Innovation Project (No. NGII20170406). Conflicts of Interest: The authors declare no conflict of interest. References 1. Antonopoulos, A.; Kartsakli, E.; Perillo, C.; Verikoukis, C. Shedding light on the Internet: Stakeholders and network neutrality. IEEE Commun. Mag. 2017, 55, 216–223. [CrossRef] 2. Peitz, M.; Schuett, F. Net neutrality and inflation of traffic. Int. J. Ind. Organ. 2016, 46, 16–62. [CrossRef] 3. Bauer, J.M.; Knieps, G. Complementary innovation and network neutrality. Telecommun. Policy 2018, 42, 172–183. [CrossRef] 4. Dustdar, S.; Duarte, E.P. Network Neutrality and Its Impact on Innovation. IEEE Internet Comput. 2018, 22, 5–7. [CrossRef] 5. Schulzrinne, H. Network Neutrality Is About Money, Not Packets. IEEE Internet Comput. 2018, 22, 8–17. [CrossRef] 6. Statovci-Halimi, B.; Franzl, G. QoS differentiation and Internet neutrality. Telecommun. Syst. 2013, 52, 1605–1614. [CrossRef] 7. Basso, S.; Meo, M.; Servetti, A.; De Martin, J.C. Estimating packet loss rate in the access through application-level measurements. In Proceedings of the ACM SIGCOMM workshop on Measurements up the stack, Helsinki, Finland, 17 August 2012; pp. 7–12. 8. Basso, S.; Meo, M.; De Martin, J.C. Strengthening measurements from the edges: Application-level packet loss rate estimation. ACM SIGCOMM Comput. Commun. Rev. 2013, 43, 45–51. [CrossRef] 9. Ellis, M.; Pezaros, D.P.; Kypraios, T.; Perkins, C. A two-level Markova model for packet loss in UDP/IP-based real-time video applications targeting residential users. Comput. Netw. 2014, 70, 384–399. [CrossRef] Symmetry 2019, 11, 442 17 of 17 10. Baglietto, M.; Battistelli, G.; Tesi, P. Packet loss detection in networked control systems via process measurements. In Proceedings of the IEEE Conference on Decision and Control, Miami Beach, FL, USA, 17–19 December 2018; pp. 4849–4854. 11. Mittag, G.; Möller, S. Single-ended packet loss rate estimation of transmitted speech signals. In Proceedings of the IEEE Global Conference on Signal and Information Processing, Anaheim, CA, USA, 26–29 November 2018; pp. 226–230. 12. Lan, H.; Ding, W.; Gong, J. Useful Traffic Loss Rate Estimation Based on Network Layer Measurement. IEEE Access 2019, 7, 33289–33303. [CrossRef] 13. Carra, D.; Avrachenkov, K.; Alouf, S.; Blanc, A.; Nain, P.; Post, G. Passive online RTT estimation for flow-aware routers using one-way traffic. In Proceedings of the 9th International IFIP TC 6 Networking Conference, Chennai, India, 11–15 May 2010; pp. 109–121. 14. Allman, M.; Eddy, W.M.; Ostermann, S. Estimating loss rates with TCP. ACM SIGMETRICS Perform. val. Rev. 2003, 31, 12–24. [CrossRef] 15. Priya, S.; Murugan, K. Improving TCP performance in wireless networks by detection and avoidance of spurious retransmission timeouts. J. Inf. Sci. Eng. 2015, 31, 711–726. 16. Anelli, P.; Lochin, E.; Harivelo, F.; Lopez, D.M. Transport congestion events detection (TCED): Towards decorrelating congestion detection from TCP. In Proceedings of the ACM Symposium on Applied Computing, Sierre, Switzerland, 22–26 March 2010; pp. 663–669. 17. Zhani, M.F.; Elbiaze, H.; Aly, W.H.F. TCP based estimation method for loss control in OBS networks. In Proceedings of the IEEE GLOBECOM, Honolulu, HI, USA, 30 November–4 December 2009; pp. 1–6. 18. Ullah, S.; Ullah, I.; Qureshi, H.K.; Haw, R.; Jang, S.; Hong, C.S. Passive packet loss detection in Wi-Fi networks and its effect on HTTP traffic characteristics. In Proceedings of the International Conference on Information Networking, Phuket, Thailand, 10–12 February 2014; pp. 428–432. 19. Hagos, D.H.; Engelstad, P.E.; Yazidi, A.; Kure, Ø. A machine learning approach to TCP state monitoring from passive measurements. In Proceedings of the IEEE Wireless Days, Dubai, UAE, 3–5 April 2018; pp. 164–171. 20. Laghari, A.A.; He, H.; Channa, M.I. Measuring effect of packet reordering on quality of experience (QoE) in video streaming. 3D Research 2018, 9, 30. [CrossRef] 21. Veal, B.; Li, K.; Lowenthal, D. New methods for passive estimation of TCP round-trip times. In International Workshop on Passive and Active Network Measurement; Springer: Berlin/Heidelberg, Germany, 2005; pp. 121–134. 22. Mirkovic, D.; Armitage, G.; Branch, P. A survey of round trip time prediction systems. IEEE Commun. Surv. Tutor. 2015, 20, 1758–1776. [CrossRef] 23. Paxson, V.; Allman, M.; Chu, J.; Sargent, M. Computing TCP’s retransmission timer. RFC 6298. 2011. [CrossRef] 24. Allman, M.; Paxson, V.; Stevens, W. TCP Congestion Control, RFC 2581. 1999. [CrossRef] 25. Floyd, S.; Henderson, T. The NewReno Modification to TCP's Fast Recovery Algorithm, RFC 2582. 1999. [CrossRef] 26. Mathis, M.; Mahdavi, J.; Floyd, S.; Romanow, A. TCP Selective Acknowledgment Options, RFC 2018. [CrossRef] 27. Lan, H.; Ding, W.; Zhang, Y. Passive overall packet loss estimation at the border of an ISP. KSII T. Internet. Inf. 2018, 12, 3150–3171. 28. Stevens, W.R. TCP/IP Illustrated, Volume 1: The protocols, 1st ed.; Addison-Wesley Publisher: Boston, MA, USA, 1994; pp. 136–139. 29. Nguyen, H.X.; Roughan, M. Rigorous statistical analysis of internet loss measurements. IEEE/ACM Trans. Netw. 2013, 21, 734–745. [CrossRef] © 2019 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).