A Modified Fast-Vegas-LIA Hybrid Congestion Control Algorithm For

electronics

Article MFVL HCCA: A Modiﬁed Fast-Vegas-LIA Hybrid Congestion Control Algorithm for MPTCP Trafﬁc Flows in Multihomed Smart Gas IoT Networks

Mumajjed Ul Mudassir 1,2,* and M. Iram Baig 2

1 Electrical and Computer Engineering Department, Air University, Islamabad 44000, Pakistan 2 Electrical Engineering Department, University of Engineering and Technology, Taxila 47050, Pakistan; [email protected] * Correspondence: [email protected]

Abstract: Multihomed smart gas meters are Internet of Things (IoT) devices that transmit information wirelessly to a cloud or remote database via multiple network paths. The information is utilized by the smart gas grid for accurate load forecasting and several other important tasks. With the rapid growth in such smart IoT networks and data rates, reliable transport layer protocols with efficient congestion control algorithms are required. The small Transmission Control Protocol/Internet Protocol (TCP/IP) stacks designed for IoT devices still lack efficient congestion control schemes. Multipath transmission control protocol (MPTCP) based congestion control algorithms are among the recent research topics. Many coupled and uncoupled congestion control algorithms have been proposed by researchers. The default congestion control algorithm for MPTCP is coupled congestion control by using the linked- increases algorithm (LIA). In battery powered smart meters, packet retransmissions consume extra power and low goodput results in poor system performance. In this study, we propose a modified Citation: Mudassir, M.U.; Baig, M.I. Fast-Vegas-LIA hybrid congestion control algorithm (MFVL HCCA) for MPTCP by considering the MFVL HCCA: A Modified requirements of a smart gas grid. Our novel algorithm operates in uncoupled congestion control Fast-Vegas-LIA Hybrid Congestion mode as long as there is no shared bottleneck and switches to coupled congestion control mode Control Algorithm for MPTCP Traffic otherwise. We have presented the details of our proposed model and compared the simulation results Flows in Multihomed Smart Gas IoT Networks. Electronics 2021, 10, 711. with the default coupled congestion control for MPTCP. Our proposed algorithm in uncoupled mode https://doi.org/10.3390/ shows a decrease in packet loss up to 50% and increase in average goodput up to 30%. electronics10060711 Keywords: multipath TCP; Internet of Things; congestion control; smart gas meter; smart gas network Academic Editor: Seok-Joo Koh

Received: 11 February 2021 Accepted: 10 March 2021 1. Introduction Published: 18 March 2021 The goal of the Internet of Things (IoT) is to connect different devices and sensors to the Internet. According to a prediction by the Statista research department, the number Publisher’s Note: MDPI stays neutral of active IoT-connected devices such as sensors, nodes, and gateways will reach up to with regard to jurisdictional claims in 30.9 billion units worldwide by 2025 [1]. Smart city is the new trend of the era and the published maps and institutional affil- IoT is playing a major role in the design and deployment of smart city infrastructure. The iations. design of a smart city is divided into multiple domains, subsystems, and blocks and it is really difficult to implement an efficient design for a smart city by using the IoT. Some of the important domains of a smart city are electricity and natural gas management, water management, irrigation, waste material, parking space, and intelligent street lighting [2,3]. Copyright: © 2021 by the authors. The datasets used in the design and simulation of such domains are publicly available on Licensee MDPI, Basel, Switzerland. the Internet [4,5]. This article is an open access article distributed under the terms and 1.1. Smart Gas Networks conditions of the Creative Commons All over the world, natural gas is being used in homes and industries for heating and Attribution (CC BY) license (https:// fueling purposes. Many research and development organizations are conducting their creativecommons.org/licenses/by/ 4.0/). research in the area of IoT-based smart grids for natural gas [6,7]. Gas utilities gather

Electronics 2021, 10, 711. https://doi.org/10.3390/electronics10060711 https://www.mdpi.com/journal/electronics Electronics 2021, 10, 711 2 of 27

real-time data for load forecasts, efﬁcient gas transportation, pressure gauging, gas theft detection, gas leaks, pipe corrosion, and remote cut off in emergency situations. To achieve this, communication networks with advanced infrastructure are required; smart meters and sensors use these networks for data transmission. All the components are connected to form a smart grid [8]. Once the network is established [9], data can be collected by gas utilities even from old non-smart gas meters by attaching some extra devices and sensors [10]. Then, the data is analyzed with the help of artiﬁcial intelligence software for further actions [11].

1.2. Smart Gas Meters A smart gas meter is an IoT device that consists of a processing unit connected with different sensors, wireless communication modules, real-time clock, power management units, electric motors to control valves, display unit, and a battery [12]. In addition to measuring gas ﬂow, it also wirelessly connects to a smart gas grid over wide area networks allowing data access, remote location monitoring, infrastructure maintenance, automatic billing, and load forecasting [13]. It can request a remote gas cut-off after detecting emergency situations by sensing earthquakes, gas leakage, etc. Batteries are used as the power source for smart meters. Low power modules are preferred in the design of a smart meter to enhance battery life [14]. Multiple wireless interface standards can be used in a smart meter design to enable it to simultaneously connect to multiple heterogeneous or homogeneous networks.

1.3. Selection of Appropriate Protocol in an IoT Network The Internet runs on hundreds of protocols; many protocols are supported by IoT and many are still under development. When designing an IoT system, the system requirements should be deﬁned very precisely, and then the right protocol should be chosen to address them. Currently, due to an increase in memory size and processing power, small, embedded devices and modules are capable of running large programs and algorithms. Development of new wireless communication standards such as IEEE 802.11 ah (Wi-Fi Halow) for IoT have also enabled the devices to communicate at much higher data rates over long distances [15,16]. In any communication network with many devices, network congestion is the main issue that causes poor data rates and packet loss [17]. In the IoT, Transmission Control Protocol (TCP) has traditionally been avoided as a transport-layer protocol due to the extra overhead associated with it. However, recent trends and developments in IoT devices and networks are favoring TCP for congestion control and end-to-end reliable delivery of data [18].

1.4. Multipath Transmission Control Protocol (MPTCP) in Multihomed Devices Modern IoT devices are equipped with multiple network interfaces, capable of simultaneously connecting to multiple network links and different Internet Protocol (IP) addresses. These links can be used for concurrent transfer of data, then if any links fail others can be used for successful data delivery. The multipath transmission control protocol (MPTCP) is embedded in all such modern multihomed devices. Multihoming is deﬁned as the ability of a host or device to simultaneously connect to multiple heterogeneous or homogeneous networks [19]. Multihomed devices with MPTCP protocol support, divide the application’s data into multiple streams, and then utilize multiple network paths simultaneously for data transmission/reception. Load balancing, congestion control, and dynamic switching are handled by the protocol in order the improve throughput and quality of service [20].

1.5. Smart Gas Grid Infrastructure In Figure 1, we present the structure of an IoT-based smart natural gas grid that uses multihomed smart gas meters with dual low power Wi-Fi Halow interfaces capable of Electronics 2021, 10, 711 3 of 27 simultaneously connecting to two gateways of different Internet service providers (ISPs) for parallel transfer of data to server. The information received from these smart meters, and from some other data sources, for example, weather information, is then used by the fromsmart some grid otherfor short-term data sources, load forforecasting example, (STFL). weather Deep information, learning ismethods then used are by used the for smart ac- gridcurate for load short-term forecasts. load Gas forecasting distribution (STFL). management Deep learning makes methods decisions are usedaccording for accurate to the loadforecasts forecasts. for intelligent Gas distribution distribution management of gas to different makes decisions areas. In accordingsuch IoT networks to the forecasts where forend-to-end intelligent reliable distribution transmission of gas toof differentdata from areas. smart In meters such IoT to networksa server is where required, end-to-end TCP is reliablealways preferred transmission over ofUser data Datagram from smart Prot metersocol (UDP) to a and server MPTCP is required, is used TCPin multihomed is always preferreddevices. over User Datagram Protocol (UDP) and MPTCP is used in multihomed devices.

FigureFigure 1.1. AnAn InternetInternet ofof ThingsThings (IoT)-based(IoT)-based smartsmart gasgas network.network.

1.6. Problem Analysis With thethe growinggrowing numbernumber ofof devicesdevices inin IoTIoT networks,networks, thethe networknetwork congestioncongestion alsoalso increases.increases. For For smooth data transfer, a reliablereliable transporttransport layerlayer protocolprotocol withwith anan efficientefficient congestion controlcontrol algorithmalgorithm isis required.required. TheThe InternetInternet ofof ThingsThings uses uses small small TCP/IP TCP/IP stacksstacks withwith veryvery limitedlimited capabilities.capabilities. Many vulnerabilitiesvulnerabilities and flawsflaws have beenbeen foundfound inin suchsuch stacks. Growing technological developments are producing powerfulpowerful smallsmall devicesdevices andand largelarge IoTIoT networks networks with with high high data data rates, rates, and an therefored therefore network network congestion congestion is a critical is a critical issue. MPTCPissue. MPTCP is a protocol is a protocol used for used multipath for multipat data transfer.h data Many transfer. congestion Many congestion control algorithms control havealgorithms been proposed have been by proposed different by researchers different researchers for MPTCP for and MPTCP the performance and the performance evaluation ofevaluation these algorithms of these algorithms is still under is still debate. under There debate. is aThere lot of is research a lot of research going on, going to design on, to newdesign congestion new congestion control algorithmscontrol algorithms for MPTCP. for TheMPTCP. default The coupled default congestion coupled congestion control by usingcontrol linked-increases by using linked-increases algorithm (LIA)algorithm of MPTCP (LIA) ensuresof MPTCP fairness ensures in thefairness case ofin athe shared case bottleneckof a shared but bottleneck suffers frombut suffers low throughput from low otherwise,throughput due otherwise, to its coupled due to architecture. its coupled In the case of a smart gas grid, the goal of a congestion control algorithm is to increase architecture. In the case of a smart gas grid, the goal of a congestion control algorithm is goodput for timely transmission of important data and decrease packet retransmissions to increase goodput for timely transmission of important data and decrease packet re- to save power. To fulfill this requirement, we have proposed a novel congestion control transmissions to save power. To fulfill this requirement, we have proposed a novel con- algorithm for MPTCP and compared our results with the default coupled congestion control algorithm of MPTCP.

1.7. Contribution

Our main contribution in this research is the design of a hybrid congestion control algorithm for MPTCP. The default MPTCP scheduler is used that distributes packets among subﬂows by observing round trip time delays and congestion windows. As soon as there is Electronics 2021, 10, 711 4 of 27

a space available in the queue of a subflow, packets are injected into the pipe. The proposed modified Fast-Vegas-LIA hybrid congestion control algorithm (MFVL HCCA) is based on a modified Fast TCP, modified TCP Vegas, and LIA congestion control algorithms. It uses a shared bottleneck detection method and works in uncoupled mode or coupled mode accordingly. We simulated our design in Network Simulator 2.34 (NS 2.34) and compared the results with the default coupled congestion control LIA of MPTCP. The remainder of this paper is organized as follows: In Section2, we give an overview of background and related research work; in Section3, we describe the details of the proposed model; the results and discussions are presented in Section4; and our conclusions are atated in Section5.

2. Background and Related Work Many protocols for the IoT are available and research is still going on to discover new protocols for this area. Researchers use different protocol stacks after carefully observing the requirements of an IoT system to fulfill its needs [21]. Some of the Application Layer protocols for IoT devices that require TCP at the transport layer for reliable communication are Extensible Messaging and Presence Protocol (XMPP), Message Queuing Telemetry Transport (MQTT) Protocol, and Advanced Message Queuing Protocol (AMQP). The MQTT Protocol is widely used for data transmission between devices and servers. However, any other suitable protocol can also be used [22]. For memory constrained devices, some TCP/IP protocol stacks with limited functionality are available. These stacks are open source and many embedded system developers and programmers from all around the world are making improvements in the codes and functionalities [23]. Millions of IoT devices are using these stacks. According to a recent report by Forescout Research Labs, 33 vulnerabilities have been found in uIP, FNET, pi- coTCP, and Nut/Net stacks. An attacker can exploit these flaws to take full control of the device, execute code remotely, and steal data [24]. Therefore, these tiny open-source protocol stacks are no longer trustworthy. With the increase in processing speed, battery capacity and memory size of IoT devices, now, more advanced algorithms and standard protocols are required to fulfill the requirements of IoT systems. By 2025, billions of IoT devices will be connected with the Internet, sending tens of zettabytes of data [1]. Some of the most commonly used physical layer protocols for IoT are 802.15.4, 802.11 (Wi-Fi), Bluetooth Low Energy, and ZigBee Smart. A new low power and long range Wi-Fi standard 802.11 ah with the name “Wi-Fi HaLow” has been developed for IoT devices [15]. It has a throughput range from hundreds of kilobits per second (kbps) to tens of megabits per second (Mbps) [25]. Table1 shows the characteristics of different physical layer standards for IoT [26].

Table 1. Characteristics of different physical layer standards for IoT.

Wi-Fi HaLow LoRaWan Sigfox NB-IoT Bluetooth Z-Wave Zigbee Idle power Low Low Low Low Low Low Low consumption Data rate 150 k–86.7 MBps <50 kbps 100–600 bps 20–25 kbps 125 k–2 Mbps 100 kbps 250 kbps Range 1 km 10 km 10 km 10 km <100 m 30 m 20 m Security Best Poor Poor Good Good Good Good Network data Best Poor Worst Poor Good Good Poor capacity Installation cost Low High High High High Med Med

IoT devices with multiple interfaces are multihomed devices. These devices can connect to different networks for concurrent transfer of data to a host/server [27]. As the number of devices and data rate increases, network congestion also increases. Therefore, a transport layer protocol with an efﬁcient congestion control algorithm is also required for IoT networks. The small TCP/IP stacks used in IoT devices are lacking such algorithms. Electronics 2021, 10, 711 5 of 27

In multihomed devices, the MPTCP is used at the transport layer. The MPTCP transmits data over multiple paths by using subflows. The aim of this scheme is to increase throughput and robustness [28]. One common application of multipath communication in mobile phones is to use Wi-Fi and 3G paths simultaneously to transfer data in parallel, so that if any network path fails then another could be used for data transfer [29]. However, an IoT device can also have multiple interfaces of same standard available, for example, Wi-Fi to connect to multiple Wi-Fi networks of different ISPs for concurrent transfer of data. The MPTCP also performs load balancing among different paths. In the MPTCP, separate congestion windows are maintained on each path; however, in order to prevent harm to fairness, especially in case of a shared bottleneck link, execution of congestion control independently on each path is avoided by most algorithms. Many congestion control algorithms have been proposed to couple together all the sublows of a single multipath flow in order to achieve fairness and efficiency [30]. A congestion control algorithm can be delay based, loss based, or hybrid. Most of the congestion control algorithms for the MPTCP are loss based. To ensure fairness and better performance over the Internet, three design goals of the MPTCP congestion control algorithm have been defined. The goals are that a multipath flow should at least perform as well as a single path TCP flow; if more than one subflow shares a single bottleneck link, then, the multipath subflows should not harm other TCP flows; and a multipath flow should utilize the less congested path more than the congested one [31]. Several protocol designs from various authors are available for multipath data transfer. A protocol pTCP was proposed to transfer data concurrently through multiple paths [32]. In [33], the authors considered the wireless link as bottleneck to ensure protocol fairness and proposed a method to utilize the total bandwidth available on multiple paths of a multihomed mobile host. In [34], the authors proposed the design of a multipath TCP and discussed, in detail, the algorithm for detecting shared congestion at the bottleneck link by using fast retransmit events of different paths. In [35], the authors proposed a concurrent multipath transfer method (CMT) by using Stream Control Transmission Protocol (SCTP). The CMT-SCTP is the improved version of SCTP for implementing multipath transfer in multihomed hosts. All these schemes use uncoupled congestion control by running separate congestion control on each subflow. In addition, some of the protocols show a high degree of unfairness, in the case of shared bottleneck. In order to solve the issue of unfairness when multiple subflows of an MPTCP connection share the same bottleneck link, coupled congestion control algorithms have been proposed. In coupled congestion control algorithms, the congestion window of each subflow is updated by keeping in view the total congestion window of all subflows. The aim is to ensure bottleneck fairness and overall fairness in the network. In the case of a shared bottleneck link, these algorithms work efficiently but the possibility of a common bottleneck link shared by multiple flows is very rare. In the absence of a shared bottleneck link, the performance of these algorithms only results in the underutilization of available bandwidth. Several loss-based coupled congestion control based schemes such as LIA [31], balanced linked adaptation (BALIA) [36], opportunistic linked-increases algorithm (OLIA) [37] have been proposed. All these algorithms only control the increase mechanism of congestion window in congestion avoidance phase and the rest of the phases are like TCP Reno. In loss-based algorithms, the congestion window is adjusted on packet loss detection, so packet retransmissions are high and packet losses only give a rough esti- mate of network congestion. Another loss-based algorithm Dynamic LIA(D-LIA) [38] has been proposed that dynamically controls the decreasing mechanism of congestion window less aggressively but this behavior is not efficient and only adds more packet losses. A delay-based congestion control algorithm weighted Vegas (WVegas) [30] based on TCP Vegas has been proposed. The WVegas algorithm uses fine grained load balancing by using packet queuing delay as congestion signals. This algorithm shows low packet losses and better intra-protocol fairness. The authors have tried to improve fairness and load balancing more than the throughput. In order to resolve the issue of underutilization Electronics 2021, 10, 711 6 of 27

of bandwidth by WVegas in large bandwidth delay product networks, another delay-based algorithm, MPFast [39], has been proposed. The MPFast algorithm uses Fast TCP as a congestion control algorithm for multipath transfer. But the aggressive behavior of Fast TCP causes more packet losses in congested networks. In [28], the authors proposed machine learning methods for MPTCP path management to select high quality paths. In [20], a new MPTCP scheme with application distributor for low memory multihomed IoT devices was proposed. The aim of the scheme was to solve the buffer blocking issues in multihoming. In [40], the authors proposed an energy efﬁcient congestion control scheme, emReno based on mutipath TCP, to shift trafﬁc from one path to a low cost energy path. Table2 shows the comparison of existing MPTCP congestion control algorithms with the proposed MFVL HCCA.

Table 2. Comparison of existing multipath transmission control protocol (MPTCP) congestion control algorithms with the modiﬁed Fast-Vegas-LIA hybrid congestion control algorithm (MFVL HCCA).

Congestion Network Suitability for Control Standardized Mode of Operation Congestion Merits Demerits and Research Gaps Smart Meters in Method Detection Method Smart Gas Grid Increased Improvements in shared Coupled and Packet queuing throughput, bottleneck detection method is MFVL HCCA No uncoupled Delay and packet High reduced packet loss, required when the subflows (both) loss TCP friendliness. suffer different network delays Reduced throughput, underutilization of bandwidth in coupled mode, TCP friendliness in Yes increased packet loss rate due LIA [31] Coupled only Packet loss the ase of shared Low RFC 6356 to loss-based detection method bottleneck link and additive increase multiplicative decrease (AIMD) of congestion window Responsive, Reduced throughput, less non-flappy, TCP aggressive behavior due to OLIA [37] No Coupled only Packet loss friendliness, less Low coupled control mode, traffic transmission increased packet loss. over congested path TCP friendliness, Reduced throughput, due to BALIA [36] No Coupled only Packet loss Low responsiveness. coupled control Increased packet retransmissions due to dynamic decrease of congestion window Increased D-LIA [38] No Coupled only Packet loss in the case of packet loss, Low throughput dynamic slow decrease causes more packet losses as compared with multiplicative decrease Reduced throughput and Better intra protocol underutilization of bandwidth Packet queuing fairness Better load due to less aggressive behavior, WVegas [30] Coupled only Low delay balancing Reduced less efficient for high packet losses bandwidth delay product networks Increased throughput for Increased packet loss due to Packet queuing MPFast [39] No Coupled only large bandwidth aggressive behavior of Fast TCP Low delay delay product on more congested link networks Unfriendliness, reduced Reduced energy throughput, increased packet emReno [40] No Semi-coupled Packet loss Low consumption loss due to aggressive behavior of TCP Reno Unfriendliness, increased Increased packet loss due to aggressive CMT-SCTP [35] No Uncoupled Packet loss Low throughput behavior based on standard TCP

3. Proposed Modiﬁed Fast-Vegas-LIA Hybrid Congestion Control Algorithm (MFVL HCCA) Design In this section we present our proposed algorithm design. The algorithm MFVL is a hybrid congestion control algorithm which uses the following algorithms as submodules: Electronics 2021, 10, 711 7 of 27

• Modiﬁed Fast TCP congestion control algorithm (MFast); • Modiﬁed TCP Vegas congestion control algorithm (MVegas); • Shared bottleneck detection method and coupled congestion control LIA. After discussing details of these submodules, we explain the functionality of our main algorithm.

3.1. Modified TCP Vegas Congestion Control Algorithm (MVegas) Brakmo and Peterson proposed a delay-based algorithm called TCP Vegas for reliable transfer of data. According to the authors, it can achieve a much higher throughput than a loss-based algorithm TCP Reno and less packet retransmissions are required [41]. However, a drawback of the TCP Vegas is the inability to receive a fair bandwidth share when competing with TCP Reno and some other TCP variants. The TCP Vegas always consumes less bandwidth as compared with others [42]. In [43], the authors proposed modifications in the TCP Vegas to overcome this problem. Hence, some modifications are required in the original TCP Vegas to improve its performance in terms of overcoming the loss of bandwidth consumption. The TCP Vegas adjusts its congestion window size by calculating the difference between expected and actual throughput. A greater difference is the result of increased round trip time (RTT) delay and indicates that the network is congested. The TCP Vegas starts with a slow start, and then switches to congestion avoidance mode.

3.1.1. Slow Start Vegas uses threshold γ during slow start. The default value of γ is 1. When the difference (expected throughput–actual throughput) is less than γ, the congestion window (cwnd) is increased by 1 every other RTT. Hence, the cwnd in slow start grows exponentially, but at a slower rate than TCP Reno. When the difference is larger than γ or the value of cwnd becomes equal to the threshold value for congestion window in slow start (ssthresh), then, the congestion avoidance phase starts. Upon leaving slow start phase the cwnd is decreased by 1/8 of its current value in order to prevent network congestion.

3.1.2. Congestion Avoidance During the congestion avoidance phase, two threshold constants, α and β, are used. In the TCP Vegas, the congestion control algorithm tries to maintain the number of packets in network queues between a minimum and maximum value in order to prevent network congestion and avoid packet loss. The minimum value is represented by α and the maximum value is represented by β. The default value of α is 1 and for β it is 3 in NS-2.34. The cwnd is updated according to Equation (1). Equation (2) gives the difference between the expected throughput and actual throughput. RTT is the observed RTT and baseRTT is the minimum observed RTT.  + <  cwnd 1 i f di f f α cwnd = cwnd − 1 i f di f f > β (1)  cwnd otherwise

cwnd cwnd di f f = t − t (2) t baseRtt rtt In our proposed design, we have made some modiﬁcations, only to the congestion avoidance phase. Let di f ft and di f f(t−1) represents the differences at current time t and previous time (t − 1). Let ϕ represent the ratio of differences, then

di f f(t−1) ϕ = . (3) di f ft

If di f ft is less than di f f(t−1), it shows a gradual decline in network congestion and, correspondingly, ϕ holds a value greater than 1. If di f ft is greater than di f f(t−1), it shows a Electronics 2021, 10, 711 8 of 27

gradual increase in network congestion and, consequently, ϕ will have a value less than 1. In order to make Vegas a bit more aggressive, for the purpose of consuming the shared bandwidth efﬁciently, we have introduced two increasing factors α0 and β0. Their current 0 0 values at time “t” are represented by αt and βt. On reset(start), in our proposed model, they are set to constant values of 1 and 2, respectively. Their values are increased or decreased dynamically at run time by sensing the current difference and previous difference between 0 0 throughputs. The updated values after dynamic change are represented by αt+1 and βt+1, 0 0 respectively. Our proposed model tries to keep packets between (α + αt) and (β + βt) in network queues by adjusting the cwnd accordingly. The difference between throughputs 0 is calculated and if (di f ft) is less than (α + αt), it shows room for more packets to be injected into the network and, consequently, the cwnd is updated according to Equation (4) as follows: cwnd = cwnd + 1 (4) 0 The value of αt+1 is calculated by using Equation (5) as follows: 0 αt+1 = min(k ∗ α, ρ) (5)

where 0 ρ = αt + ϕ. (6) A value of ϕ > 1 indicates that there is a decrease in network congestion over time, and as a result, the proposed model increases α0 and β0 dynamically by adding the value of ϕ in 0 their current values. The updated αt+1 can have a maximum value equal to (k ∗ α) where 0 k is a constant and its value lies between 1 to 2. With k = 1, αt+1 can achieve a maximum 0 value of 1, because the default value of α is 1, and with k = 2, αt+1 can have a maximum value of 2. In the proposed design, k was set to 2 after carefully evaluating it through different experiments. We observed that increasing it further, increases the maximum value 0 of αt+1, resulting in a more aggressive increase in congestion window, and thus resulting 0 in more packet drops. Decreasing k below 1, results in a decreased maximum value of αt+1, which results in underutilization of available bandwidth due to less aggressive growth of the congestion window. 0 0 The value of βt+1 is calculated by using Equation (7) (βt+1 can have a maximum value equal to β) as follows: 0 βt+1 = min(β, ω) (7) where 0 ω = βt + ϕ. (8) Any value greater than the maximum value can result in higher packet loss due to a more aggressive increase in the congestion window. 0 If di f ft > β + βt, it indicates that the network is currently congested, hence, the congestion window is decreased, according to Equation (9), as follows:

cwnd = cwnd − 1 (9)

0 and βt+1 is calculated, according to Equation (10), as follows: 0 0 βt+1 = max (α + 1), ω (10)

where 0 0 ω = βt − ϕ. (11) 0 βt+1 can only be decreased to a minimum value of α + 1, and further reducing it will make it equal to or less than α which would result in the underutilization of available bandwidth and network queues. For the sake of avoiding severe degradation in the utilization of available bandwidth, we decided not to decrease α0 dynamically in the case Electronics 2021, 10, 711 9 of 27

of network congestion, as it is a factor that decides the minimum number of packets to be kept in network queues.

3.1.3. Loss Recovery When packet loss is detected (due to time out), then ssthresh is set to half of the cwnd, cwnd is reset to 2, and slow start phase starts again. When three duplicate ACKs are received, then fast retransmission and fast recovery is executed. After fast retransmit, cwnd 3 is set to 4 of the current cwnd, and congestion avoidance phase is executed again.

3.2. Modiﬁed Fast TCP Congestion Control Algorithm Another congestion control algorithm Fast TCP [44] uses queuing delay along with packet loss to detect network congestion. The Fast TCP updates its window according to Equation (12), under which the network moves toward equilibrium rapidly in larger steps and slows down by taking smaller steps near equilibrium. The Fast TCP adjusts its window in three phases as follows: slow start (SS), multiplicative increase (MI), and exponential convergence (EC). The Fast TCP uses the same slow start algorithm of TCP Reno with only a slight variation by using a threshold gamma. The Fast TCP exits slow start when the number of packets present in the network queue exceeds gamma. In order to achieve equilibrium, the Fast TCP uses MI; the Fast TCP increases or decreases its window on alternate RTTs in both MI and EC phases as a protection measure. When a packet loss is detected, the window is reduced to half and loss recovery phase starts. In EC phase, the window is increased exponentially. The new window size is calculated by using Equation (12) as follows:

baseRTT w ← min 2w, (1 − γ)w + γ w + α(w, qdelay) (12) RTT

where γ ∈ [0, 1], baseRTT is the current minimum RTT, and qdelay represents the end-to- end queuing delay (average), and α(w, qdelay) is a constant that represents the number of packets each flow tries to maintain in the network buffer(s) at equilibrium [45]. In our algorithm, we modified this behavior of the Fast TCP by introducing a controlling factor σ. The modified equation is given as follows:

baseRTT w ← min 2w, (1 − γ)w + σ ∗ γ w + α(w, qdelay) (13) avgRTT

where σ is a constant and its value can be adjusted from 0 to 1. Towards 0, Fast TCP will behave less aggressively and towards 1, Fast TCP will move towards its natural aggressive behavior. In our design, we used σ = 0.5, after testing its different values for different simulation scenarios. By increasing the value of this controlling factor, more throughput can be achieved but unfairness and packet drop also increases.

3.3. Shared Bottleneck Detection and Coupled Congestion Control In [34], the authors proposed a shared congestion detection method using fast re- transmits. If the two subflows or paths share the same bottleneck congestion, they are said to be “correlated” otherwise “independent”. We have assumed that the latency of paths is equal. Every time a fast retransmit event happens at any subflow, the time of the event in that specific subflow is recorded in a list. In this way, two lists, i.e., A and B, with timestamps (a1, a2, ... , am) and (b1, b2, ... , bn) from subflows S1 and S2 are obtained. Then, timestamps from A and B are compared in such a way that if |ai − bj| < interval, then (ai,bj) is said to be a match. A match shows that packets were lost and that a fast retransmit event took place at both paths at about the same time. Therefore, both paths probably share the same congested link. The maximum number of matched pairs (ai,bj) are represented by match(A,B). The term min(m,n) gives the minimum number of recorded Electronics 2021, 10, 711 10 of 27

fast retransmit timestamps from lists A and B. Two subﬂows are considered to be sharing the same bottleneck congested link if detection results in a value greater than a threshold δ.

Match(A, B) Detection = > δ, (14) min(m.n)

The authors in [34], after performing several experiments, showed that by using interval = 200 ms and δ = 0.5, the shared congestion could be detected successfully. We used the same method to detect shared bottleneck. If shared bottleneck is detected, then, our proposed model switches to coupled congestion control mode and selects default coupled congestion control LIA of MPTCP [31] for both subﬂows. LIA uses the following steps for only increasing the cwnd in congestion avoidance phase (the remaining phases behave similar to a standard TCP algorithm): • cwnd_count_i is maintained for each subﬂow ‘i’; • cwnd_count_i is the number of segments acked since the last cwndi increment; • cwndi is incremented by 1 (and after increment, cwnd_count_i is set to 0); when cwnd cwnd > max alpha ∗ total , cwnd , (15) count scale alpha i where cwnd alpha = alpha ∗ cwnd ∗ max , (16) scale total 2 sum rttmax∗cwndi rtti

and alphascale is the precision parameter. According to [31], setting alphascale to 512 works well in most cases. We used the same algorithm in our model without any modiﬁcations.

3.4. Main Functionality of MFVL HCCA and Modes of Operation Here, we discuss the main flow of our algorithm. Figure2a shows the IoT protocol stack with MPTCP as transport layer protocol. The system architecture is shown in Figure2b . A smart meter with two Wi-Fi Halow interfaces is connected to a remote database server via multipath connectivity. The multipath flow is divided into two subflows. Two gateways from different ISPs are used for multipath connectivity. Our proposed congestion control algorithm maintains separate congestion windows for each subflow. The MFVL HCCA operates in the following two modes: 1. Uncoupled mode; 2. Coupled mode. On reset, the MFVL algorithm starts in an uncoupled mode and runs the modified TCP Vegas congestion control algorithm for each subflow. On every packet loss, packet retransmission takes place. For a duration of T (s), the number of retransmitted packets belonging to a particular subflow are counted, and the timestamps are also saved in an array. After the time T, shared bottleneck detection is performed by using the timestamps of packet retransmissions for each subflow. The probability of a bottleneck link shared by two subflows belonging to the same multipath flow is very rare but if shared bottleneck is detected, then the MFVL algorithm switches to coupled congestion control mode and LIA is selected as the coupled congestion control algorithm for both subflows. If no shared bottleneck is detected, then, the MFVL Algorithm keeps working in uncoupled congestion control mode. The number of retransmitted packets of Subflow 1 and Subflow 2 are compared. The modified Fast algorithm is selected for subflow with less packet retransmissions and the modified Vegas is selected for subflow with more packet retransmissions. After time T, the shared bottleneck detection and selection process are repeated again by observing packet retransmissions ElectronicsElectronics 20212021,, 1010,, 711x FOR PEER REVIEW 1112 ofof 2728

(a)

(b)

Figure 2. Cont.

Electronics 2021,, 10,, 711x FOR PEER REVIEW 1312 of 28 27

(c)

Figure 2.2. (a(a)) IoT IoT Protocol Protocol stack stack with with proposed proposed modiﬁed modified Fast-Vegas-LIA Fast-Vegas-LIA hybrid hybrid congestion congestion control control algorithm algorithm (MFVL HCCA)(MFVL forHCCA) multipath for multipath transmission transmission control protocol control (MPTCP); protocol (b(MPTCP);) System architecture;(b) System (architecture;c) Proposed MFVL(c) Proposed HCCA ﬂowchart.MFVL HCCA flowchart.

Electronics 2021, 10, 711 13 of 27

3.4.1. Algorithm Limitations The shared bottleneck detection method used in the proposed algorithm works well when the two subflows (paths) experience the same network latency, but in the case of different network delays, time synchronization is an issue. Hence, further improvements in the design are required. In the future, we are planning to improve the efficiency of the shared bottleneck detection method by reducing the time required to detect a shared congested link and to handle multiple subflows experiencing different network delays.

3.4.2. Algorithm Explanation The notations used in the proposed algorithm are given in Table3 along with their deﬁnitions.

Table 3. Notations used in the proposed algorithm.

Notation Definition ‘t’ Simulation time in seconds A constant value for the time duration to count packet retransmissions, in simulations we T used T = 50 s A time interval used in the calculation of shared bottleneck detection, in simulations we used Interval interval = 200 ms bn_thresh The bottleneck detection threshold δ. In simulations we used bn_thresh = 0.5 P1_R A variable to store number of packet retransmissions belonging to subflow 1 (path 1) P2_R A variable to store number of packet retransmissions belonging to subflow 2 (path 2) P1[ ] An array to store packet retransmission time stamps belonging to subflow 1 (path 1) P2[ ] An array to store packet retransmission time stamps belonging to subflow 2 (path 2) Match A variable to store matched packet retransmission pairs from both subflows Bneck_Detect A variable to store result of bottleneck detection calculation i, j Variables i and j for loops and array index i_max, j_max Variables to store maximum values of i and j

Algorithm 1 shows the pseudo code. The algorithm starts with initial values of T = 50 s, bn_thresh = 0.5, interval = 200 ms, and all other variables equal to 0. At the start, the modified TCP Vegas is selected for each subflow as the congestion control algorithm. For a simulation time t = 0 to t = T s, on each packet retransmission event belonging to Subflow 1 (Path 1), the variable P1_R and “i” are incremented and the time stamp for the event is stored in array P1[i]. On each packet retransmission event belonging to Subflow 2 (Path 2), the variable P2_R and “j” are incremented and the time stamp for the event is stored in array P2[j]. When the loop ends, i_max holds the maximum value of i and j_max holds the maximum value of j. The values of P1_R and P2_R show the number of packet retransmissions that took place during time T s for Subflows 1 and 2, respectively. The arrays P1[ ] and P2[ ] have the total number of stored time stamp values equal to i_max and j_max, respectively. The values stored in P1[ ] and P2[ ] are the time stamps of packet retransmission events belonging to Subflow 1 and Subflow 2, respectively. In the next step, by using nested loops the matched pairs from two arrays P1[ ] and P2[ ] are detected in such a way that each element of P1[ ] is compared with each element of P2[ ] one by one, by subtracting their values. If the absolute value results in a number less than the “interval”, the pair is said to be matched. The value of the interval in our case is 200 ms. Therefore, all the pairs of two arrays P1[ ] and P2[ ] are said to be matched that have a value difference of less than 200 ms. The “match” variable is incremented on detecting each matched pair. The matched pair indicates that the retransmission took place on both subflows at almost the same time. A greater number of matches indicate that both the subflows are sharing the same congested bottleneck link. For bottleneck detection, the number of matched pairs is divided by the minimum number of array elements. If the answer is greater than bn_thresh (bottleneck detection threshold value), then, there is a high probability that a shared bottleneck is present, otherwise absent. In the case of shared bottleneck, the default coupled congestion control LIA is used for both subflows (paths). Electronics 2021, 10, 711 14 of 27

If shared bottleneck is not detected, then the numbers of packet retransmissions in both subflows are compared. The modified TCP Vegas congestion control algorithm is selected for subflow with more packet retransmissions and the modified Fast TCP congestion control algorithm is selected for subflow with less packet retransmissions. The algorithm then jumps to label “Again” and from t = 0 to t = T s the whole procedure is repeated again for the next cycle. The flow chart of the proposed algorithm is shown in Figure2c.

Algorithm 1. MFVL HCCA 1: Inputs: T, Interval, bn_thresh //T = 50 s, Interval = 200 ms, //bn_thresh = 0.5 2: Outputs: P1_R, P2_R, P1[ ], P2[ ], Match,Bneck_Detect 3: Start: 4: Run Modified Vegas Congestion Control on subflow1 (Path1) 5: Run Modified Vegas Congestion Control on subflow 2(Path2) 6: Again: 7: Initialize: t = 0, i = 0, j = 0, i_max = 0, j_max = 0, P1_R = 0, P2_R = 0, P1[0] = 0, P2[0] = 0 8: For (t = 0 to t = T s) do 9: if (Packet Retransmission takes place at subflow1 ) then 10: i++ 11: P1_R = P1_R + 1 12: P1[i] = t 13: End if 14: if (Packet Retransmission takes place at subflow2 ) then 15: j++ 16: P2_R = P2_R + 1 17: P2[j] = t 18: End if 19: End For 20: i_max = i 21: j_max = j 22: For( i = 1 to i = i_max) do 23: For(j = 1 to j = j_max) do 24: If (|P1[i] − P2[j]| < Interval) then 25: Match++ 26: End if 27: End For 28: End For 29: Bneck_Detect = Match/min(i_max,j_max) 30: If (Bneck_Detect > bn_thresh) then 31: Run Coupled Congestion Control on subflow1 and subflow2 32: End if 33: If (P1_R < P2_R) then 34: Run Modified Vegas Congestion Control on subflow 2 35: Run Modified Fast Congestion Control on subflow 1 36: End if 37: if (P1_R > P2_R) then 38: Run Modified Vegas Congestion Control on subflow 1 39: Run Modified Fast Congestion Control on subflow 2 40: End if 41: Go to Again

4. Simulations, Results, and Discussions All the simulations were done using NS-2.34. We used wired-cum-wireless network topology. For the MPTCP simulation, the available MPTCP module [46] with coupled congestion control LIA (MPTCP-CC-LIA) for NS2 was used. Modifications were done to the original Fast TCP and TCP Vegas codes in NS-2.34, and then these modified codes were used to implement the proposed MFVL HCCA for the MPTCP. Different experiments were performed to compare the performance of our proposed model with MPTCP-CC-LIA. The final results were plotted using GNUPLOT. Figure3 shows the network connections. Node “S” is a multihomed source node with two interfaces “S_0” and “S_1” for wireless connections to gateways. A multipath flow is divided into two subflows. Packets transmitted through S_0 represent Subflow 1. Packets transmitted through S_0 represent Subflow 2. Selected parameters for wireless connections are based on 802.11 standard of NS2. The MPTCP agent with File Transfer Electronics 2021, 10, x FOR PEER REVIEW 16 of 28

Packets transmitted through S_0 represent Subflow 2. Selected parameters for wireless connections are based on 802.11 standard of NS2. The MPTCP agent with File Transfer Protocol (FTP) traffic generator is connected to source node. The source node connects wirelessly to two different gateway nodes, GW1 and GW2, via interfaces S_0 and S_1. Electronics 2021, 10, 711 15 of 27 The gateways are further connected to routers through wired links. Node “D” is the destination multihomed node with two wireless interfaces “D_0” and “D_1”. Node “D” was connected wirelessly to two gateway nodes, “GW3” and“GW4”, through D_0 and D_1, Protocolrespectively. (FTP) The traffic connection generator between is connected R1 and toR2 source shows node. the bottleneck The source link node for Path connects 1 of wirelesslydata rate 3 to Mbps, two different 50 ms delay, gateway and nodes, queue GW1 limit and of 100 GW2, packets. via interfaces R3-R4 shows S_0 and the S_1. bottle- The gatewaysneck link for are Path further 2 of connected data rate to 1 routersMbps, 50 through ms delay, wired and links. queue Node limit “D” of is100 the packets. destination The multihomedremaining wired node connections with two wireless have data interfaces rate 10 “D_0” Mbps and and “D_1”. 10 ms Nodedelay. “D” Nodes was N1 connected abd N3 wirelesslyare used to to inject two gateway background nodes, traffic “GW3” to crea and“GW4”,te path congestion; through D_0 constant and D_1, bit respectively. rate (CBR) Thetraffic connection generators between with UDP R1 and agents R2 shows are used. the bottleneck The packet link size for of Path CBR 1 of and data TCP rate is 3 Mbps,set to 501000 ms bytes. delay, The and maximum queue limit window of 100 packets. size for R3-R4each interface shows the is bottleneckset to 100 linkpackets. for Path Null 2 ofagents data are rate attached 1 Mbps, with 50 ms nodes delay, N2 andand queueN4. The limit UDP of agents 100 packets. are connected The remaining to null agents. wired connectionsGW1-R1-R2-GW3 have datarepresents rate 10 Path Mbps 1. andGW2-R3-R4-GW4 10 ms delay. Nodes represents N1 abd Path N3 2. are used to inject backgroundIn the presented traffic to createsimulation path congestion;scenarios, no constant shared bit bottleneck rate (CBR) path traffic is used generators because with we UDPare more agents interested are used. in The analyzing packet size the ofresult CBRs and of our TCP proposed is set to 1000 algorithm bytes. Thein uncoupled maximum windowmode which size foruses each the interfacemodified is Vegas set to and 100 packets.modified Null Fast agents algorithms. are attached In coupled with mode, nodes N2the andalgorithm N4. The works UDP according agents are to connectedMPTCP LIA, to nulltherefore, agents. the GW1-R1-R2-GW3 behavior is the same represents as that Pathof default 1. GW2-R3-R4-GW4 coupled congestion represents control Path of MPTCP. 2.

Figure 3. Simulation topology.

InTo theimplement presented our simulation proposed scenarios, design, nothe sharedrequired bottleneck modifications path is were used done because in wens-defaults.tcl are more interested and other in NS2 analyzing “c” files. the NS2 results was of recompiled our proposed by algorithmusing “make”. in uncoupled Various modeexperiments which useswere therun modified by using Vegas the same and modifiedsimulation Fast topology. algorithms. The Insimulation coupled mode,time, rep- the algorithmresented by works “T” was according set to 50 to s. MPTCP The CBR LIA, data therefore, rate was the initially behavior set to is 1 the Mbps. same Same as that CBR of defaultstart stop coupled pattern congestion was usedcontrol for all ofexperiment MPTCP. s. The CBR start stop pattern is shown in FigureTo 4 implement and throughput our proposed in Figure design, 5. We ha theve requiredwritten different modifications awk scripts were doneto calculate in ns- defaults.tclthe average andgoodput, other average NS2 “c” packet files. NS2drop wasrate,recompiled average goodput, by using and “make”. packet drop Various vs. experimentsCBR datarate. were Simulation run by using parameters the same are simulation given in Table topology. 4. The The default simulation values time, used repre- for sentedmodified by Vegas “T” was and set modified to 50 s. Fast The algorithms CBR data rate are wasshown initially in Tables set to5 and 1 Mbps. 6, respectively. Same CBR start stop pattern was used for all experiments. The CBR start stop pattern is shown in Figure4 and throughput in Figure5. We have written different awk scripts to calculate the average goodput, average packet drop rate, average goodput, and packet drop vs. CBR datarate. Simulation parameters are given in Table4. The default values used for modified Vegas and modified Fast algorithms are shown in Tables5 and6, respectively. Electronics 2021, 10, x FOR PEER REVIEW 17 of 28

Table 4. Simulation parameters for wired links.

Simulation Parameter Value CBR packet size 1000 bytes TCP packet size 1000 bytes Queue type (wired links) Drop Tail Queue size (packets) 100 R1–R2 Bottleneck for Path 1 R3–R4 Bottleneck for Path 2

Table 5. Modified TCP Vegas parameters.

Parameter Value on Reset 𝛼 1 𝛽 3 𝛾 1 𝛼′ 1 𝛽′ 2

Table 6. Modified Fast TCP parameters.

Parameter Value on Reset 𝛼 100 Electronics 2021, 10, 711 𝛾 0.5 16 of 27 MI threshold 0.00075 𝜎 0.5

Electronics 2021, 10, x FOR PEER REVIEW 18 of 28 FigureFigure 4. 4. ConstantConstant bit bit rate rate (CBR) (CBR) traffic trafﬁc generator generator star startt stop stop pattern pattern for for each path with CBR data rate rate1 Mbps. 1 Mbps.

FigureFigure 5. 5. AggregateAggregate throughput throughput of of CBR CBR traffic trafﬁc when when source source node node “S” “S” is is using using MPTCP MPTCP with with coupled coupled congestioncongestion control control linked-increases linked-increases algorithm algorithm (MPTCP-CC-LIA). (MPTCP-CC-LIA).

4.1.Table Setup 4. Simulation One parameters for wired links. In the first setup, we implemented our multipath uncoupled congestion control al- Simulation Parameter Value gorithm as follows: CBR packet size 1000 bytes (a) Using modifiedTCP packet TCP Vegas size (MVegas) congestion control on 1000 both bytes subflows; (b) Using modifiedQueue type Fast (wired TCP links) (MFast) congestion control on both Drop subflows. Tail Queue size (packets) 100 We compared itsR1–R2 performance with MPTCP-CC-LIA. Bottleneck The results for Pathin Figure 1 6 show the average goodputR3–R4 graphs of three different multipath Bottleneckflows. The for goodput Path 2 of a single multipath flow is the aggregate goodput of its individual subflows. The multipath flow with MFast (MPTCP-MFast) achieves better average goodput than MPTCP-MVegas and MPTCP-CC-LIA. The CBR traffic generator starts generating background traffic at t = 5 s and stops at t = 15 s. It again starts at t = 35 s and stops at t = 45 s. The CBR traffic pattern for both paths is the same. In simulation, the FTP traffic for both paths starts at t = 1.5 s and stops at t = 50 s. When FTP traffic starts, then MVegas and MFast keep on increasing their congestion windows until packet drops are observed. The more congested path (Path 2) experiences more packet drops when CBR is started in the background.

Figure 6. Average goodput result of Setup 1.

Electronics 2021, 10, x FOR PEER REVIEW 18 of 28

Electronics 2021, 10, 711 17 of 27

Table 5. Modiﬁed TCP Vegas parameters.

Parameter Value on Reset α 1 β 3 γ 1 α0 1 β0 2

Table 6. Modiﬁed Fast TCP parameters.

Parameter Value on Reset α 100 Figure 5. Aggregate throughputγ of CBR traffic when source node “S” is using0.5 MPTCP with coupled congestion control linked-increasesMI threshold algorithm (MPTCP-CC-LIA). 0.00075 σ 0.5 4.1. Setup One 4.1.In Setup the Onefirst setup, we implemented our multipath uncoupled congestion control al- gorithmIn as the follows: first setup, we implemented our multipath uncoupled congestion control algorithm as follows: (a) Using modified TCP Vegas (MVegas) congestion control on both subflows; (a) Using modified TCP Vegas (MVegas) congestion control on both subflows; (b) Using modified Fast TCP (MFast) congestion control on both subflows. (b) Using modified Fast TCP (MFast) congestion control on both subflows. WeWe compared compared its its performance performance with with MPTCP-CC-LIA. MPTCP-CC-LIA. The The results results in in Figure Figure 66 showshow thethe average average goodput goodput graphs graphs of of three three different different multipath multipath flows. flows. The The goodput goodput of of a asingle single multipathmultipath flow flow is isthe the aggregate aggregate goodput goodput of of its its individual individual subflows. subflows. The The multipath multipath flow flow withwith MFast MFast (MPTCP-MFast) (MPTCP-MFast) achieves achieves better better average average goodput goodput than than MPTCP-MVegas MPTCP-MVegas and and MPTCP-CC-LIA.MPTCP-CC-LIA. The The CBR CBR tr trafficaffic generator generator starts starts generating generating background background traffic traffic at at t t= =5 5s s andand stops stops at at t t= =15 15 s. s.It Itagain again starts starts at at t = t =35 35 s sand and stops stops at at t t= =45 45 s. s.The The CBR CBR traffic traffic pattern pattern forfor both both paths paths is is the the same. same. In simulation, thethe FTPFTP traffic traffic for for both both paths paths starts starts at tat = t 1.5 = 1.5 s and s andstops stops at tat = t 50 = s.50 Whens. When FTP FTP traffic traffic starts, starts then, then MVegas MVegas and and MFast MFast keep keep on increasingon increasing their theircongestion congestion windows windows until until packet packet drops drops are observed. are observed. The more The congestedmore congested path (Path path 2) (Pathexperiences 2) experiences more packet more packet drops whendrops CBR when is CBR started is started in the background. in the background.

FigureFigure 6. 6.AverageAverage goodput goodput result result of of Setup Setup 1. 1.

The packet drop starts after t = 5 s and rises sharply. Figure7 shows the aggregate packet drops of both subﬂows. On detecting packet loss, the congestion window of that

Electronics 2021, 10, x FOR PEER REVIEW 19 of 28 Electronics 2021, 10, 711 18 of 27

The packet drop starts after t = 5 s and rises sharply. Figure 7 shows the aggregate packet drops of both subflows. On detecting packet loss, the congestion window of that specific subflow is decreased, and therefore the aggregate goodput is also decreased. specific subflow is decreased, and therefore the aggregate goodput is also decreased. Figure7 shows the average packet drop rate corresponding to each multipath flow. The Figure 7 shows the average packet drop rate corresponding to each multipath flow. The packetpacket drop drop rate rate of of a asingle single multipath multipath flow flow is the is the aggregate aggregate packet packet drop drop rate rateof its of indi- its individual vidualsubflows. subflows. The averageThe average packet packet drop drop rate ra ofte MPTCP-MFastof MPTCP-MFast is is also also higher higher as as compared com- with paredMPTCP-MVegas with MPTCP-MVegas and MPTCP-CC-LIA. and MPTCP-CC-LIA.

Figure 7. Average packet loss rate of Setup 1. Figure 7. Average packet loss rate of Setup 1. The congestion window plots of multipath flows MPTCP-MVegas and MPTCP-MFast The congestion window plots of multipath flows MPTCP-MVegas and MPTCP-MFastare shown in are Figures shown8 inand Figures9, respectively; 8 and 9, respectively; the congestion the congestion window window graph graph belonging to belongingeach subflow to each is subflow shown. is Path shown. 1 is lessPath congested1 is less congested than Path than 2. Path Subflow 2. Subflow 1 utilizes 1 uti- Path 1 and lizesSubflow Path 1 2 and utilizes Subflow Path 2 2.utilizes The results Path 2. show The results that MPTCP-MVegas show that MPTCP-MVegas is less aggressive is less on both aggressivesubflows on as both compared subflows with as compared MPTCP-MFast. with MPTCP-MFast. This behavior This of behavior MVegas of is veryMVegas beneficial to isavoid very beneficial packet drop to avoid on path packet with drop more on congestion. path with more MPTCP-MFast congestion.behaves MPTCP-MFast aggressively on behavesmore congested aggressively path on whichmore congested results in path more which packet results drops. in more By comparing packet drops. the By congestion comparingwindows the of MFastcongestion and windows MVegas of for MFast more and congested MVegas for path more (Path congested 2), we path observe (Path that when 2),CBR we observe in the background that when CBR starts in the at tbackground = 5 s, then starts both at the t = windows 5 s, then both are alreadythe windows in congestion areavoidance already in phase congestion with the avoidance window phase size ofwith MFast the window more than size double of MFast the more sizeof than MVegas, but doubleon detecting the size packetof MVegas, drop, but MFast on detecting decreases packet its windowdrop, MFast and decreases reduces its it window to almost zero at and reduces it to almost zero at t = 7 s, but MVegas reduces the congestion window in a t = 7 s, but MVegas reduces the congestion window in a very slow rate by using cwnd = very slow rate by using cwnd = cwnd − 1. The multiplicative increase in cwnd of MFast Electronics 2021, 10, x FOR PEER REVIEWcwnd − 1. The multiplicative increase in cwnd of MFast gives rise to more20 of 28 packet drops. gives rise to more packet drops. After t = 15 s, both go into slow start again. After t = 15 s, both go into slow start again.

Figure 8. 8. CongestionCongestion windows windows of individual of individual subflows subﬂowsbelonging to belonging MPTCP modified to MPTCP TCP Vegas modiﬁed TCP Vegas (MPTCP-MVegas).

Figure 9. Congestion Windows of individual subflows belonging to MPTCP modified Fast TCP (MPTCP-MFast).

4.2. Setup Two In the second setup, for further analysis we implemented our multipath hybrid congestion control algorithm MFVL as follows: (a) Using modified TCP Vegas congestion control (MVegas) on subflow with less congested path; (b) Using modified Fast TCP congestion control (MFast) on subflow with more congested path. Our algorithm achieved higher average goodput than MPTCP-CC-LIA, as shown in Figure 10; however, the packet drop rate, as shown in Figure 11, also increased due to the aggressive behavior of MFast on subflow with more congested path (Path 2). Congestion windows are plotted in Figure 12.

Electronics 2021, 10, x FOR PEER REVIEW 20 of 28

Electronics 2021, 10, 711 19 of 27 Figure 8. Congestion windows of individual subflows belonging to MPTCP modified TCP Vegas (MPTCP-MVegas).

FigureFigure 9. 9. CongestionCongestion Windows Windows of ofindividual individual subflows subﬂows belonging belonging to MPTCP to MPTCP modified modiﬁed Fast FastTCP TCP (MPTCP-MFast).(MPTCP-MFast).

4.2.4.2. Setup Setup Two Two InIn the the second second setup, setup, for for further further analysis analysis we we implemented implemented our our multipath multipath hybrid hybrid congestioncongestion control control algorithm algorithm MFVL MFVL as as follows: follows: (a) Using modiﬁed TCP Vegas congestion control (MVegas) on subﬂow with less con-

(a) Usinggested modified path; TCP Vegas congestion control (MVegas) on subflow with less con- (b) gestedUsing path; modified Fast TCP congestion control (MFast) on subflow with more con- (b) Usinggested modified path. Fast TCP congestion control (MFast) on subflow with more congested path. Our algorithm achieved higher average goodput than MPTCP-CC-LIA, as shown in FigureOur 10 algorithm; however, achieved the packet higher drop average rate, as showngoodput in than Figure MPTCP-CC-LIA, 11, also increased as dueshown to thein Electronics 2021, 10, x FOR PEER REVIEW 21 of 28 Figureaggressive 10; however, behavior the of MFastpacket ondrop subflow rate, as with shown more in congestedFigure 11, pathalso increased (Path 2). Congestiondue to the aggressivewindows arebehavior plotted of in MFast Figure on 12 subflow. with more congested path (Path 2). Congestion windows are plotted in Figure 12.

FigureFigure 10. 10. AverageAverage goodput goodput result result of of Setup Setup 2. 2.

Figure 11. Average packet loss rate of Setup 2.

Figure 12. Congestion window graphs when MVegas congestion control is used on subflow with less congested path (Path 1), and MFast is used on subflow with more congested path (Path 2).

ElectronicsElectronics 2021 2021, ,10 10, ,x x FOR FOR PEER PEER REVIEW REVIEW 2121 of of 28 28

Electronics 2021, 10, 711 20 of 27

FigureFigure 10. 10. Average Average goodput goodput result result of of Setup Setup 2. 2.

FigureFigure 11. 11. Average Average packet packet loss loss rate rate of of Setup Setup 2. 2.

FigureFigure 12. 12. Congestion CongestionCongestion window window window graphs graphs graphs when when when MVegas MVegas MVegas cong cong congestionestionestion control control control is is isused used used on on on subflow subflow subflow with with with lesslessless congested congested congested path path (Path (Path 1), 1), an an anddd MFast MFast is is used used on on subflow subflowsubflow with with more more congested congestedcongested path pathpath (Path (Path(Path 2). 2).2). 4.3. Setup Three After observing the results of the first two experiments, we finalized the design of our proposed algorithm by deciding to use MVegas on subflow with more congested path and MFast on subflow with less congested path. The path congestion was detected by recording packet retransmissions for each subflow. We also implemented a shared bottleneck detection method in our algorithm. In real scenarios, the probability of a congested path shared by both subflows is very low and if a shared bottleneck is detected, then our algorithm switches to the standard coupled congestion control algorithm LIA. The simulations were run by using CBR data rate of 1 Mbps. The results show that MPTCP with our proposed model MFVL HCCA (MPTCP-MFVL) achieves higher throughput and less packet drop rate as compared with MPTCP with coupled congestion control LIA (MPTCP-CC-LIA). The results are shown in Figures 13 and 14. To analyze the performance of our proposed model, here, we plotted the results for the time duration of simulation time t = 0 to t = T s (here, T = 50) when the algorithm has already selected MVegas for subflow with more congested path (Path 2) and MFast for subflow with less congested path (Path 1) after detecting path congestion by calculating packet retransmissions for each Electronics 2021, 10, x FOR PEER REVIEW 22 of 28 Electronics 2021, 10, x FOR PEER REVIEW 22 of 28

4.3. Setup Three 4.3. Setup Three After observing the results of the first two experiments, we finalized the design of After observing the results of the first two experiments, we finalized the design of our proposed algorithm by deciding to use MVegas on subflow with more congested our proposed algorithm by deciding to use MVegas on subflow with more congested path and MFast on subflow with less congested path. The path congestion was detected path and MFast on subflow with less congested path. The path congestion was detected by recording packet retransmissions for each subflow. We also implemented a shared by recording packet retransmissions for each subflow. We also implemented a shared bottleneck detection method in our algorithm. In real scenarios, the probability of a bottleneck detection method in our algorithm. In real scenarios, the probability of a congested path shared by both subflows is very low and if a shared bottleneck is de- congested path shared by both subflows is very low and if a shared bottleneck is detected, then our algorithm switches to the standard coupled congestion control algorithm tected, then our algorithm switches to the standard coupled congestion control algorithm LIA. The simulations were run by using CBR data rate of 1 Mbps. The results show that LIA. The simulations were run by using CBR data rate of 1 Mbps. The results show that MPTCP with our proposed model MFVL HCCA (MPTCP-MFVL) achieves higher MPTCP with our proposed model MFVL HCCA (MPTCP-MFVL) achieves higher throughput and less packet drop rate as compared with MPTCP with coupled congestion throughput and less packet drop rate as compared with MPTCP with coupled congestion control LIA (MPTCP-CC-LIA). The results are shown in Figures 13 and 14. To analyze the control LIA (MPTCP-CC-LIA). The results are shown in Figures 13 and 14. To analyze the performance of our proposed model, here, we plotted the results for the time duration of performance of our proposed model, here, we plotted the results for the time duration of Electronics 2021, 10, 711 simulation time t = 0 to t = T s (here, T = 50) when the algorithm has already selected21 of 27 simulation time t = 0 to t = T s (here, T = 50) when the algorithm has already selected MVegas for subflow with more congested path (Path 2) and MFast for subflow with less MVegas for subflow with more congested path (Path 2) and MFast for subflow with less congested path (Path 1) after detecting path congestion by calculating packet retrans- congested path (Path 1) after detecting path congestion by calculating packet retransmissions for each subflow. Now, for this time duration, the packet retransmissions are missionssubﬂow. for Now, each for subflow. this time Now, duration, for this the time packet duration, retransmissions the packet are retransmissions calculated again are for calculated again for each subflow to take decisions for the next cycle. calculatedeach subﬂow again to for take each decisions subflow for to thetake next decisions cycle. for the next cycle.

FigureFigure 13. 13. AverageAverage goodput goodput result result of of Setup Setup 3. Figure 13. Average goodput result of Setup 3.

FigureFigure 14. 14. AverageAverage packet packet drop drop rate rate of of Setup Setup 3. 3. Figure 14. Average packet drop rate of Setup 3. Figure 15 shows the congestion window graphs of MPTCP-CC-LIA. The behavior is aggressive on both subﬂows in slow start phase. After detecting congestion, MPTCP shifts

more data to subflow with less congested path. The behavior of congestion window on less congested path (Path 1) is very aggressive, thus, increasing packet drop rate in addition to throughput. On more congested path (Path 2) the congestion window is decreased to half on packet loss detection and almost goes to zero on the next packet loss event, thus, avoiding packet drop rate but decreasing throughput also. The aim of the coupled congestion control algorithm LIA of MPTCP is to ensure fairness in the case of a common bottleneck path shared by multiple subflows of the same multipath flow. By using the total congestion window of multiple subflows as a factor in Equations (15) and (16) of Section 3.3 to calculate the increase of congestion window of a single subflow, MPTCP has successfully achieved Goal 2 [29], i.e., a multipath TCP should not take up more capacity than a single TCP flow in the case of a shared bottleneck link and Goal 3, i.e., a multipath flow should shift its traffic to the less congested link as much as possible. But to achieve these two goals there is also a decrease in the overall throughput due to the underutilization of available bandwidth. The probability of a Electronics 2021, 10, x FOR PEER REVIEW 23 of 28

Figure 15 shows the congestion window graphs of MPTCP-CC-LIA. The behavior is aggressive on both subflows in slow start phase. After detecting congestion, MPTCP shifts more data to subflow with less congested path. The behavior of congestion window on less congested path (Path 1) is very aggressive, thus, increasing packet drop rate in addition to throughput. On more congested path (Path 2) the congestion window is decreased to half on packet loss detection and almost goes to zero on the next packet loss event, thus, avoiding packet drop rate but decreasing throughput also. The aim of the coupled congestion control algorithm LIA of MPTCP is to ensure fairness in the case of a common bottleneck path shared by multiple subflows of the same multipath flow. By using the total congestion window of multiple subflows as a factor in Equations (15) and (16) of Section 3.3 to calculate the increase of congestion window of a single subflow, MPTCP has successfully achieved Goal 2 [29], i.e., a multipath TCP should not take up more capacity than a single TCP flow in the case of a shared bottleneck link and Goal 3, i.e., a multipath flow should shift its traffic to the less congested link as much as possible. But to achieve these two goals there is also a decrease in the overall throughput due to the underutilization of available bandwidth. The probability of a common bottleneck link shared by multiple subflows is very low. In the case of a shared bottleneck link, the behavior of a coupled congestion control is justified but, in the absence of a shared bottleneck, it only results in underutilization of bandwidth. The congestion window plots of our proposed model, MPTCP-MFVL, are also shown in Figure 16. MVegas is used to control congestion window of subflow with more congested path (Path 2) and MFast is used to control congestion window of subflow on less congested path (Path 1), thus, inceasing the aggregate goodput of both subflows and decreasing aggregate packet loss rate as compared with MPTCP-CC-LIA. The congestion window of MPTCP-CC-LIA increases up to a maximum size of 100 packets on less congested path which shows its behavior is more aggressive as compared with the congestion window of our proposed MFVL scheme on less congested path Electronics 2021, 10, 711 22 of 27 which goes up to 80 only at the start and for the remaining period only has a maximum value of 75. We achieved this by using our modified version of Fast TCP on less congested path. The original version of Fast TCP is much more aggressive and unfair towardscommon other bottleneck TCP linkflows; shared however, by multiple our modified subﬂows version is very low.shows In theits caseless ofaggressive a shared behavior,bottleneck as link, the theresults behavior prove of in a the coupled congesti congestionon window control graph is justiﬁedand this but,behavior in the shows absence it isof not a shared harmful bottleneck, to other itTCP only flows. results in underutilization of bandwidth.

Electronics 2021, 10, x FOR PEER REVIEW 24 of 28

We used the modified version of TCP Vegas on the more congested path. The Figurechanges 15. wereCongestion made window in the original plots of MPTCP-CC-LIA TCP Vegas code for to Setup make 3. it a bit aggressive. The congestion window graph, in Figure 16, shows that our proposed MFVL algorithm is able to use theThe congested congestion path window bandwidth plots of very our proposedefficiently model, by not MPTCP-MFVL, reducing the congestion are also shown win- indow Figure close 16 to. MVegas zero as iscompared used to control with MPTCP- congestionCC-LIA window that of almost subflow reduces with more its window congested to pathzero (Pathon the 2) congested and MFast link, is used as shown to control in Figu congestionre 15. Therefore, window of our subflow algorithm on less has congested success- pathfully (Pathused the 1), thus,less congested inceasing path the aggregatefor more traffic goodput and of the both more subflows congested and path decreasing for less aggregatetraffic, thus, packet achieving loss rate throughput as compared and withavoiding MPTCP-CC-LIA. packet drop at the same time.

FigureFigure 16.16. CongestionCongestion window window plots plots of of the the proposed proposed model model MFVL MFVL HCCA HCCA (MPTCP-MFVL) (MPTCP-MFVL) for Setup for 3. Setup 3. The congestion window of MPTCP-CC-LIA increases up to a maximum size of 1004.4. packetsSetup Four on less congested path which shows its behavior is more aggressive as compared with the congestion window of our proposed MFVL scheme on less congested path In our final experiment, the CBR data rate was varied from 0.1 Mbps to 2 Mbps and which goes up to 80 only at the start and for the remaining period only has a maximum the results were plotted. The average goodput and average packet drop rate with CBR 2 value of 75. We achieved this by using our modified version of Fast TCP on less congested Mbps are shown in Figures 17 and 18 respectively. For different values of CBR ranging path. The original version of Fast TCP is much more aggressive and unfair towards other from 0.1 Mbps to 2 Mbps, the average goodput and packet drop rate are plotted in Figure TCP flows; however, our modified version shows its less aggressive behavior, as the results 19 and Figure 20 respectively. The results show that our proposed model MPTC-MFVL prove in the congestion window graph and this behavior shows it is not harmful to other can achieve up to 30% higher average goodput and decrease average packet loss up to 50% TCP flows. than standard multipath TCP with coupled congestion control MPTCP-CC-LIA. Hence, in uncoupled mode, our algorithm has successfully achieved the goal of increasing goodput, decreasing packet drop, balanced congestion, and less aggressive increase of congestion windows show its harmless behavior to other TCP flows. To achieve the goal of fairness in the case of shared bottleneck, we propose to use MPTC-CC-LIA in coupled mode for our algorithm only after detecting shared bottleneck link. The simulation results are not shown for coupled mode as we used only the existing algorithms without any modifications.

Electronics 2021, 10, 711 23 of 27

We used the modified version of TCP Vegas on the more congested path. The changes were made in the original TCP Vegas code to make it a bit aggressive. The congestion window graph, in Figure 16, shows that our proposed MFVL algorithm is able to use the congested path bandwidth very efficiently by not reducing the congestion window close to zero as compared with MPTCP-CC-LIA that almost reduces its window to zero on the congested link, as shown in Figure 15. Therefore, our algorithm has successfully used the less congested path for more traffic and the more congested path for less traffic, thus, achieving throughput and avoiding packet drop at the same time.

4.4. Setup Four In our final experiment, the CBR data rate was varied from 0.1 Mbps to 2 Mbps and the results were plotted. The average goodput and average packet drop rate with CBR 2 Mbps are shown in Figures 17 and 18 respectively. For different values of CBR ranging from 0.1 Mbps to 2 Mbps, the average goodput and packet drop rate are plotted in Figures 19 and 20 respectively. The results show that our proposed model MPTC-MFVL can achieve up to 30% higher average goodput and decrease average packet loss up to 50% than standard multipath TCP with coupled congestion control MPTCP-CC-LIA. Hence, in uncoupled mode, our algorithm has successfully achieved the goal of increasing goodput, decreasing packet drop, balanced congestion, and less aggressive increase of congestion windows show its harmless behavior to other TCP flows. To achieve the goal of fairness in the case of shared bottleneck, we propose to use MPTC-CC-LIA in coupled mode for our Electronics 2021, 10, x FOR PEER REVIEWalgorithm only after detecting shared bottleneck link. The simulation results25 of 28 are not shown for coupled mode as we used only the existing algorithms without any modifications. Electronics 2021, 10, x FOR PEER REVIEW 25 of 28

Figure 17. 17. AverageAverage goodput goodput result result with CBR with data CBR rate data of 2 rateMbps. of 2 Mbps. Figure 17. Average goodput result with CBR data rate of 2 Mbps.

Figure 18.18. 18. Average AverageAverage packet packet packet drop drop rate droprate with with rate CBR CBR with data data rate CBR rate of data2 of Mbps. 2 Mbps. rate of 2 Mbps.

Figure 19. Average goodput result with CBR data rate 0.1 to 2 Mbps. Figure 19. Average goodput result with CBR data rate 0.1 to 2 Mbps.

Electronics 2021, 10, x FOR PEER REVIEW 25 of 28

Figure 17. Average goodput result with CBR data rate of 2 Mbps.

Electronics 2021, 10, 711 24 of 27

Figure 18. Average packet drop rate with CBR data rate of 2 Mbps.

Electronics 2021, 10, x FOR PEER REVIEW 26 of 28

FigureFigure 19. 19. AverageAverage goodput goodput result result with with CBR CBR data data rate rate 0.1 0.1 to to 2 2 Mbps. Mbps.

FigureFigure 20. 20. AverageAverage packet packet loss loss rate rate with with CBR CBR data data rate rate 0.1 0.1 to to 2 2 Mbps. Mbps. 5. Conclusions 5. Conclusions Modern IoT devices are equipped with high-speed processors and large on-chip Modern IoT devices are equipped with high-speed processors and large on-chip memories. The technological advancements in these IoT devices and networks require the memories. The technological advancements in these IoT devices and networks require design of new intelligent algorithms and reliable protocols. With the development of new the design of new intelligent algorithms and reliable protocols. With the development of low power standards such as 802.11 ah (Wi-Fi HaLow) for the Internet of Things, now, high new low power standards such as 802.11 ah (Wi-Fi HaLow) for the Internet of Things, data rates over long distances are possible. now, Inhigh this data study, rates we over have long presented distances the are case possible. of an IoT-based smart gas grid with multi- homedIn this smart study, gas meters.we have The presented information the sentcase byof smartan IoT-based meters issmart used gas for loadgrid forecasting,with mul- tihomedtherefore smart reliable gas transmission meters. The of information data is required sent withby smart efficient meters congestion is usedcontrol for load scheme. forecasting,To reduce therefore packet reliable retransmissions transmission and increaseof data is goodput, required we with proposed efficient a novelcongestion congestion controlcontrol scheme. algorithm To reduce for MPTCP. packet Our retransmis algorithmsions operates and increase in uncoupled goodput, and we coupled proposed modes. a novelIn an congestion uncoupled control mode, modifiedalgorithm versions for MPTCP. of the Our delay-based algorithm Fast operates TCP and in uncoupled TCP Vegas andcongestion coupled controlmodes. algorithms In an uncoupled are used. mode, We mo madedified changes versions to theof the Fast delay-based TCP algorithm, Fast TCPso that and theTCP congestion Vegas congestion window control is updated algorithms to be are less used. aggressive We made and changes packet to drops the Fast are TCPdecreased. algorithm, The so modifications that the congestion in TCP window Vegas byis updated introducing to be increasing less aggressive factors and for packet alpha dropsand beta, are decreased. result in better The modifications goodput and in low TCP packet Vegas dropsby introducing over the congestedincreasing factors path. Ourfor alphaalgorithm and beta, detects result path in better congestion goodput by recordingand low packet packet drops retransmissions over the congested for each path. subflow Our algorithmand selects detects the congestion path congestion control by algorithm recording modified packet retransmissions Fast (MFast)for forless each congested subflow andpath selects and modified the congestion Vegas control (MVegas) algorithm for more modified congested Fast path. (MFast) In simulation for less congested experiments, path and modified Vegas (MVegas) for more congested path. In simulation experiments, we have shown that MFast is better suited for less congested path and MVegas for more congested path. Our algorithm uses a shared bottleneck detection method and on detecting shared bottleneck, switches to coupled congestion mode. In coupled congestion mode, we used the same LIA algorithm as MPTCP. As our main contribution lies in the design of uncoupled mode for our algorithm, therefore, we compared the simulation results of our algorithm in uncoupled mode with the default MPTCP coupled congestion control algorithm LIA. The results prove that our algorithm achieves better goodput and reduces packet loss. Packet retransmissions consume extra power; with reduced packet retransmissions our algorithm is extremely beneficial for battery powered IoT devices. For implementation on embedded platforms, we plan to analyze our design further for improvements in reducing the extra overhead associated with TCP, efficient memory utilization, and power saving by using sleep modes.

Author Contributions: Conceptualization, M.U.M.; methodology, M.U.M.; software, M.U.M.; validation, M.U.M.; formal analysis, M.U.M.; investigation, M.U.M.; resources, M.U.M.; data curation, M.U.M.; writing—original draft preparation, M.U.M.; writing—review and editing, M.U.M.;

Electronics 2021, 10, 711 25 of 27

we have shown that MFast is better suited for less congested path and MVegas for more congested path. Our algorithm uses a shared bottleneck detection method and on detecting shared bottleneck, switches to coupled congestion mode. In coupled congestion mode, we used the same LIA algorithm as MPTCP. As our main contribution lies in the design of uncoupled mode for our algorithm, therefore, we compared the simulation results of our algorithm in uncoupled mode with the default MPTCP coupled congestion control algorithm LIA. The results prove that our algorithm achieves better goodput and reduces packet loss. Packet retransmissions consume extra power; with reduced packet retransmissions our algorithm is extremely beneﬁcial for battery powered IoT devices. For implementation on embedded platforms, we plan to analyze our design further for improvements in reducing the extra overhead associated with TCP, efﬁcient memory utilization, and power saving by using sleep modes.

Author Contributions: Conceptualization, M.U.M.; methodology, M.U.M.; software, M.U.M.; validation, M.U.M.; formal analysis, M.U.M.; investigation, M.U.M.; resources, M.U.M.; data curation, M.U.M.; writing—original draft preparation, M.U.M.; writing—review and editing, M.U.M.; visual- ization, M.U.M. and M.I.B.; supervision, M.I.B.; project administration, M.I.B. All authors have read and agreed to the published version of the manuscript. Funding: This research received no external funding. Acknowledgments: The authors would like to thank Adeel Akram and Jamil Khan for their expert suggestions throughout the research work. Conﬂicts of Interest: The authors declare no conﬂict of interest.

References 1. IoT Active Devices. Available online: https://www.statista.com/statistics/1101442/iot-number-of-connected-devices- worldwide/ (accessed on 25 January 2021). 2. Leccese, F. Remote-Control System of High Efficiency and Intelligent Street Lighting Using a ZigBee Network of Devices and Sensors. IEEE Trans. Power Deliv. 2013, 28, 21–28. [CrossRef] 3. Leccese, F.; Cagnetti, M.; Trinca, D. A Smart City Application: A Fully Controlled Street Lighting Isle Based on Raspberry-Pi Card, a ZigBee Sensor Network and WiMAX. Sensors 2014, 14, 24408–24424. [CrossRef][PubMed] 4. Rathore, M.M.; Ahmad, A.; Paul, A. IoT-based smart city development using big data analytical approach. In Proceedings of the 2016 IEEE International Conference on Automatica (ICA-ACCA), Curico, Chile, 19–21 October 2016; pp. 1–8. [CrossRef] 5. Arasteh, H.; Hosseinnezhad, V.; Loia, V.; Tommasetti, A.; Troisi, O.; Shafie-Khah, M.; Siano, P. Iot-based smart cities: A survey. In Proceedings of the 2016 IEEE 16th International Conference on Environment and Electrical Engineering (EEEIC), Florence, Italy, 7–10 June 2016; pp. 1–6. [CrossRef] 6. Industry Elite Gathered at “Smart Gas Network Management Conference” in Suzhou. Available online: https://www. towngaschina.com/getattachment/d00072ae-e6b9-4716-a1b8-9d8f16e2cfba/Industry-Elite-gathered-at-Smart-Gas-Network- Management-Conference-in-Suzhou.pdf.aspx?lang=en-US&ext=.pdf (accessed on 20 January 2020). 7. NB-IoT Smart Gas Solution White Paper. Available online: https://www-file.huawei.com/-/media/CORPORATE/PDF/News/ NB-IoT-Smart-Gas-Solution-EN.pdf?la=en (accessed on 1 January 2020). 8. The Role of Smart Gas in the Smart City. Available online: https://ucononline.com/magazine/2018/june-2018-vol-73-no-6/ features/the-role-of-smart-gas-in-the-smart-city#:~{}:text=Withsmarttechnologysolutions%2Cgas,timedatawithintheirnetworks (accessed on 1 January 2020). 9. Khan, W.Z.; Aalsalem, M.Y.; Khan, M.K.; Hossain, M.S.; Atiquzzaman, M. A reliable Internet of Things based architecture for oil and gas industry. In Proceedings of the International Conference on Advanced Communication Technology (ICACT), PyeongChang, Korea, 19–22 February 2017; pp. 705–710. [CrossRef] 10. Gomes, J.B.A.; Rodrigues, J.J.P.C.; Rabêlo, R.A.L.; Kumar, N.; Kozlov, S. IoT-Enabled Gas Sensors: Technologies, Applications, and Opportunities. J. Sens. Actuator Netw. 2019, 8, 57. [CrossRef] 11. Wei, N.; Li, C.; Duan, J.; Liu, J.; Zeng, F. Daily Natural Gas Load Forecasting Based on a Hybrid Deep Learning Model. Energies 2019, 12, 218. [CrossRef] 12. Smart Gas Meter. Available online: https://www.st.com/en/applications/metering/gas-metering.html (accessed on 5 March 2020). 13. Mujeeb, S.; Javaid, N.; Akbar, M.; Khalid, R.; Nazeer, O.; Khan, M. Big Data Analytics for Price and Load Forecasting in Smart Grids. Innov. Data Commun. Technol. Appl. 2018, 25, 77–87. 14. Lloret, J.; Tomas, J.; Canovas, A.; Parra, L. An Integrated IoT Architecture for Smart Metering. IEEE Commun. Mag. 2016, 54, 50–57. [CrossRef] Electronics 2021, 10, 711 26 of 27

15. Baños-Gonzalez, V.; Afaqui, M.S.; Lopez-Aguilera, E.; Garcia-Villegas, E. IEEE 802.11ah: A Technology to Face the IoT Challenge. Sensors 2016, 16, 1960. [CrossRef] 16. Ahmed, N.; Rahman, H.; Hussain, I. A comparison of 802.11ah and 802.15.4 for IoT. ICT Express 2016, 2, 100–102. [CrossRef] 17. Al-Kashoash, H.A.; Hassen, F.; Kharrufa, H.; Kemp, A.H. Analytical modelling of congestion for 6LoWPAN networks. ICT Express 2018, 4, 209–215. [CrossRef] 18. Gomez, C.; Arcia-Moret, A.; Crowcroft, J. TCP in the Internet of Things: From Ostracism to Prominence. IEEE Internet Comput. 2018, 22, 29–41. [CrossRef] 19. Raj, K.; Smys, S. Iot Based Multi-Homing Applications—A Review. IRO J. Sustain. Wirel. Syst. 2019, 01, 31–41. [CrossRef] 20. Hwang, J.; Yoo, J. A Memory-Efﬁcient Transmission Scheme for Multi-Homed Internet-of-Things (IoT) Devices. Sensors 2020, 20, 1436. [CrossRef][PubMed] 21. Sethi, P.; Sarangi, S.R. Internet of Things: Architectures, Protocols, and Applications. J. Electr. Comput. Eng. 2017, 2017, 1–25. [CrossRef] 22. Karagiannis, V.; Chatzimisios, P.; Vazquez-Gallego, F.; Alonso-Zarate, J. A survey on application layer protocols for the internet of things. Trans. IoT Cloud Comput. 2015, 3, 11–17. 23. Yibo, C.; Hou, K.-M.; Zhou, H.; Shi, H.-L.; Liu, X.; Diao, X.; Ding, H.; Li, J.-J.; De Vaulx, C. 6LoWPAN stacks: A survey. In Proceedings of the 2011 7th International Conference on Wireless Communications, Networking and Mobile Computing, Wuhan, China, 23–25 September 2011. [CrossRef] 24. AMNESIA:33. Available online: https://www.forescout.com/research-labs/amnesia33/ (accessed on 5 January 2021). 25. Khan, S.; Zeeshan, M. Performance and Throughput Analysis of IEEE 802.11ah for Multiband Multimode Operation. In Proceedings of the 2018 21st International Symposium on Wireless Personal Multimedia Communications (WPMC), Chiang Rai, Thailand, 25–28 November 2018. [CrossRef] 26. Wi-Fi HaLow. Available online: https://enterpriseiotinsights.com/20200724/channels/fundamentals/hello-wiﬁ-halow-the- new-tech-for-low-power-mid-range-iot (accessed on 1 January 2021). 27. Balan, T.; Robu, D.; Sandu, F. Multihoming for Mobile Internet of Multimedia Things. Mob. Inf. Syst. 2017, 2017, 1–16. [CrossRef] 28. Ji, R.; Cao, Y.; Fan, X.; Jiang, Y.; Lei, G.; Ma, Y. Multipath TCP-Based IoT Communication Evaluation: From the Perspective of Multipath Management with Machine Learning. Sensors 2020, 20, 6573. [CrossRef][PubMed] 29. Nikravesh, A.; Guo, Y.; Qian, F.; Mao, Z.M.; Sen, S. An in-depth understanding of multipath TCP on mobile devices: Measurement and system design. In Proceedings of the Annual International Conference on Mobile Computing and Networking, MOBICOM, New York, NY, USA, 3–7 October 2016; pp. 189–201. [CrossRef] 30. Cao, Y.; Xu, M.; Fu, X. Delay-based congestion control for multipath TCP. In Proceedings of the International Conference on Network Protocols, ICNP, Austin, TX, USA, 30 October–2 November 2012. [CrossRef] 31. RFC 6356 Coupled Congestio Control. Available online: https://tools.ietf.org/html/rfc6356 (accessed on 5 August 2020). 32. Hsieh, H.Y.; Sivakumar, R. PTCP: An end-to-end transport layer protocol for striped connections. In Proceedings of the 10th IEEE International Conference on Network Protocols, Paris, France, 12–15 November 2002. [CrossRef] 33. Hsieh, H.Y.; Sivakumar, R. A transport layer approach for achieving aggregate bandwidths on multi-homed mobile hosts. Wirel. Netw. 2005, 11, 99–114. [CrossRef] 34. Zhang, M.; Lai, J.; Krishnamurthy, A.; Peterson, L.; Wang, R. A transport layer approach for improving end-to-end performance and robustness using redundant paths. In Proceedings of the FREENIX Track: 2004 USENIX Annual Technical Conference, Boston, MA, USA, 27 June–2 July 2004. 35. Iyengar, J.; Amer, P.; Stewart, R. Concurrent Multipath Transfer Using SCTP Multihoming Over Independent End-to-End Paths. IEEE/ACM Trans. Netw. 2006, 14, 951–964. [CrossRef] 36. Peng, Q.; Walid, A.; Hwang, J.; Low, S.H. Multipath TCP: Analysis, Design, and Implementation. IEEE/ACM Trans. Netw. 2014, 24, 596–609. [CrossRef] 37. Khalili, R.; Gast, N.; Popovic, M.; Le Boudec, J.-Y. MPTCP Is Not Pareto-Optimal: Performance Issues and a Possible Solution. IEEE/ACM Trans. Netw. 2013, 21, 1651–1665. [CrossRef] 38. Lubna, T.; Mahmud, I.; Cho, Y.-Z. D-LIA: Dynamic congestion control algorithm for MPTCP. ICT Express 2020, 6, 263–268. [CrossRef] 39. Ha, B.; Tran, B.; Le, T.; Tran, C. Multipath FAST TCP for Large Bandwidth-Delay Product Networks. In Proceedings of the International Conference on Green and Human Information Technology (ICGHIT), Ho Chi Minh City, Vietnam, 12–14 February 2014; pp. 228–231. 40. Pham, L.; Vo, P.L.; Le, T.A. An energy-aware multipath congestion control protocol for mobile devices. In Proceedings of the 2017 International Conference on Recent Advances in Signal Processing, Telecommunications & Computing, SigTelCom 2016, Da Nang, Vietnam, 9–11 January 2017; pp. 44–48. [CrossRef] 41. Brakmo, L.S.; Peterson, L.L. Tcp Vegas: End-to-end congestion avoidance on global Internet. IEEE J. Sel. Areas Commun. 1995, 13, 1465–1480. [CrossRef] 42. Jain, N.; Srivastava, N. TCP I-Vegas in Mobile-IP Network. Int. J. Adv. Comput. Sci. Appl. 2014, 5, 142–147. [CrossRef] 43. Srijith, K.N.; Jacob, L.; Ananda, A.L. TCP Vegas-A: Solving the fairness and rerouting issues of TCP Vegas. In Proceedings of the IEEE International Performance, Computing, and Communications Conference, Phoenix, AZ, USA, 9–11 April 2003; pp. 309–316. [CrossRef] Electronics 2021, 10, 711 27 of 27

44. Jin, C.; Wei, D.X.; Low, S.H. FAST TCP: Motivation, architecture, algorithms, performance. In Proceedings of the IEEE INFOCOM, Hong Kong, China, 7–11 March 2004; Volume 4, pp. 2490–2501. [CrossRef] 45. Mudassir, M.U.; Akram, A. Performance evaluation of FAST TCP trafﬁc-ﬂows in multihomed MANETs. Commun. Comput. Inf. Sci. 2010, 120, 450–458. [CrossRef] 46. Multipath TCP Patch for NS2. Available online: https://code.google.com/archive/p/multipath-tcp/downloads (accessed on 5 March 2020).