Effects on TCP of Routing Strategies in Satellite Constellations

SATELLITE-BASED INTERNET TECHNOLOGY AND SERVICES Effects on TCP of Routing Strategies in Satellite Constellations

Lloyd Wood, George Pavlou, and Barry Evans, University of Surrey

ABSTRACT number of satellites required to cover the Earth. A lower altitude decreases free space A broadband satellite network uses a constella- loss and propagation delay, but means that the tion of a number of similar satellites to provide service each satellite can offer is limited to wireless networking services to the Earth. A users in a smaller visible area of the ground number of these constellation networks are (the satellite’s footprint). To fully cover the under development. This article introduces the globe, more satellites are needed. This increas- types of satellite constellation networks, and es frequency reuse and overall system capacity, examines how overall performance of TCP com- but will also increase overall system construc- munications carried across such a network can tion and maintenance costs. Satellites at lower be affected by the choice of routing strategies altitudes must move faster relative to the used within the network. Constellations utilizing ground to stay in their orbits, increasing the rate direct intersatellite links are capable of using of handoff and Doppler effects between termi- multiple paths between satellites simultaneously nals and satellites. as a strategy to spread network load. This allows Most proposed systems use circular orbits more general routing strategies than shortest- with constant altitudes, since this means that path routing, but we show these strategies to be satellite overhead pass times and power levels detrimental to the performance of individual needed for communication are constant, and TCP connections. each satellite can be used for traffic throughout its orbit. Other systems have been proposed NTRODUCTION using multiple elliptical orbits, where satellites I are only intended to be available for service Broadband satellite constellation networks have while moving relatively slowly around high-alti- been proposed and are now being developed. tude apogee (e.g., the commercial Ellipso and These will provide medium- and high-capacity Pentriad proposals). In those elliptical systems wireless data services, and will interconnect with the communication payloads are unused existing telco backbones and the Internet. Once throughout the rest of the orbit. launched and operational, proposed networks, Figure 1 compares the altitudes of the sys- such as Teledesic, Spaceway, and Skybridge, can be tems discussed above. expected to transport significant amounts of TCP traffic between ground endhosts on the Internet. EOMETRY OPOLOGY AND ELAY Although the use of satellite constellations G , T , D for delivering broadband IP services is new, From a networking viewpoint, there are two fun- satellite-constellation-based systems are already damentally different approaches taken by these being used for applications such as providing: broadband constellations. Each constellation • Navigation and position information: Glob- consists of a number of similar satellites, where al Positioning System (GPS) and the satellite design is based on one of the follow- GLONASS ing approaches: • Voice telephony and low-bit-rate communi- • Each satellite is a space-based retransmitter cation services (although not financially suc- of traffic received from user terminals and cessful, the Iridium and Globalstar systems local gateways below it on the ground, have made service available, and ICO Glob- returning the traffic to the ground. This al Communications is to follow) allows isolated user terminals to exchange • Low-bit-rate messaging data services for a traffic with nearby ground stations that are variety of applications (e.g., Orbcomm) gateways into the terrestrial network. The The altitude of the satellites in the constella- satellite forms a wireless last hop to an tion is a significant factor in determining the extensive ground network. Commercial sys-

172 0163-6804/01/$10.00 © 2001 IEEE IEEE Communications Magazine • March 2001 tems taking this approach include Global- star and many geostationary satellites, and the planned ICO and Skybridge. Molnya elliptical orbit • Each satellite is a network switch that is also Pentriad, Russian television able to communicate with neighboring satel- (useful at apogee) lites by using radio or laser intersatellite links (ISLs). This allows user terminals on the ground below the satellite to exchange traffic Global Positioning with gateways to the terrestrial network or System (GPS) users below distant satellites, without requir- Glonass ing a local gateway or significant terrestrial infrastructure to do so. Commercial systems Teledesic, Skybridge, utilizing ISLs include the deployed Iridium Globalstar and the proposed Teledesic, Hughes Space- way, and Astrolink networks. Proposals with circular orbits are the most Concordia Earth Ellipso common, while proposals using ISLs are the Borealis most interesting and complex from a networking Iridium, Orbcomm and systems viewpoint, so we will focus on those. Low Earth Orbit (LEO) Geostationary satellites can be interconnected to form a simple ring network around the ICO, Earth’s equator, as shown in Fig. 2a for the min- Spaceway NGSO imum three geostationary satellites needed to Medium Earth Orbit (MEO) cover most of the Earth’s surface. At low- and medium-earth-orbit altitudes the ISLs between Geostationary orbital ring (GEO) Spaceway, Astrolink, DirecPC, VSAT, satellites form a more complex dynamic mesh 0 10,000 kilometres Inmarsat, Intelsat, television etc. topology. That topology is a variation of the regions of Van Allen radiation belt peaks (high-energy protons) Manhattan network, named after the grid of orbits are not shown at actual inclination; this is a guide to altitude only streets in Manhattan, which is well-known in theoretical computer networking for use in par- Figure 1. Orbits used by satellite constellations. allel computer architectures. Depending on the specific orbital geometry chosen, this overall satellite topology will be either: ground terminals over the course of a day, as • A fully toroidal network, created from an the Earth rotates beneath the moving satellites. inclined Walker delta or Ballard rosette con- Here, we see smooth changes in delay due to stellation [1], where the ascending (moving the orbital motion of satellites, as well as abrupt northward) and descending (moving south- changes due to handoffs and path changes; ward) planes overlap and span the full 360˚ these changes are small compared to the granu- of longitude; for example, the Hughes larity of a TCP timer. Spaceway NGSO MEO proposal (Fig. 2b). Network latency results from queuing, switch- This geometry offers the best satellite visi- ing, and link-layer medium access control (MAC) bility and highest degree of satellite capaci- bandwidth allocation as well as from serialization ty over the populated mid-latitudes. and propagation delays. With the exception of the • A form of cylindrical mesh network, from a total path propagation delay shown in Fig. 3i, near-polar Walker star constellation [2], where the speed of light is a common constraint, where the interconnected ascending and these contributing delays will vary considerably descending orbital planes of satellites each according to the specific system design. As a first cover around 180˚ of longitude and are approximation, these delays can be thought of as separate. Systems using this approach proportional to the number of wireless links tra- include Iridium and the proposed Teledesic versed, indicated in Fig. 3ii. (Teledesic’s redun- system (a proposed Teledesic design is dant “geodesic” eight-link design, shown in Fig. shown in Fig. 2c). This geometry offers the 2c, minimizes delay and the number of links tra- best satellite visibility at the poles. versed, while also providing the two ISLs needed In a near-polar star constellation, the gap for frequent smooth acquisition and handoff of a between the counter-rotating ascending and single link across the seam.) descending planes, shown in the Teledesic Examining the shortest-path propagation delay design in Fig. 2c, is known as the counter-rotat- across a network provides a best-effort goal with ing seam. Cross-seam ISLs, which interconnect which real-world performance could be compared. satellites at the edges of the cylindrical mesh to However, relying on a metric of minimum propa- form a seamless spherical network, must be gation delay for cost-based routing will tend to handed off frequently as the planes of satellites load interplane links at the highest latitudes, where move past each other. These cross-seam links distances between satellites are shortest. have not yet been demonstrated in practice; Without cross-seam links, the seam increases Iridium, the only system constructed using ISLs both delay and the number of links traversed to date, has none. when it intersects the shortest path between However, cross-seam ISLs do have a consid- ground terminals. The seam alters the shortest erable effect on network performance. Figure 3 path between the terminals so that the path shows the total propagation delay experienced lengthens and lies over the highest latitudes. The by traffic taking the shortest path across the duration of this disruption is proportional to constellation networks between two fixed separation in longitude between the terminals,

IEEE Communications Magazine • March 2001 173 a) Minimum three-satellite Clarke GEO satellite network Unlike the with ISLs, showing simulated ground terminals “polar star” 90 constellations, 60 London rosette Tokyo constellations can 30 also offer two 0 Quito distinct sets of 312

paths between Unprojected latitude –30 terminals, depending on –60

whether the –90 terminals’ uplink –180 –150 –120 –90 –60 –30 0 30 60 90 120 150 180 Unprojected longitude and downlink b) Spaceway NGSO MEO satellite network with ISLs at a point in time, showing simulated ground terminals communications ascending (moving north) and descending (moving south) satellites overlap have been 90 allocated to 60 3b 4b 1b 2b ascending or London Tokyo descending 30 2c 3c 4c 1c

satellites. 3a 4a 1a 2a 0 Quito

Unprojected latitude –30 1d 2d 3d 4d

4e 1e 2e 3e

–60

–90 –180 –150 –120 –90 –60 –30 0 30 60 90 120 150 180 Unprojected longitude

c) Teledesic LEO satellite network (Boeing 288 satellite design, optimized coverage), showing simulated ground terminals Descending satellites (moving south) Ascending satellites (moving north) 90 7g 9g 11g 1g 3g 5g 4g 12f 6g 8g 10g 12g 2f 4f 6f 8f 2g 10f 3h 5h 7h 9h 11h Seam 1f 3f 5f 7f 9f 1h 11f 2h 4h 6h 8h 10h 12h 2e 4e 6e 8e 10e 12e 3i 5i 7i 9i 11i 1e 3e 5e 7e 9e 11e 1i 60 2i 4i 6i 10i 12i L. 2d 4d 6d 8d 10d 12d 3j 5j 7j 9j 11j 1d 3d 5d 7d 9d 11d 1j 2j 4j 6j 8j 10j 12j 2c 4c 6c 8c 10c 12c 3k 5k 7k 9k 11k 1c 3c 5c 7c 9c T. 11c 1k 30 2k 4k 6k 8k 10k 12k 2b 4b 6b 8b 10b 12b 3l 5l 7l 9l 11l 1b 3b 5b 7b 9b 11b 1l 2l 4l 6l 8l 10l 12l 2a 4a 6a 8a 10a 12a 0 1m 3m 5m 7m 9m 11m 1a 3a 5a 7a 9a 11a 2m 4m 6m Q. 8m 10m 12m 2x 4x 6x 8x 10x 12x 1n 3n 5n 7n 9n 11n 1x 3x 5x 7x 9x 11x 2n 4n 6n 8n 10n 12n 2w 4w 6w 8w 10w 12w

Unprojected latitude 1o 3o 5o 7o 9o 11o 1w 3w 5w 7w 9w 11w –30 2o 4o 6o 8o 10o 12o 2v 4v 6v 8v 10v 12v 1p 3p 5p 7p 9p 11p 1v 3v 5v 7v 9v 11v 2p 4p 6p 8p 10p 12p 2u 4u 6u 8u 10u 12u 1q 3q 5q 7q 9q 11q 1u 3u 5u 7u 9u 11u –60 2q 4q 6q 8q 10q 12q2t 4t 6t 8t 10t 12t 1r 3r 5r 7r 9r 1t 11r 3t 5t 7t 9t 11t 2r 4r 6r 8r 2s 10r 4s 12r 6s 8s 10s 12s 1s 3s 5s 7s 9s 11s Seam –90 –180 –150 –120 –90 –60 –30 0 30 60 90 120 150 180 Unprojected longitude

Terminal-satellite uplinks/downlinks are handed off at MEO and LEO. Intraplane ISLs are permanent and are never handed off. Interplane ISLs may break as satellites near each other at speed, and may be handed off near highest latitudes as planes cross. Cross-seam ISLs are temporary regularly handed off as satellites pass. Thickest ISLs show shortest path from Quito to Tokyo. Less thick ISLs show alternate paths with same hop metric.

Figure 2. Satellite constellation networks at different altitudes.

174 IEEE Communications Magazine • March 2001 0.600 0.600

0.500 0.500

0.400 0.400

0.300 0.300

0.200 0.200

0.100 0.100 Shortest-path propagation delay time (s) 0.000 Shortest-path propagation delay time (s) 0.000 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 Time (h) Time (h) i. One-way propagation delay showing smooth satellite motion and abrupt handoffs

14 14 13 13 12 12 11 11 10 10 9 9 8 8 7 7 6 6 5 5 4 4 3 3 2 2 1 1 Number of wireless links (hops) traversed Number of wireless links (hops) traversed 0 0 0 1 2345678910111213141516171819202122230 1234567891011121314151617181920212223 Time (h) Time (h) ii. Hop counts of wireless links traversed showing path changes due to handoffs

(a) (b)

Clarke geostationary ( alternate Quito-3-2-London) Spaceway NGSO Teledesic (seamed) Teledesic (cross-seam links) uplink and downlink delays are included

Figure 3. One-way total propagation delays of path across the constellations over 24 hours: a) routing packets between ground terminals from Quito to London; b) routing packets between ground terminals from Quito to Tokyo. due to the Earth’s rotation. Similarly, the size of takes place only when the currently used satel- the change in path is inversely related to the lite passes below the terminal’s minimum eleva- separation between terminals. As Quito and tion angle threshold. If the terminal at Quito in London are relatively close, traffic between Fig. 2b was allocated to the descending satellites them, shown in Fig. 3a, shows this disruption of 2c or 2d, it would be only one or two ISL more clearly than traffic between Quito and hops from Tokyo, instead of three while using Tokyo (Fig. 3b). Cross-seam links reduce the satellite 4a. maximum propagation delay, path lengths, path changes and visible end-to-end disruption of net- AND work traffic in star constellations. TCP IP Rosette constellations with ISLs, such as the TCP/IP is the protocol suite on which the Inter- MEO Spaceway NGSO network shown in Fig. net depends. IP provides a connectionless data- 2b, offer more regular repeating routing and gram service that provides enough header delay patterns over time between terminals. information for packets to be routed to their Unlike the “polar star” constellations, rosette destinations. TCP makes reliable ordered com- constellations can also offer two distinct sets of munication of streams of data such as files possi- paths between terminals, depending on whether ble by implementing a duplex protocol, based the terminals’ uplink and downlink communica- around sliding windows, on top of IP [3]. tions have been allocated to ascending or A number of intertwined algorithms deter- descending satellites. The satellites forming the mine the rate at which TCP puts new data into shortest set of ISL paths for the traffic may not the network, and how losses are detected and be selected by the terminal if the terminal is at handled. In particular, a TCP sender maintains a high latitude and does not have a wide choice a congestion window variable that determines of satellites, if demand for link capacity around how large the size of its sliding window of unac- a terminal is high, preventing a choice of the knowledged data given to the network can be at “optimum” satellite for that traffic, or if handoff any point in time. This variable is increased over is not coordinated for traffic across the entire time from start, or after congestion losses, as network but is a local terminal decision. This is the TCP sender probes the network and discov- the case for our simulations, where handover ers that data can be sent with increasing fre-

IEEE Communications Magazine • March 2001 175 quency. It is decreased when losses due to con- meshes, such as the Spaceway NGSO and Given the gestion are inferred by the sender. Slowstart and Teledesic networks shown in Fig. 2, can offer other algorithms control the congestion window more than one possible path for packets to take widespread use and other TCP settings. These are described in between two satellites communicating with of TCP/IP-based detail in [4]. ground terminals. When the path is longer than one ISL hop, there are likely to be a number of applications and equivalent paths, of the same number of hops TCP IN THE between the satellites, whose queuing and fram- interconnection SATELLITE ENVIRONMENT ing delays will be roughly equivalent. If a ground with the terminal has the opportunity to communicate Working across satellite links has been a motiva- with more than one satellite visible to it above terrestrial Inter- tion in the design of TCP since the mid-1970s. its local horizon (using “diversity”), the number net, it is likely TCP treats all losses on the journey between of possible paths between source and destination source and destination as an indication of con- terminals can be increased still further. that broadband gestion, or overflowing packet buffers in router This redundantly connected multipath ISL queues leading to discards. Any losses due to topology allows a change in behavior when net- satellite networks errors in transmission on a less-than-perfect work congestion occurs: rather than dropping will be satellite link, rather than congestion, cause TCP packets at routers instead of queuing them on a to reduce its overall sending rate to help allevi- congested path, the routers can instead divert transporting large ate perceived congestion. those packets to another, less congested, path amounts of traffic Communication between terminals using a instead, introducing load balancing. geostationary satellite has a comparatively long This gives us the opportunity to examine TCP’s generated by round-trip time (RTT), from sender to receiver behavior when exposed to multipath routing in and back, on the order of 0.5 s. This means that the satellite environment. We found that, of TCP’s algorithms. considerable time is needed in slowstart, on TCP’s algorithms, two in particular affect TCP’s opening a new connection for TCP, to increase performance over multipath routing: fast retrans- its throughput to what the link capacity will bear. mit and recovery and delayed acknowledgments. After an errored packet is inferred as lost due to congestion, and the congestion window is reduced, the return to the previous high rate of THE EFFECTS OF throughput is slowed by the large RTT. In both AST ETRANSMIT AND ECOVERY cases, increasing the congestion window takes a F R R number of slow 0.5-s roundtrips, and the capaci- A TCP sender can infer congestion in the net- ty of an expensive satellite link will not be fully work and losses of data transmitted from two used as a result. Satellite considerations for TCP mechanisms: are detailed in [5, 6]. •A timeout, where no new acknowledgments Given the widespread use of TCP/IP-based are received for a calculated period of time, applications and interconnection with the terres- typically a multiple of 0.5 s. The sender trial Internet, it is likely that broadband satellite believes that all traffic sent to the receiver networks will transport large amounts of traffic is being lost due to congestion, and generated by TCP’s algorithms, even if the satel- attempts to avoid contributing to this per- lite networks are not themselves based around ceived congestion by reducing its conges- the IP packet structure. tion window to one segment. It then probes The desire to utilize satellite capacity effi- the network with an exponentially increas- ciently has led to the development of a number ing flow of packets in slowstart, much as it of optimized-for-satellite TCP variants, with did when first sending data when the con- altered transmission control behavior. These run nection was opened. between ground terminals across the satellite • Receipt of a number of duplicate acknowl- link, while other TCP implementations complete edgments, dupacks, where the receiver the end-to-end communication across the terres- sends an acknowledgment which repeats trial Internet. The end-to-end TCP connection the current position of the left edge of the across the congested but relatively link-error- TCP window. The fact that the receiver is free terrestrial Internet and the less congested issuing dupacks indicates it is receiving new but more error-prone satellite link is split into data from the sender — but not the data individual TCP connections tailored to each necessary to be able to deliver previous environment; this is easy to do with an HTTP data received to its application and move proxy cache, but less elegant in implementation the left edge of the window along, allowing when other applications are concerned. the sender to inject more new data into the Improvements in coding on satellite links network. A loss just after the repeatedly have altered the link error characteristics to be indicated window position can therefore be in one of two states: either error-free or extreme- inferred by the sender, which can do a fast ly errored. With a correctly dimensioned satellite retransmission of the lost packet. The link, we can assume that the satellite link is as sender assumes that the loss is due to con- reliable as any terrestrial link. gestion in the network and reduces its congestion window to take this into account. AND ESH OUTING Since the received dupacks indicate that TCP M R data is still getting through to the receiver, Unlike terrestrial networks, where there is often the sender can be less conservative in reduc- only one fixed route between sender and receiving its window to avoid congestion than er at any point in time, MEO and LEO ISL when encountering a timeout. The conges-

176 IEEE Communications Magazine • March 2001 tion window, controlling the rate at which • Selective acknowledgments (SACK), an new data is sent, is halved before growing enhancement to the Reno algorithms that The choice of slowly to its previous size as new data is provides more information in the TCP acknowledged. This is fast recovery. options field of acks sent from receiver to path at each Receipt of three dupacks, after the original sender, allowing the sender to retransmit satellite was ack of the last in-sequence packet received, is specific data indicated as unreceived [9] taken to indicate that a packet has been lost in The graphs in Fig. 4 show the effects of fast determined by a the network. Three is an ideal value for an retransmit and recovery on overall TCP through- ordered flow of packets between sender and put, as a result of dupacks received due to out- simple round- receiver, since it gives the fast warning of con- of-order reception of both segments and acks robin selection, gestion-induced losses for which the retransmit over the multiple paths across the ISL meshes. and recovery processes were named, while cop- In each case, TCP’s throughput, or goodput, is updated on the ing with minor amounts of packet interleaving degraded from its best performance over a single arrival of each and reordering. shortest path by the use of multiple paths, even However, if more than one possible route though there are no losses due to transmission new packet. This between source and destination can be used errors or congestion in the network. simultaneously, as in a mesh of ISLs between By increasing the arbitrary requirement of made each satellites where congestion leads to dynamic three dupacks before entering fast retransmit multipath rerouting rather than straightforward discards, and recovery to a higher threshold, we were packets can be received out of order due to slight gradually able to improve TCP’s throughput to simulation differences in latency between paths. This will approach that when using the single shortest deterministic cause dupacks to be issued even when no losses path. Similarly, lowering the dupack threshold have taken place. Any resulting fast retransmis- decreased even further TCP’s tolerance of mis- and repeatable. sion and recovery will be entirely unnecessary and ordering caused by interleaving packets sent detrimental to TCP’s performance. over multiple paths and lowered its resulting To examine this, we used the network simu- throughput still further. lator ns [7] to simulate multipath routing of Although the mesh topologies of these satel- traffic between Quito and Tokyo across all lite networks provide multiple paths between available routes with the same number of wire- points and are clearly multipath, packet reorder- less hops at a point in time. We simulated ing has been observed in the terrestrial Internet approximations of proposed Teledesic and due to parallelism or load balancing [10], and Spaceway NGSO designs, based on details given can be considered a natural state of affairs rather in their applications to the U.S. FCC for fre- than something rare or easily avoided. quency allocation. SACK’s performance could be improved by We compared shortest-path routing, based on using the dupacks’ ack options field to provide using the path with the smallest propagation more information to indicate what is causing delay, with arbitrary multipath routing, selecting each dupack to be generated, rather than just any minimum-hop path to the destination at a sending a dupack without extra information [11]. point in time. This multipath routing approxi- This can then be used by the sender to infer mates the behavior of temporarily congested reordering in the network and to alter its behav- satellites diverting newly arrived packets from a ior to suit, although this is still a research area. currently congested link onto a less congested alternate path, or simple “hot-potato” local routing decisions. The choice of path at each satellite THE EFFECTS OF was determined by a simple round-robin selec- ELAYED CKNOWLEDGMENTS tion, updated on the arrival of each new packet. D A This made each multipath simulation determin- In most TCP implementations, not all segment- istic and repeatable. bearing packets are immediately acknowledged by Since we wanted only to examine the effects the receiver as you might expect. Instead, a nomi- of path diversion on TCP without confusing its nally optional, but widespread, delayed acknowl- effects with the effects of congestion, we pre- edgment mechanism is used. This allows the sumed that the links were both error- and con- receiver to skip acknowledging TCP segments gestion-free, allowing any difference in before issuing a cumulative ack covering all the propagation delays of the links to reorder the segments received since the last ack was sent. packets. We presumed high-capacity links — 5 The receiver should acknowledge every second Mb/s uplink and downlink, and 10 Mb/s ISL in-order TCP segment received, and should wait links — so that the time required to receive each between 0.1 and 0.5 s for new packets containing 1000-byte packet was smaller than the delay TCP segments to arrive before issuing an ack. introduced by each wireless hop, but imposed a This allows acks to be piggybacked on any data small receiver window so that the TCP connec- segments that may be sent by the receiver (since tion would not fill the channel. We simulated a TCP is duplex), reducing overall network traffic. bulk FTP transfer of a large file over a period of In practice, it is commonplace to wait for a 100 s: a time short enough that terminal-satellite second segment before sending an ack. The handoffs and resulting routing transients did not receiver can wait until a system timer goes off affect our simulations. We used abstractions of every 200 ms, when an ack must be sent. We two widely implemented variations of TCP: included the common 200 ms delay when simu- • New Reno, based on a stack implementa- lating the effect of the dupack threshold on mul- tion originally developed on a machine tipath throughput, as shown in Fig. 4. named Reno, and widely deployed since Only in-order segments are subject to this with some modifications [8] delay. Out-of-order segments are generally

IEEE Communications Magazine • March 2001 177 FTP transfer over Spaceway NGSO using TCP New Reno FTP transfer over Spaceway NGSO using TCP SACK Amount of file transferred as seen by application (K) 103 Amount of file transferred as seen by application (K) 103

4.0 4.0 3.8 Short any dupacks 3.8 Short any dupacks Multipath 5 dupacks Multipath 4 or 5 dupacks 3.6 Multipath 4 dupacks 3.6 Multipath 3 dupacks 3.4 Multipath 3 dupacks 3.4 Multipath 2 dupacks 3.2 Multipath 2 dupacks 3.2 3.0 3.0 2.8 2.8 2.6 2.6 2.4 2.4 2.2 2.2 2.0 2.0 1.8 1.8 1.6 1.6 1.4 1.4 1.2 1.2 1.0 1.0 0.8 0.8 0.6 0.6 0.4 0.4 0.2 0.2 0.0 0.0 20.0 40.0 60.0 80.0 100.0 20.0 40.0 60.0 80.0 100.0 Time (s) Time (s) i. Progress of FTP transfer between terminals at Quito and Tokyo using Spaceway NGSO

FTP transfer over Teledesic using TCP New Reno FTP transfer over Teledesic using TCP SACK Amount of file transferred as seen by application (K) 103 Amount of file transferred as seen by application (K) 103 12.0 12.0 Short any dupacks Short any dupacks 11.0 Multipath 5 dupacks 11.0 Multipath 3 to 5 dupacks Multipath 4 dupacks Multipath 2 dupacks 10.0 Multipath 3 dupacks 10.0 Multipath 2 dupacks 9.0 9.0

8.0 8.0

7.0 7.0

6.0 6.0

5.0 5.0

4.0 4.0

3.0 3.0

2.0 2.0

1.0 1.0

0.0 0.0 20.0 40.0 60.0 80.0 100.0 20.0 40.0 60.0 80.0 100.0 Time (s) Time (s) ii. Progress of FTP transfer between terminals at Quito and Tokyo using Teledesic (a) (b)

Figure 4. The effect of fast recovery dupack threshold on throughput over multiple paths: a) transfers using New Reno TCP; b) transfers using SACK TCP.

acknowledged immediately, on the principle that the sender. This is useful for asymmetric net- this dupack information is useful to the sender works, such as DirecPC, where the downlink can in determining what to resend in the case of be around 400 kb/s via broadcast from geosta- losses. The effects of delayed acks are discussed tionary satellite, but the uplink is actually a ter- in detail in [12]. restrial dialup modem of 56 kb/s or less. Delayed acks have the advantages of conserv- However, delayed acks are widely recognized ing processing resources at the receiver, and as degrading TCP throughput in some situations. decreasing back traffic and the number of pack- Since TCP slowstart at the beginning of a trans- ets and resulting load along the return path to fer generally uses the number of acknowledg-

178 IEEE Communications Magazine • March 2001 FTP transfer over Spaceway NGSO using TCP New Reno FTP transfer over Spaceway NGSO using TCP SACK 3 Amount of file transferred as seen by application (K) 103 Amount of file transferred as seen by application (K) 10 4.0 4.0 3.8 Short 50ms delack 3.8 Short 50ms delack Short 200ms delack Short 200ms delack 3.6 Short 500ms delack 3.6 Short 500ms delack 3.4 Multipath 500ms delack 3.4 Multipath 50ms delack Multipath 200ms delack Multipath 200ms delack 3.2 Multipath 50ms delack 3.2 Multipath 500ms delack 3.0 3.0 2.8 2.8 2.6 2.6 2.4 2.4 2.2 2.2 2.0 2.0 1.8 1.8 1.6 1.6 1.4 1.4 1.2 1.2 1.0 1.0 0.8 0.8 0.6 0.6 0.4 0.4 0.2 0.2 0.0 0.0 20.0 40.0 60.0 80.0 100.0 20.0 40.0 60.0 80.0 100.0 Time (s) Time (s) i. Progress of FTP transfer between terminals at Quito and Tokyo using Spaceway NGSO

12.0 Short 200ms delack 12.0 Short 500ms delack Short 200ms delack 11.0 Multipath 50ms delack 11.0 Short 500ms delack Multipath 100ms delack Multipath 50 ms delack 10.0 Multipath 200ms up delack Multipath 100ms delack 10.0 Multipath 200ms up delack 9.0 9.0

8.0 8.0

7.0 7.0

6.0 6.0

5.0 5.0

4.0 4.0

3.0 3.0

2.0 2.0

1.0 1.0

0.0 0.0 20.0 40.0 60.0 80.0 100.0 20.0 40.0 60.0 80.0 100.0 Time (s) Time (s) ii. Progress of FTP transfer between terminals at Quito and Tokyo using Teledesic (a) (b)

Figure 5. Delayed acks degrading the rate of file transfer over multiple paths; a) transfers using New Reno TCP; b) transfers using SACK TCP. ments received as an indication of how much ual TCP flows using delayed acks are less aggres- new traffic can be injected into the network, the sive in gaining network capacity. initial slowstart phase is slowed further by the It is this damping which causes delayed acks decreased number of acks sent by a delayed-ack to adversely affect TCP’s sensitivity to out-of- receiver. Increase of the congestion window after order segments received over multiple paths. a timeout or in fast recovery is similarly slowed Although receivers with and without delayed by half. From the viewpoint of Internet traffic as acks should behave identically when receiving a whole, these effects can be considered benefi- out-of-order segments — issuing acks immedi- cial, since window growth is damped and individ- ately so that the sender can make the same deci-

IEEE Communications Magazine • March 2001 179 sion whether to do a fast retransmit after receiv- The recent advent of multiprotocol label It is unlikely that ing three dupacks and enter fast recovery — the switching (MPLS) can allow IP routing of ATM growth of the congestion window after entering flows of IP traffic, and seems promising for any broadband fast recovery is slowed because the sender satellite constellation networks since it allows sup- satellite receives fewer acknowledgments to in-order port for IP multicast and quality of service (QoS) packets. [14]. How such IP routing would be implemented constellation will If entering fast recovery becomes a regular (e.g., for optical ISLs, possibly using wavelength- occurrence, as we have seen happen in multipath division multiplexing), remains to be seen. be a true IP environments, the delayed growth of the conges- Although TCP is tolerant of out-of-order packet-switching tion window severely limits overall throughput, traffic, other non-IP-based traffic is less so and and considerably more time is spent in fast would be even more likely to be routed across a network based recovery with a low congestion window setting. single path, with priority over best-effort Inter- on the IP packet Figure 5, from our ns simulations as described net traffic. If flow-unaware multipath routing is earlier, shows that throughput in multipath envi- used, it would be possible to have buffering and structure. Instead, ronments can be affected considerably by alter- reordering of entire traffic flows at the edges of ations in receiver ack delays. The effect on the satellite network, rather than just the mini- the need for shortest-path throughput, due to the change in mal buffering needed to reconstruct higher-layer management of slowstart when the connection is opened, is com- frames from MAC-level frames. However, this paratively minimal. Delayed acks contribute to would add to overall latency, state held in the frequency the degradation of TCP’s performance in multi- network, and system complexity. allocation in the path environments. This degradation could be It would be practical to implement split-TCP reduced by selectively avoiding use of delayed connections across the constellation network, terminal uplink acknowledgements when the TCP sender is where the TCP connection used across the satel- attempting to grow its congestion window. The lites is optimized for local conditions. If the con- and downlink sender would need to provide additional informa- stellation uses multipath routing, the TCP dictates the need tion to the receiver to make this possible. Such a implementation used in terminals and gateways change has already been suggested in [12] for the for satellite communication would include modifi- for MAC with a slowstart algorithm, and would benefit TCP traf- cations, such as a higher dupack threshold for fast fixed frame size. fic over geostationary satellites by speeding up the recovery, to improve performance relative to end- initial startup and post-timeout phases. However, to-end TCP traffic, while the TCP implementa- the impact of such changes on the characteristics tion used for communicating with the terrestrial of Internet traffic as a whole is unknown. Internet is configured differently for the terrestri- Delayed-ack use has been tightened in special environment. This approach is reminiscent of fication so that only acks acknowledging receipt many of the split-TCP implementations used with of new in-order packets received at the right existing geostationary satellites today. edge of the window are delayed, rather than delaying all acks to all in-order packets. Acks to ONCLUSIONS packets “filling in” gaps in window data should C be sent immediately, even if the packet is “in Examining TCP over LEO and MEO satellite order” relative to the previous packet. As a constellation networks with ISLs has clearly should, this is optional, although recommended. shown that using multiple paths to spread net- Following this recommendation, as the simula- work load and avoid packet drops due to conges- tions presented here do, goes part of the way tion can cause a TCP sender to behave just as if toward avoiding use of delayed acks when the it was on a single congested path where packets TCP sender is attempting to grow its congestion are dropped. This is due to TCP’s fast recovery window in reordering-induced fast recovery. This algorithm, which is intended to handle losses in increases FTP throughput over that of receivers an ordered flow of segments. Delayed acknowl- not following this recommendation, particularly edgments can damp window growth to impair for SACK’s fast recovery algorithm. the performance of TCP over multipath routing still further. RACTICAL ONSIDERATIONS Although TCP tolerates the misordering that P C results from multipath routing and load balanc- It is possible that dynamic routing changes due to ing, its overall throughput suffers as a result of link failures or handoff in a satellite constellation its assumptions about congestion. Performance could result in dupacks and a drop into fast recov- considerations dictate a single ordered flow of ery at high throughputs, due to loss of packets traffic between source and destination. TCP is being transmitted between nodes, or to later encouraging a circuit or flow paradigm in the interleaving of packets and acknowledgements in underlying packet networks. The design of TCP flight along both original and new routes as rout- is affecting and restricting the design of future ing information is propagated. Such transient networks. TCP should be examined with a view effects are discussed further in [13]. to improving its tolerance of load balancing and It is unlikely that any broadband satellite con- multipath routing. stellation will be a true IP packet-switching net- As a result of the design of TCP, various Man- work based on the IP packet structure. Instead, hattan mesh or simple forwarding approaches to the need for management of frequency alloca- routing, based on local information, are less desir- tion in the terminal uplink and downlink dictates able than a global shortest-path routing or flow- the need for MAC with a fixed frame size. A based approach within the satellite constellation. number of proposed schemes plan a MAC frame The effects of simple flow-unaware routing encapsulating ATM cells, so IP packets would be approaches can be compensated for by reordering tunnelled over an ATM-based network. of packets at terrestrial gateways or implementing

180 IEEE Communications Magazine • March 2001 a split-TCP approach, using a satellite-optimized [12] M. Allman, “On the Generation and Use of TCP TCP with more tolerance of misordering-induced Acknowledgments,” ACM Comp. Commun. Rev., vol. 28, no. 5, Oct. 1998, pp. 4–21. Although duplicated acknowledgments. [13] K. Varadhan, D. Estrin, and S. Floyd, “Impact of Net- Although examining bulk FTP transfers clearly work Dynamics on End-to-End Protocols: Case Studies examining bulk in TCP and Reliable Multicast,” Tech. rep. 98-672, Dept. shows the limits of TCP’s tolerance for multipath FTP transfers routing, future work would examine the impact of of Comp. Sci., USC, Mar. 1998. [14] L. Wood et al., “IP Routing Issues in Satellite Constellation multipath routing on the shorter, burstier, trans- Networks,” Accepted/to appear in Special Issue on Internet clearly shows the actional data transfers seen using HTTP. Protocols over Satellite, Int’l. J. Sat. Commun.. [15] T. R. Henderson, “Networking over Next-Generation limits of TCP’s ACKNOWLEDGMENTS Satellite Systems,” Ph.D. dissertation, Comp. Sci. Div., UC Berkeley, Fall 1999. tolerance for We thank the maintainers of and contributors to the network simulator ns, used in our simula- multipath rout- BIOGRAPHIES tions, for its ongoing development. Tom Hen- ing, future work derson’s work in extending ns to simulate LLOYD WOOD ([email protected]) received his masters in satellite constellation networks more realistically electronic and electrical engineering from Loughborough Uni- would examine was particularly valuable. That work and explo- versity. He gained an M.Sc. in satellite communication engineering from the University of Surrey in 1995, after studying the impact of ration of split-TCP implementations are constellation topology at ENST Toulouse. He then joined described in detail in [15]. what is now the Centre for Communication Systems Research multipath routing (CCSR), where he has contributed to a number of European REFERENCES ACTS, Esprit, and IST projects. He maintains the Lloyd’s satel- on the shorter, lite constellations Web resource while pursuing his Ph.D. [1] A. H. Ballard, “Rosette Constellations of Earth Satel- burstier, lites,” IEEE Trans. Aerospace and Elect. Sys., vol. 16, no. GEORGE PAVLOU ([email protected]) gained a Diploma 5, Sept. 1980, pp. 656–73. in electrical engineering from the National Technical Uni- transactional data [2] J. G. Walker, “Satellite Constellations,” J. Brit. Interplan- versity of Athens. He obtained M.Sc. and Ph.D. degrees in etary Soc., vol. 37, 1984, pp. 559–71. computer science from University College London, where transfers seen [3] W. R. Stevens, TCP/IP Illustrated, Volume 1: The Proto- he acted as a senior research fellow and lecturer. He is cur- cols, Addison-Wesley, 1994. rently professor of information networking in the Centre using HTTP. [4] M. Allman, V. Paxson, and W. R. Stevens, “TCP Conges- for Communication Systems Research at the University of tion Control,” IETF RFC 2581, Apr. 1999. Surrey, where he leads the activities of the networks [5] C. Partridge and T. Shepard, “TCP Performance over research group. Over the last 10 years he has led a number Satellite Links,” IEEE Network, vol. 11, no. 5, Sept./Oct. of collaborative research projects in networking, with 1997, pp. 44–49. emphasis on management and control. Routing and quali- [6] M. Allman, D. Glover, and L. Sanchez, “Enhancing TCP ty of service are among his key research areas of interest. over Satellite Channels Using Standard Mechanisms,” He has contributed to ISO and ITU-T standardization work. IETF RFC 2488, Jan. 1999. [7] K. Fall and K. Varadhan, Eds., “ns manual/ns Notes and BARRY EVANS ([email protected]) received his B.Sc. and Documentation,” available with ns from http://www. Ph.D. degrees in electrical engineering and in microwave isi.edu/nsnam/. systems from the University of Leeds. From 1969 to 1983 [8] S. Floyd and T. R. Henderson, “The NewReno Modifica- he was British Telecom Lecturer and later Reader in tion to TCP’s Fast Recovery Algorithm,” IETF RFC 2582, Telecommunication Systems at the University of Essex. He Apr. 1999. was then appointed to the Alec Harley Reeves Chair of [9] M. Mathis, J. Mahdavi, and S. Floyd, “TCP Selective Information Systems Engineering at the University of Sur- Acknowledgment Options,” IETF RFC 2018, Oct. 1996. rey, and in 1990 became the founding director of the post- [10] J. C. R. Bennett, C. Partridge and N. Shectman, “Packet graduate Centre for Satellite Engineering Research and of a Reordering Is Not Pathological Network Behavior,” spinoff company, Surrey Satellite Technology Ltd. Since IEEE/ACM Trans. Net., vol. 7, no. 6, Dec. 1999, pp. 789–98. 1996 he has been director of the Centre for Communica- [11] S. Floyd et al., “An Extension to the Selective Acknowledg- tion Systems Research at Surrey. He is the chief editor of ment (SACK) Option for TCP,” IETF RFC 2883, July 2000. the International Journal in Satellite Communications.

IEEE Communications Magazine • March 2001 181