Improving latency for interactive, thin-stream applications over reliable transport by Andreas Petlund Doctoral Dissertation submitted to the Faculty of Mathematics and Natural Sciences at the University of Oslo in partial fulfilment of the requirements for the degree of Philosophiae Doctor October 2009 Dedicated to Bertha-Helene Solberg Abstract A large number of network services use IP and reliable transport protocols. For applications with constant pressure of data, loss is handled satisfactorily, even if the application is latency- sensitive [110]. For applications with data streams consisting of intermittently sent small packets, users experience extreme latencies more frequently [50]. Due to the fact that such thin-stream applications are commonly interactive and time-dependent, increased delay may severely reduce the experienced quality of the application. When TCP is used for thin-stream applications, events of highly increased latency are common, caused by the way retransmis- sions are handled [50]. Other transport protocols that are deployed in the Internet, like SCTP, model their congestion control and reliability on TCP, as do many frameworks that provide re- liability on top of unreliable transport. We have tested several application- and transport layer solutions, and based on our findings, we propose sender-side enhancements that reduce the application-layer latency in a manner that is compatible with unmodified receivers. We have implemented the mechanisms as modifications to the Linux kernel, both for TCP and SCTP. The mechanisms are dynamically triggered so that they are only active when the kernel identi- fies the stream as thin. To evaluate the performance of our modifications, we have conducted a wide range of experiments using replayed thin-stream traces captured from real applications as well as artificially generated thin-stream data patterns. From the experiments, effects on latency, redundancy and fairness were evaluated. The analysis of the performed experiments shows great improvements in latency for thin streams when applying the modifications. Sur- veys where users evaluate their experience of several applications’ quality using the modified transport mechanisms confirmed the improvements seen in the statistical analysis. The positive effects of our modifications were shown to be possible without notable effects on fairness for competing streams. We therefore conclude that it is advisable to handle thin streams separately, using our modifications, when transmitting over reliable protocols to reduce retransmission la- tency. iii Acknowledgements I would like to thank Dr. Carsten Griwodz and Dr. Pål Halvorsen for encouragement, ideas, feedback and enthusiasm in abundance. Their dedication and passion rubs off, making good days delightful and bad days endurable. I would also like to thank Espen Søgård Paaby, Jon Pedersen and Kristian Riktor Evensen for their important contributions to the work that has been done. The working environment at Simula Research Laboratory is one of collaboration, support and friendly competition, which I find inspiring. I want to thank all colleagues and friends at Simula for input, feedback and discussions over the Friday cake. During the work on this thesis, I have worked with two of the most excellent companions one could wish for: Knut- Helge Vik has taught me the meaning of hard work as well as lightened the days with brilliantly dry humour. Håvard Espeland always has some interesting information to share, especially about Linux, FOSS and whisky. Thanks to my fiancee, Anne, for love, patience, support. When I get too focused on comput- ers, she puts my life back into perspective. Finally, I want to thank my parents for supporting me in all I decide to do. v Contents Abstract iii Acknowledgements v Contents vii List of Figures xi List of Tables xv 1 Introduction 1 1.1 Background and motivation . ...... 1 1.2 Thesiscontext................................... ..... 3 1.3 Problemstatement ................................ ..... 4 1.4 Contributions .................................... .... 6 1.5 Outline ......................................... ... 7 2 Thin-stream applications 9 2.1 Games ........................................... 9 2.2 Other thin-stream applications . .......... 12 2.2.1 Remoteoperationsystems. 12 2.2.2 Sensornetworks................................ 13 2.2.3 Audioconferences .............................. 13 2.3 Greedystreamsvs.thinstreams . ........ 14 2.4 Latency-analysis for an interactive thin-stream scenario............... 14 2.5 Summary......................................... 16 3 Transport 17 3.1 TCP.............................................. 18 3.1.1 TCP developments culminating in TCP “New Reno” . 19 3.1.2 Retransmission timeout calculation . ....... 22 vii 3.1.3 Delayed acknowledgements . 24 3.1.4 TCPVegas..................................... 25 3.1.5 Selective Acknowledgements . ..... 26 3.1.6 DuplicateSACK ................................. 27 3.1.7 Forward acknowledgements . 27 3.1.8 Congestion control for high-speed links . ....... 27 3.1.9 Summary...................................... 30 3.2 Evaluation of TCP retransmission delays . .......... 30 3.3 SCTP............................................. 33 3.3.1 SCTPchunksandbundling .......................... 34 3.3.2 AcknowledgementsinSCTP ......................... 35 3.3.3 SCTPRTOcalculation............................. 35 3.3.4 SCTP retransmission strategies . ...... 36 3.3.5 OtherSCTPoptions ............................... 36 3.4 Evaluation of SCTP retransmission delays . .......... 37 3.5 UDP with application layer reliability . ........... 40 3.6 Analysis of retransmission delays for UDP and application layer reliability . 42 3.7 DCCP............................................. 43 3.8 Related transport protocol mechanisms . .......... 44 3.8.1 Timercalculation .............................. 44 3.8.2 Exponentialbackoff ............................ 44 3.8.3 Fast retransmit modifications . ...... 45 3.8.4 RTT estimation and congestion detection . ........ 46 3.8.5 RTOmin anddelayedSACKs .......................... 46 3.8.6 Unreliable and partially reliable transport . ........... 46 3.8.7 Latency-sensitive streams . ...... 47 3.9 Thin-streamchallenges ........................... ....... 47 4 Thin-stream modifications 49 4.1 TCP modifications and implementation . ........ 50 4.1.1 TCPthinstreamdetection......................... 50 4.1.2 Switches for enabling the thin-stream modifications . ........... 52 4.1.3 Linear retransmission time-outs . ....... 54 4.1.4 Modifiedfastretransmit . .. .. .. .. .. .. 56 4.1.5 Redundant Data Bundling . 58 4.2 SCTP modifications and implementation . ........ 62 4.2.1 Thinstreamdetection ........................... 62 4.2.2 Implementation of switches to enable thin-stream modifications . 66 4.2.3 Modified minimum retransmission timeout . ...... 67 4.2.4 Correcting the RTO timer reset . 68 4.2.5 Linear retransmission timeouts . ....... 70 4.2.6 Modifiedfastretransmit . .. .. .. .. .. 72 4.2.7 Bundling on fast retransmit . 73 4.2.8 Other possible avenues of investigation . ......... 74 4.3 Applicability of modifications . ......... 75 5 Analysis and evaluation 77 5.1 Test environment, evaluation setup, tools and metrics . ............... 77 5.1.1 Alternative evaluation methods . ....... 78 5.1.2 Laboratory tests with artificial loss . ........ 78 5.1.3 Laboratory tests with cross-traffic induced loss . ........... 79 5.1.4 Internettests................................. 80 5.1.5 Testdata ...................................... 81 5.1.6 Metrics ....................................... 81 5.1.7 Lossestimation ................................ 82 5.1.8 Calculation of transmission overhead . ....... 83 5.1.9 Calculation of delivery delay . ..... 83 5.2 Evaluation of TCP modifications . ...... 84 5.2.1 Artificial, uniform loss in an emulated network . ......... 85 5.2.2 Congestion-Caused, variable loss in an emulated network ........ 94 5.2.3 Bundle-limittests.............................. 100 5.2.4 Internettests................................. 103 5.2.5 Fairness...................................... 109 5.2.6 Comparison of thin stream performance in different operating systems . 111 5.2.7 Summary...................................... 114 5.3 Evaluation of SCTP modifications . ....... 115 5.3.1 RTOcalculation................................ 115 5.3.2 Artificial, uniform loss in an emulated network . ......... 116 5.3.3 Congestion-caused, variable loss in an emulated network......... 118 5.3.4 Analysis of which mechanisms triggers retransmissions.......... 120 5.3.5 Internettests................................. 122 5.3.6 Fairness...................................... 125 5.3.7 Summary...................................... 128 5.4 UDP and application layer approaches . ......... 129 5.4.1 Latencycomparison ............................. 130 5.4.2 Bandwidth consumption . 130 5.5 Summary......................................... 133 6 Experienced effect for the users 135 6.1 Skypeusertests.................................. 135 6.2 Secureshellusertest............................. ....... 138 6.3 BZFlag hit probability evaluation . ......... 140 6.4 An interactive demonstration . ......... 144 6.5 Summary......................................... 145 7 Conclusions 147 7.1 Summary......................................... 147 7.2 Contributions .................................... 147 7.3 Criticalassessmentoftheresults. .......... 149 7.4 Futurework ...................................... 153 Bibliography