The elephant symbolises a big download, commonly referred to as an ”elephant flow”, which blocks the link with its bulk so every- thing has to wait behind it. The mice are smaller flows, such as web pages, that take up little space but should be able to move freely. Through better queue management, the mice are guided around the elephant, so they don’t have to wait; and interactivity is restored. The penguin is Tux, the Linux mascot, who stands at the bottleneck and directs traffic onto the right path. Toke Høiland-Jørgensen | Bufferbloat and Beyond Toke Bufferbloat and Beyond The topic of this thesis is the performance of computer networks in general, and the internet in particular. While network performance has generally improved with time, over the last several years we have seen examples of performance barriers limiting network performance. In this work we explore such performance barriers and look for solutions. 2018:42 Bufferbloat and Beyond Our exploration takes us through three areas where performance barriers are Removing Performance Barriers in Real-World Networks found: The bufferbloat phenomenon of excessive queueing latency, the performance anomaly in WiFi networks and related airtime resource sharing problems, and the problem of implementing high-speed programmable packet processing in an operating system. In each of these areas we present solutions that significantly advance the state of the art. Toke Høiland-Jørgensen The work in this thesis spans all three aspects of the field of computing, namely mathematics, engineering and science. We perform mathematical analysis of algorithms, engineer solutions to the problems we explore, and perform scientific studies of the network itself. All our solutions are implemented as open source software, including both contributions to the upstream Linux kernel, as well as the Flent test tool, developed to support the measurements performed in the rest of the thesis. ISBN 978-91-7063-878-7 (Print) ISBN 978-91-7063-973-9 (pdf) Faculty of Health, Science and Technology ISSN 1403-8099 Computer Science DOCTORAL THESIS | Karlstad University Studies | 2018:42 DOCTORAL THESIS | Karlstad University Studies | 2018:42 Bufferbloat and Beyond Removing Performance Barriers in Real-World Networks Toke Høiland-Jørgensen DOCTORAL THESIS | Karlstad University Studies | 2018:42 Bufferbloat and Beyond - Removing Performance Barriers in Real-World Networks Toke Høiland-Jørgensen DOCTORAL THESIS Karlstad University Studies | 2018:42 urn:nbn:se:kau:diva-69416 ISSN 1403-8099 ISBN 978-91-7063-878-7 (Print) ISBN 978-91-7063-973-9 (pdf) Revised version in which printing errors have been corrected © The author Distribution: Karlstad University Faculty of Health, Science and Technology Department of Mathematics and Computer Science SE-651 88 Karlstad, Sweden +46 54 700 10 00 Print: Universitetstryckeriet, Karlstad 2018 WWW.KAU.SE iii Bufferbloat and Beyond: Removing Performance Barriers in Real-World Networks Toke Høiland-Jørgensen Department of Mathematics and Computer Science Karlstad University Abstract The topic of this thesis is the performance of computer networks. While net- work performance has generally improved with time, over the last several years we have seen examples of performance barriers limiting network performance. In this work we explore such performance barriers and look for solutions. The problem of excess persistent queueing latency, known as bufferbloat, serves as our starting point; we examine its prevalence in the public internet, and evaluate solutions for better queue management, and explore how to improve on existing solutions to make them easier to deploy. Since an increasing number of clients access the internet through WiFi networks, examining WiFi performance is a natural next step. Here we also look at bufferbloat, as well as the so-called performance anomaly, where stations with poor signal strengths can severely impact the performance of the whole network. We present solutions for both of these issues, and additionally design a mechanism for assigning policies for distributing airtime between devices on a WiFi network. We also analyse the “TCP Small Queues” latency minimisation technique implemented in the Linux TCP stack and optimise its performance over WiFi networks. Finally, we explore how high-speed network processing can be enabled in software, by looking at the eXpress Data Path framework that has been gradu- ally implemented in the Linux kernel as a way to enable high-performance programmable packet processing directly in the operating system’s networking stack. A special focus of this work has been to ensure that the results are car- ried forward to the implementation stage, which is achieved by releasing implementations as open source software. This includes parts that have been accepted into the Linux kernel, as well as a separate open source measurement tool, called Flent, which is used to perform most of the experiments presented in this thesis, and also used widely in the bufferbloat community. Keywords: Bufferbloat, AQM, WiFi, XDP, TSQ, Flent, network measure- ment, performance evaluation, fairness, queueing, programmable packet pro- cessing v Acknowledgements The journey is almost over. It has been challenging, exciting, annoying, incredible and above all interesting. And it would not have been the same without the incredible people who have accompanied and supported me along the way, all of whom deserve a huge thank you. First of all, thank you to the many collaborators I have had these past years. Foremost among them of course my advisors, Anna Brunström and Per Hurtig, who have always been ready to offer competent feedback, ideas, suggestions and support; thank you so much to both of you! A special thanks also to Dave Täht, with whom I have collaborated extensively, on both the papers we have co-authored, and a list of other projects too long to reproduce here. Dave was also among the people who initially introduced me to the concept of bufferbloat, and he and Jim Gettys both played major roles in setting me on the right path, towards the work represented in this thesis; for this I am exceedingly grateful. Thank you, as well, to Jesper Brouer for his collaboration, and for convincing me to continue the work with Linux and open source post-thesis; and to Fanny Reinholtz for designing the awesome dust jacket for the printed version of this thesis. And finally, thank you to my other co-authors, and all the members of the Bufferbloat, make-wifi-fast, Linux and OpenWrt communities that I have collaborated with over the last six years. A journey such as this would not have been possible without the friends who have shared my joys and sorrows along the way. Thank you to my colleagues at the university, my old friends in Denmark, and my new(er) friends in Karlstad. Thank you to Lea, Stefan, Stina and Leonardo for climbing, skiing and everything else. To Martin, Daniel, Alex and Miriam for hanging out and having fun. To Timo, Leo and Ricardo for bringing the band (back) together. To the innebandy team (in its various incarnations) for showing me how much fun it can be to chase a ball around with a stick. And to the dwarves, and to Sunni, Morten, Sidsel and Torben for keeping in touch through my five-year Swedish exile. Last, but certainly not least, thank you so much to my wonderful loving family. Thank you mum and dad for helping me get to this point, for always being there for me, and for your love and respect. And thank you Sascha, for coming with me to Sweden and brightening up my days. Karlstad, October 2018 Toke Høiland-Jørgensen Well then. A year and a half later, I find myself preparing a second print run; who would have thought? Many thanks to Dave Täht for insisting on this, and to Karlstad University for agreeing to print another batch, with all that entails. Roskilde, April 2020 Toke Høiland-Jørgensen vii List of appended papers I. Toke Høiland-Jørgensen, Bengt Ahlgren, Per Hurtig and Anna Brun- strom. Measuring Latency Variation in the Internet. ACM CoNEXT ’16, December 12–15, 2016, Irvine, CA, USA. II. Toke Høiland-Jørgensen, Per Hurtig and Anna Brunstrom. The Good, the Bad and the WiFi: Modern AQMs in a Residential Setting. Computer Networks, vol 89, pg 90–106, October 2015. III. Toke Høiland-Jørgensen. Analysing the Latency of Sparse Flows in the FQ-CoDel Queue Management Algorithm. IEEE Communication Letters, October 2018. IV. Toke Høiland-Jørgensen, Dave Täht and Jonathan Morton. Piece of CAKE: A Comprehensive Queue Management Solution for Home Gateways. IEEE International Symposium on Local and Metropolitan Area Networks (LANMAN 2018), 25–27 June 2018, Washington, DC. V. Toke Høiland-Jørgensen, Michal Kazior, Dave Täht, Per Hurtig and Anna Brunstrom. Ending the Anomaly: Achieving Low Latency and Airtime Fairness in WiFi. 2017 USENIX Annual Technical Conference (USENIX ATC 17), July 12–14 2017, Santa Clara, CA. VI. Toke Høiland-Jørgensen, Per Hurtig and Anna Brunstrom. PoliFi: Air- time Policy Enforcement for WiFi. Under Submission. VII. Carlo Augusto Grazia, Natale Patriciello, Toke Høiland-Jørgensen, Mar- tin Klapez and Maurizio Casoni. Adapting TCP Small Queues for IEEE 802.11 Networks. The 29th Annual IEEE International Symposium on Personal, Indoor and Mobile Radio Communications (IEEE PIMRC 2018), 9-12 September 2018, Bologna, Italy. VIII. Toke Høiland-Jørgensen, Jesper Dangaard Brouer, Daniel Borkmann, John Fastabend, Tom Herbert, David Ahern and David Miller. The eXpress Data Path: Fast Programmable Packet Processing in the Op- erating System Kernel. CoNEXT ’18: International Conference on emerging Networking EXperiments and Technologies, December 4–7, 2018, Heraklion, Greece. IX. Toke Høiland-Jørgensen, Carlo Augusto Grazia, Per Hurtig and Anna Brunstrom. Flent: The FLExible Network Tester. 11th EAI Interna- tional Conference on Performance Evaluation Methodologies and Tools (VALUETOOLS 2017), December 5–7, 2017, Venice, Italy. Comments on my participation For all papers, the ideas have been developed in cooperation with my co- authors.