Operating System Support for Low-Latency Streaming Ashvin Goel B.S., I.I.T Kanpur, India, 1992 M.S., UCLA, 1996 A dissertation presented to the faculty of the OGI School of Science & Engineering at Oregon Health & Science University in partial fulfillment of the requirements for the degree Doctor of Philosophy in Computer Science and Engineering July 2003 c Copyright 2003 by Ashvin Goel All Rights Reserved ii The dissertation “Operating System Support for Low-Latency Streaming” by Ashvin Goel has been examined and approved by the following Examination Committee: Jonathan Walpole Professor, OGI School of Science and Engineering Thesis Research Adviser Calton Pu Professor, Georgia Institute of Technology Mark Jones Associate Professor, OGI School of Science and Engineering Wu-chang Feng Assistant Professor, OGI School of Science and Engineering iii Dedication To my parents, whose wholehearted support helped me start this endeavor. To Nadia, my love, whose wholehearted support helped me complete it. iv Acknowledgments The ideas in this dissertation, its character and its form, have been heavily influenced by a long list of people with whom I have been fortunate enough to collaborate during my PhD. Jon Walpole, my advisor, has been instrumental in the successful completion of this work. He provided an environment that nurtured my growth during the early stages of the PhD process, while giving sufficient freedom and responsibility during the later stages of my PhD. His invaluable advice and his philosophy about keeping a balance between work, hobbies and personal life inspired me and will continue to inspire me to do my best. My confidence as a researcher grew under my previous advisor, Calton Pu, who fully sup- ported and encouraged me at each stage of my work. Later, Molly Shor and David Steere mo- tivated my interest in using control theory for software systems. I have enjoyed interacting with Charles Consel who sparked my interest in using programming language techniques for systems work. OGI would not be the same without the Feng brothers, W2 and W4, whose arrival has energized our SYSL group and made the work environment a lot more fun and productive. I wish they had come to OGI earlier! During my Ph.D, I collaborated extensively with my office mate Buck Krasic. Our daily discussion about every aspect of computer science helped me shape and strengthen my views about what is important in software systems. Luca Abeni, a visiting Ph.D. student from Italy, showed me the power of story writing in scientific papers! I could always count on Kang Li, our resident networking expert. I would like to thank many other students and interns who made OGI a fun environment, including Jie Huang, Dan Revel, Anne-Francoise Le Meur, Jim Snow, Francis Chang, Mike Shea and Brian Code. v Contents Dedication : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : iv Acknowledgments : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : v Abstract : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : xiv 1 Introduction : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : 1 1.1 Low-Latency Application Requirements . 3 1.1.1 Network Traffic Generator . 4 1.1.2 Soft Modems . 5 1.1.3 Video Conferencing . 6 1.2 Our Approach . 8 1.2.1 Latencies in Operating Systems . 8 1.2.2 Time-Sensitive Linux . 12 1.3 Contributions of this Dissertation . 16 1.4 Outline of this Dissertation . 17 2 High-Resolution Timing : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : 19 2.1 Introduction . 19 2.2 Firm Timers Design . 20 2.3 Firm Timers Implementation . 22 2.4 Firm Timers Evaluation . 24 2.4.1 Timer Latency . 24 2.4.2 Overhead . 27 2.5 Summary . 32 vi 3 Fine-Grained Kernel Preemptibility : : : : : : : : : : : : : : : : : : : : : : : : : : 33 3.1 Introduction . 33 3.2 Improving Kernel Responsiveness . 34 3.2.1 Explicit Preemption . 34 3.2.2 Preemptible Kernels . 37 3.2.3 Preemptible Lock-Breaking Kernels . 38 3.3 Evaluation . 39 3.3.1 Preemption Latency . 39 3.3.2 Overhead . 46 3.4 Application-Level Evaluation . 48 3.4.1 Non-kernel CPU Competing Load . 49 3.4.2 Kernel CPU Competing Load . 50 3.4.3 File System Competing Load . 51 3.5 Conclusions and Future Work . 52 4 Adaptive Send-Buffer Tuning : : : : : : : : : : : : : : : : : : : : : : : : : : : : : 54 4.1 Adapting Send-Buffer Size . 55 4.2 Effect on Throughput . 59 4.3 Protocol Latency . 62 4.3.1 Effect of Packet Dropping on Latency . 62 4.3.2 Effect of TCP Congestion Control on Latency . 63 4.4 Implementation . 65 4.5 Evaluation Methodology . 66 4.5.1 Experimental Scenarios . 67 4.5.2 Network Setup . 68 4.6 Evaluation . 69 4.6.1 Protocol Latency . 69 4.6.2 Throughput Loss . 75 4.6.3 System Overhead . 80 4.6.4 Understanding Worst Case Behavior . 81 vii 4.6.5 Protocol Latency with ECN . 83 4.7 Application-Level Evaluation . 86 4.7.1 Evaluation Methodology . 89 4.7.2 Results . 90 4.7.3 Discussion . 93 4.8 Conclusions . 94 5 Real-Rate Scheduling : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : 95 5.1 Scheduling Model . 98 5.1.1 Proportion-Period Scheduler . 99 5.1.2 Monitoring Progress . 100 5.1.3 Real-Rate Controller . 103 5.2 Implementation . 117 5.2.1 Proportion-Period Scheduler . 117 5.2.2 Real-Rate Controller . 118 5.3 Evaluation . 119 5.3.1 Proportion-Period Scheduler . 119 5.3.2 Real-Rate Controller . 122 5.3.3 Discussion . 131 5.4 Conclusions . 132 6 Tools for Visualization : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : 134 6.1 Introduction . 134 6.2 Polling in Gscope . 136 6.3 A Gscope Example . 137 6.4 Gscope API . 138 6.4.1 Signal Interface . 139 6.4.2 Control Parameter Interface . 141 6.4.3 Tuple Format . 141 6.4.4 Programming With Gscope . 142 6.5 Discussion . 142 viii 6.5.1 Implementation Portability . 142 6.5.2 Signal Types . 143 6.5.3 Single versus Multi-Threaded Applications . 145 6.5.4 Distributed Applications . 146 6.5.5 Polling Granularity . 146 6.5.6 Scope Overhead . 147 6.6 Conclusions . 147 7 Related Work : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : 148 7.1 Timing Control in OSs . 149 7.2 Kernel Preemptibility . 150 7.3 Buffering for I/O . 150 7.3.1 TCP-based Streaming . 150 7.3.2 Buffer Tuning . ..