How Linux community fixed Bufferbloat? Stephen Hemminger [email protected] #LinuxPiter BufferbloatBufferbloat ● What is the problem ● What causes the problem ● Solutions – Demonstration ● Recent Progress NetworkNetwork PerformancePerformance Fairness Throughput Latency Mbits/sec Round Trip Time HowHow badbad isis it?it? WhyWhy isis worseworse now?now? ● No longer in FTP universe ● TCP Initial Window Size = 10 ● Advanced TCP algorithms ● Network Offload packet trains ● Bad benchmarks ● Router/switch memory is cheap ● Everyone fears dropping packets HeadHead ofof LineLine blockingblocking QueuingQueuing TheoryTheory BasicsBasics utilization servicerate averagetimeinqueue= 1- utilization From Fred Baker: Bufferbloat! Graphic courtesy Sprint, Apricot 2004 Netalyzr:Netalyzr: DownstreamDownstream Netalyzr:Netalyzr: UpstreamUpstream TCPTCP throughputthroughput dynamicsdynamics effectivewindow meanthroughput = t meanroundtriptime u p h Bottleneck Capacity g u o r “knee” Queue “cliff” h T Depth e l b a r u s a e M g n i s a e r c n I Increasing TCP Window From Fred Baker: Bufferbloat! BlameBlame LinuxLinux ● Windows XP – Maximum window 64k ● Windows 7 – Bandwidth limit to 80 mbits ● Android – Receive window limited BlameBlame thethe customercustomer ● Customers call support ● Applications are using more bandwidth ● Block and charge WhyWhy Queueing?Queueing? PriorityPriority Queue:Queue: TheoryTheory PriorityPriority Queue:Queue: RealityReality HierarchicalHierarchical TokenToken BucketBucket RandomRandom EarlyEarly DetectDetect ExplicitExplicit CongestionCongestion NotificationNotification PACKETS IdealIdeal ActiveActive QueueQueue ManagementManagement ● Fair – All flows get some bandwidth ● Simple – No tuning ● Easy to deploy – No special hardware, no protocol changes ● Reasonable – Won't create multi-second latency StocasticStocastic FairFair QueueQueue CodelCodel PIEPIE CakeCake CoDel Fair Queue Priority Back to Reality BenchmarkBenchmark issuesissues ● Bad ● Good – Bytes/sec – Throughput + – Packes/sec Latency – Latency – Multiple connections – Real not simulated BetterBetter toolstools ● DSLreport's – http://www.dslreports.com/speedtest/ ● ICSI netalyzr – http://netalyzr.icsi.berkeley.edu/ ● Flent: The Flexible Network Tester – https://flent.org/ – RRUL - Real time Response Under Load DSLReportsDSLReports speedtestspeedtest 5 sec delay!! FIFOFIFO –– thethe defaultdefault Ping 1sec! Upload Download Høiland-Jørgensen T., Battling Bufferbloat StochasticStochastic FairFair QueueQueue Høiland-Jørgensen T., Battling Bufferbloat ControlledControlled DelayDelay -- codelcodel Høiland-Jørgensen T., Battling Bufferbloat FairFair QueueQueue ControlledControlled DelayDelay Høiland-Jørgensen T., Battling Bufferbloat RecentRecent OnceOnce youyou startstart looking,looking, BufferbloatBufferbloat cancan bebe everywhereeverywhere ● Edge – Home router ● Provider ● Network ● Accelerators – Caches – Load balancers,... ● Servers LinuxLinux BufferbloatBufferbloat fixes:fixes: 2011-20162011-2016 ● Linux 3.3: Byte Queue Limits ● Linux 3.4 RED bug fixes & IW10 added & SFQRED ● Linux 3.5 Fair/Flow Queuing packet scheduling (fq_codel, codel) ● Linux 3.7 TCP small queues (TSQ) ● Linux 3.12 TSO/GSO improvements ● Linux 3.13 Host FQ + Pacing (sch_fq) ● Linux 3.15 Change to microseconds from milliseconds throughout networking kernel ● Linux 3.17 Network Batching API ● The Linux stack is now mostly “pull through”, where it used to be “push”, and looks nothing like it did 6 years ago. ● At least a dozen other improvements I forget ● Linux 4.8 – TCP BBR … (and BSD just got fq_codel!) Basically – everything – except WiFi (and lte) can be debloated now. – And we just made a big dent in WiFi BufferbloatBufferbloat inin Wi-FiWi-Fi 1+sec1+sec latencylatency LinuxLinux 4.44.4 SubSub 40-ms40-ms LinuxLinux 4.9?4.9? HowHow isis BufferbloatBufferbloat SolvedSolved onon Linux?Linux? ● Queuing disciplines – Codel, PIE, ... ● Linux internal – BQL, FQ, Pacing ● Enable good defaults – ecn – fq_codel Questions? Thank you Stephen Hemminger @networkplumber [email protected] Bufferbloat resources Bufferbloat.net: http://bufferbloat.net Email Lists: http://lists.bufferbloat.net CeroWrt: http://www.bufferbloat.net/projects/cerowrt Other talks: http://mirrors.bufferbloat.net/Talks Jim Gettys Blog – http://gettys.wordpress.com A big thanks to the bloat mailing list, Jim, Kathie, Van, Dave, Eric, ISC, and all the other the CeroWrt/OpenWrt contributors .