How Linux Community Fixed Bufferbloat?

Home , Bufferbloat, CoDel, Latency (engineering)

How Linux community fixed Bufferbloat?

#LinuxPiter

BufferbloatBufferbloat

● What is the problem ● What causes the problem ● Solutions – Demonstration ● Recent Progress NetworkNetwork PerformancePerformance

Fairness

Throughput Latency Mbits/sec Round Trip Time HowHow badbad isis it?it? WhyWhy isis worseworse now?now?

● No longer in FTP universe ● TCP Initial Window Size = 10 ● Advanced TCP algorithms ● Network Offload packet trains ● Bad benchmarks ● Router/switch memory is cheap ● Everyone fears dropping packets HeadHead ofof LineLine blockingblocking QueuingQueuing TheoryTheory BasicsBasics

utilization servicerate averagetimeinqueue 1 utilization

From Fred Baker: Bufferbloat! Graphic courtesy Sprint, Apricot 2004 Netalyzr:Netalyzr: DownstreamDownstream Netalyzr:Netalyzr: UpstreamUpstream

TCPTCP throughputthroughput dynamicsdynamics

effectivewindow meanthroughput 

t meanroundtriptime u p

h Bottleneck Capacity g u o

r “knee” Queue “cliff” h T Depth e l b a r u s a e M

g n i s a e r c n

I Increasing TCP Window

From Fred Baker: Bufferbloat! BlameBlame LinuxLinux

● Windows XP – Maximum window 64k

● Windows 7 – Bandwidth limit to 80 mbits

● Android – Receive window limited BlameBlame thethe customercustomer

● Customers call support ● Applications are using more bandwidth ● Block and charge

WhyWhy Queueing?Queueing? PriorityPriority Queue:Queue: TheoryTheory PriorityPriority Queue:Queue: RealityReality HierarchicalHierarchical TokenToken BucketBucket RandomRandom EarlyEarly DetectDetect ExplicitExplicit CongestionCongestion NotificationNotification

PACKETS IdealIdeal ActiveActive QueueQueue ManagementManagement

● Fair – All flows get some bandwidth ● Simple – No tuning ● Easy to deploy – No special hardware, no protocol changes ● Reasonable – Won't create multi-second latency StocasticStocastic FairFair QueueQueue CodelCodel PIEPIE CakeCake

CoDel

Fair Queue

Priority Back to Reality BenchmarkBenchmark issuesissues

● Bad ● Good – Bytes/sec – Throughput + – Packes/sec Latency – Latency – Multiple connections – Real not simulated BetterBetter toolstools

● DSLreport's – http://www.dslreports.com/speedtest/ ● ICSI netalyzr – http://netalyzr.icsi.berkeley.edu/ ● Flent: The Flexible Network Tester – https://flent.org/ – RRUL - Real time Response Under Load DSLReportsDSLReports speedtestspeedtest

5 sec delay!! FIFOFIFO –– thethe defaultdefault

Ping 1sec! Upload

Download

Høiland-Jørgensen T., Battling Bufferbloat StochasticStochastic FairFair QueueQueue

Høiland-Jørgensen T., Battling Bufferbloat ControlledControlled DelayDelay -- codelcodel

Høiland-Jørgensen T., Battling Bufferbloat FairFair QueueQueue ControlledControlled DelayDelay

Høiland-Jørgensen T., Battling Bufferbloat RecentRecent OnceOnce youyou startstart looking,looking, BufferbloatBufferbloat cancan bebe everywhereeverywhere

● Edge – Home router ● Provider ● Network ● Accelerators – Caches – Load balancers,... ● Servers LinuxLinux BufferbloatBufferbloat fixes:fixes: 2011-20162011-2016

● Linux 3.3: Byte Queue Limits

● Linux 3.4 RED bug fixes & IW10 added & SFQRED ● Linux 3.5 Fair/Flow Queuing packet scheduling (fq_codel, codel)

● Linux 3.7 TCP small queues (TSQ) ● Linux 3.12 TSO/GSO improvements

● Linux 3.13 Host FQ + Pacing (sch_fq)

● Linux 3.15 Change to microseconds from milliseconds throughout networking kernel

● Linux 3.17 Network Batching API

● The Linux stack is now mostly “pull through”, where it used to be “push”, and looks nothing like it did 6 years ago.

● At least a dozen other improvements I forget

● Linux 4.8 – TCP BBR … (and BSD just got fq_codel!) Basically – everything – except WiFi (and lte) can be debloated now. – And we just made a big dent in WiFi BufferbloatBufferbloat inin Wi-FiWi-Fi

1+sec1+sec latencylatency LinuxLinux 4.44.4 SubSub 40-ms40-ms LinuxLinux 4.9?4.9? HowHow isis BufferbloatBufferbloat SolvedSolved onon Linux?Linux?

● Queuing disciplines – Codel, PIE, ...

● Linux internal – BQL, FQ, Pacing

● Enable good defaults – ecn – fq_codel Questions?

Thank you

Stephen Hemminger @networkplumber [email protected] Bufferbloat resources

Bufferbloat.net: http://bufferbloat.net Email Lists: http://lists.bufferbloat.net

CeroWrt: http://www.bufferbloat.net/projects/cerowrt Other talks: http://mirrors.bufferbloat.net/Talks Jim Gettys Blog – http://gettys.wordpress.com

A big thanks to the bloat mailing list, Jim, Kathie, Van, Dave, Eric, ISC, and all the other the CeroWrt/OpenWrt contributors