Fair Queuing • Fair Queuing • Core-stateless Fair queuing 15-744: Computer Networking • Assigned reading • [DKS90] Analysis and Simulation of a Fair Queueing Algorithm, Internetworking: Research and Experience L-5 Fair Queuing • [SSZ98] Core-Stateless Fair Queueing: Achieving Approximately Fair Allocations in High Speed Networks 2 Overview Example • 10Gb/s linecard • TCP and queues • Requires 300Mbytes of buffering. • Queuing disciplines • Read and write 40 byte packet every 32ns. • RED • Memory technologies • DRAM: require 4 devices, but too slow. • Fair-queuing • SRAM: require 80 devices, 1kW, $2000. • Core-stateless FQ • Problem gets harder at 40Gb/s • Hence RLDRAM, FCRAM, etc. • XCP 3 4 1 Rule-of-thumb If flows are synchronized • Rule-of-thumb makes sense for one flow • Typical backbone link has > 20,000 flows • Does the rule-of-thumb still hold? t • Aggregate window has same dynamics • Therefore buffer occupancy has same dynamics • Rule-of-thumb still holds. 5 6 If flows are not synchronized Central Limit Theorem B • CLT tells us that the more variables (Congestion 0 Windows of Flows) we have, the narrower the Gaussian (Fluctuation of sum of windows) • Width of Gaussian decreases with • Buffer size should also decreases with Buffer Size Probability Distribution 7 8 2 Required buffer size Overview • TCP and queues • Queuing disciplines • RED • Fair-queuing • Core-stateless FQ Simulation • XCP 9 10 Queuing Disciplines Packet Drop Dimensions • Each router must implement some queuing discipline Aggregation Per-connection state Single class • Queuing allocates both bandwidth and buffer space: Class-based queuing • Bandwidth: which packet to serve (transmit) Drop position next Head Tail • Buffer space: which packet to drop next (when required) Random location • Queuing also affects latency Early drop Overflow drop 11 12 3 Typical Internet Queuing FIFO + Drop-tail Problems • FIFO + drop-tail • Leaves responsibility of congestion control • Simplest choice to edges (e.g., TCP) • Used widely in the Internet • FIFO (first-in-first-out) • Does not separate between different flows • Implies single class of traffic • No policing: send more packets get more • Drop-tail service • Arriving packets get dropped when queue is full regardless of flow or importance • Synchronization: end hosts react to same • Important distinction: events • FIFO: scheduling discipline • Drop-tail: drop policy 13 14 Active Queue Management Active Queue Designs • Design active router queue management to • Modify both router and hosts aid congestion control • DECbit – congestion bit in packet header • Why? • Modify router, hosts use TCP • Routers can distinguish between propagation • Fair queuing and persistent queuing delays • Per-connection buffer allocation • Routers can decide on transient congestion, • RED (Random Early Detection) based on workload • Drop packet or set bit in packet header as soon as congestion is starting 15 16 4 Overview Internet Problems • Full queues • TCP and queues • Routers are forced to have have large queues • Queuing disciplines to maintain high utilizations • TCP detects congestion from loss • RED • Forces network to have long standing queues in steady-state • Fair-queuing • Lock-out problem • Core-stateless FQ • Drop-tail routers treat bursty traffic poorly • Traffic gets synchronized easily allows a few • XCP flows to monopolize the queue space 17 18 Design Objectives Lock-out Problem • Keep throughput high and delay low • Random drop • Accommodate bursts • Packet arriving when queue is full causes some random packet to be dropped • Queue size should reflect ability to accept bursts rather than steady-state queuing • Drop front • Improve TCP performance with minimal • On full queue, drop packet at head of queue hardware changes • Random drop and drop front solve the lock- out problem but not the full-queues problem 19 20 5 Full Queues Problem Random Early Detection (RED) • Drop packets before queue becomes full • Detect incipient congestion, allow bursts (early drop) • Keep power (throughput/delay) high • Intuition: notify senders of incipient • Keep average queue size low congestion • Assume hosts respond to lost packets • Example: early random drop (ERD): • Avoid window synchronization • If qlen > drop level, drop each new packet with fixed • Randomly mark packets probability p • Does not control misbehaving users • Avoid bias against bursty traffic • Some protection against ill-behaved users 21 22 RED Algorithm RED Operation • Maintain running average of queue length Max thresh Min thresh • If avgq < minth do nothing • Low queuing, send packets through • If avgq > maxth, drop packet Average Queue Length • Protection from misbehaving sources P(drop) • Else mark packet in a manner proportional 1.0 to queue length • Notify sources of incipient congestion maxP minth maxth Avg queue length 23 24 6 RED Algorithm Queue Estimation • Maintain running average of queue length • Standard EWMA: avgq = (1-wq) avgq + wqqlen • Byte mode vs. packet mode – why? • Special fix for idle periods – why? • Upper bound on w depends on min • For each packet arrival q th • Want to ignore transient congestion • Calculate average queue size (avg) • Can calculate the queue average if a burst arrives • If minth ≤ avgq < maxth • Set wq such that certain burst size does not exceed minth • Calculate probability P a • Lower bound on wq to detect congestion relatively • With probability Pa quickly • Mark the arriving packet • Typical wq = 0.002 • Else if maxth ≤ avg • Mark the arriving packet 25 26 Thresholds Packet Marking • minth determined by the utilization • maxp is reflective of typical loss rates requirement • Paper uses 0.02 • Tradeoff between queuing delay and utilization • 0.1 is more realistic value • Relationship between max and min th th • If network needs marking of 20-30% then • Want to ensure that feedback has enough time to make difference in load need to buy a better link! • Depends on average queue increase in one • Gentle variant of RED (recommended) RTT • Vary drop rate from maxp to 1 as the avgq • Paper suggest ratio of 2 varies from maxth to 2* maxth • Current rule of thumb is factor of 3 • More robust to setting of maxth and maxp 27 28 7 Extending RED for Flow Isolation Stochastic Fair Blue • Problem: what to do with non-cooperative • Same objective as RED Penalty Box flows? • Identify and penalize misbehaving flows • Fair queuing achieves isolation using per- • Create L hashes with N bins each flow state – expensive at backbone routers • Each bin keeps track of separate marking rate (pm) • How can we isolate unresponsive flows without • Rate is updated using standard technique and a bin per-flow state? size • Flow uses minimum p of all L bins it belongs to • RED penalty box m • Non-misbehaving flows hopefully belong to at least one • Monitor history for packet drops, identify flows bin without a bad flow that use disproportionate bandwidth • Large numbers of bad flows may cause false positives • Isolate and punish those flows 29 30 Stochastic Fair Blue Overview • False positives can continuously penalize • TCP and queues same flow • Solution: moving hash function over time • Queuing disciplines • Bad flow no longer shares bin with same flows • RED • Is history reset does bad flow get to make trouble until detected again? • Fair-queuing • No, can perform hash warmup in background • Core-stateless FQ • XCP 31 32 8 Fairness Goals What is Fairness? • Allocate resources fairly • At what granularity? • Isolate ill-behaved users • Flows, connections, domains? • What if users have different RTTs/links/etc. • Router does not send explicit feedback to • Should it share a link fairly or be TCP fair? source • Maximize fairness index? • Still needs e2e congestion control 2 2 • Fairness = (Σxi) /n(Σxi ) 0<fairness<1 • Still achieve statistical muxing • Basically a tough question to answer – typically • One flow can fill entire pipe if no contenders design mechanisms instead of policy • Work conserving scheduler never idles link if • User = arbitrary granularity it has a packet 33 34 Max-min Fairness Max-min Fairness Example • Allocate user with “small” demand what it • Assume sources 1..n, with resource wants, evenly divide unused resources to demands X1..Xn in ascending order “big” users • Assume channel capacity C. • Formally: • Give C/n to X1; if this is more than X1 wants, • Resources allocated in terms of increasing demand divide excess (C/n - X1) to other sources: each • No source gets resource share larger than its gets C/n + (C/n - X1)/(n-1) demand • If this is larger than what X2 wants, repeat • Sources with unsatisfied demands get equal share process of resource 35 36 9 Implementing max-min Fairness Bit-by-bit RR • Generalized processor sharing • Single flow: clock ticks when a bit is • Fluid fairness transmitted. For packet i: • Bitwise round robin among all queues • Pi = length, Ai = arrival time, Si = begin transmit time, Fi = finish transmit time • Why not simple round robin? • Fi = Si+Pi = max (Fi-1, Ai) + Pi • Variable packet length can get more service • Multiple flows: clock ticks when a bit from all by sending bigger packets active flows is transmitted round number • Unfair instantaneous service rate • Can calculate Fi for each packet if number of • What if arrive just before/after packet departs? flows is know at all times • This can be complicated 37 38 Bit-by-bit RR Illustration Fair Queuing • Not feasible to • Mapping bit-by-bit schedule onto packet interleave bits on transmission schedule real networks • Transmit packet with the lowest Fi at any • FQ simulates bit-by- given time bit RR • How do you compute Fi? 39 40 10 FQ Illustration Bit-by-bit RR Example Output Flow 1 Flow 1 Flow 2 Flow 2 I/P O/P F=10 F=8 Flow 1 Flow 2 F=5 Output (arriving) transmitting Cannot preempt packet F=10 Flow n currently being transmitted Variation: Weighted Fair Queuing (WFQ) F=2 41 42 Fair Queuing Tradeoffs Overview • FQ can control congestion by monitoring flows • TCP and queues • Non-adaptive flows can still be a problem – why? • Complex state • Queuing disciplines • Must keep queue per flow • Hard in routers with many flows (e.g., backbone routers) • RED • Flow aggregation is a possibility (e.g.