CSE 237A Scheduling of communication
Tajana Simunic Rosing Department of Computer Science and Engineering University of California, San Diego.
1 RlReal-time communi ca tion sc hdliheduling
Value of comm unication depends on the time at which the message is delivered to the recipient Metrics Throughput Delay Delay jitter Loss rate Fairness Deadlines can be hard or soft Deterministic vs. statistical guarantees
2 Key Problem
Allocation/ sched uling of comm unication Point-to-point link e.g. wire (scheduling at transmitter) Distributed link e. g. wireless (MAC) or bus (arbiter) Entire network (routing) Anywhere there is a shared resource! Different from scheduling to CPUs Often no pppreemption or coarse pre-emption Channels of time-varying quality/capacity
3 Type of Traffic So urces
Cons tan t bit rat e: peri odi c t raffi c Fixed-size packets at periodic intervals Analogous to periodic tasks with constant computation time in RM model Variable bit rate: bursty traffic Fixed-size packets at irregular intervals Variable-size packets at regular intervals
4 Key Iss u es
Scheduling Admission control PliiPolicing Goals: meet performance and fairness metrics high resource utilization easy to implement small work per data item , scale slowly with # of flows or tasks easy admission control decisions
5 ShdliScheduling for commun ica tion
Determine who sends data when: FIFO Priority queuing: preemptive, non-preemptive Round robin Weighted fair queuing EDF Discard mechanism when buffer is f ull Distributed implementation Need multiple access mechanism 6 Problems du e to FIFO qu eu es
1. In order to maximize its chances of ss
ee success, a source has an incentive to maximize the rate at which it transmits. Fairn 2. FIFO queue is “unfair” since it favors the most greedy flow. es 3.ee It i s h ar d to con trol the de lay o f pac ke ts
Delay through a network of FIFO queues. Guarant
7 Fairness 10 Mb/s 0.55 A Mb/s 1.1 Mb/s 100 R C Mb/s 1
e.g. an http flow with a given B 0.55 (IP SA, IP DA, TCP SP, TCP DP) Mb/s
What is the “fair” allocation: (0.55Mb/s, 0.55Mb/s) or (0.1Mb/s, 1Mb/s)?
8 Fairness 10 Mb/s A 1.1 Mb/s 100 R1 D Mb/s
B
0.2 C Mb/s What is the “fair” allocation?
9 Fairness
Intuitively each connection gets no more than what it wants the excess, if any, is equally shared Fairness is intuitively a good idea Fairness also provides protection traffic hogs cannot overrun others automatically builds firewalls around heavy users reverse is not true: protection may not lead to fairness
Transfer half of excess Unsatisfied demand
ABC ABC 10 Max-min Fairness
Maximize the minimum share of task or flow whose demand is not fully satisfied Resources are allocated in order of increasing demand, normalized by weight No task or flow gets a share larger than its demand Task or flows with unsatisfied demands get resource shared in ppproportion to their wei ghts
11 Max-Min Fairness A common way to allocate flows N flows share a link of rate C. Flow f wishes to send at rate W(f), and is allocated rate R(f). 1. Pick the flow, f, with the smallest requested rate. 2. If W(f)
12 Max-Min Fairness An example
W(f1) = 0.1 W(f ) = 0.5 1 2 C R1 W(f3) = 10 W(f4) = 5
Round 1: Set R(f1) = 0.1 Round 2: Set R(f2) = 0.9/3 = 0.3 Round 3: Set R(f4) = 0.6/2 = 0.3 Round 4: Set R(f3) = 0.3/1 = 0.3
13 Round Robin
Scan class queues serving one from each class that has a non-empty queue
Figure from: Kurose14 & Ross Weighted Round Robin
Round-robin unfair if packets are of different length or weihtights are no t equa l Different weights, fixed packet size serv e m or e th an on e pack et per vi si t, af ter n orm alizin g to obtain integer weights Different weights, variable size packets normalize weights by mean packet size Problems with variable size packets and different weights, need to know mean packtket si ze i n a dvance fair only over time scales > round time round time can be large can lead to long periods of unfairness
15 Priority Q ue uing
Flows classified according to priorities Preemptive and non-preemptive versions
Figure from: Kurose16 & Ross Fair Que ueing
1. Packets belonging to a flow are placed in a FIFO. This is called “per-flow queuing”. 2. FIFOs are scheduled one bit at a time, in a round-robin fashion. 3. This is called Bit-by-Bit Fair Queuing.
Flow 1 Bit-by-bit round robin
Classification Flow N Scheduling
17 Weighted Bit-by-Bit Fair Queueing Likew ise, flows can be a lloca te d differen t rates by servicing a different number of bits f or each fl ow dur ing eac h roun d.
R(f1) = 0.1 R(f ) = 0.3 1 2 C R1 R(f3) = 0.3 R(f4)03) = 0.3 OdOrder of service for the four queues: … f1, f2, f2, f2, f3, f3, f3, f4, f4, f4, f1,…
Also called “ Generalized Processor Sharing (GPS)”
18 Generalized Processor Sharing
Generalized Round Robin In any time interval, allocates resource in proportion to the wei gh t am on g th e set of all backl ogged co nn ecti on s Serves infinitesimal resource to each Achieves max-min fairness But is non-implementable
Figure from: S. Keshav,19 Cornell Weighted Fair Queueing (WFQ)
Problem: We need to serve a whole packet at a time. Solution: 1. Determine what time a packet, p, would complete if we served flows bit-
by-bit. Call this the packet’s finishing time, Fp. 2. Serve packets in the order of increasing finishing time.
Theorem: PktPacket p will depar t be fore Fp+ TRANSP max
Also called “Packetized Generalized Processor Sharing (PGPS)” 20 Understanding bit by bit WFQ 4 queues, sharing 4 bits/sec of bandwidth, Equal Weights
6 5 4 3 2 1 0 Time
A1 = 4 1
B1 = 3 1
C2 = 1 C1 = 1 1
D2 = 2 D1 = 1 1
Weights : 1:1:1:1
6 5 4 3 2 1 0 Time D1, C1 Depart at R=1 A2, C3 arrive A2 = 2 A1 = 4 1
B1 = 3 D1 C1 B1 A1 1
C3 = 2 C2 = 1 C1 = 1 1 Round 1 D2 = 2 D1 = 1 1
Weights : 1:1:1:1
6 5 4 3 2 1 0 Time C2 Departs at R=2
A2 = 2 A1 = 4 1
B1 = 3 D2 C2 B1 A1 D1 C1 B1 A1 1
C3 = 2 C2 = 1 C1 = 1 1 Round 2 Round 1 D2 = 2 D1 = 1 1
Weights : 1:1:1:1 21 Understandinggy bit by bit WFQ 4 queues, sharing 4 bits/sec of bandwidth, Equal Weights
6 5 4 3 2 1 0 Time D2, B1 Depart at R=3
A2 = 2 A1 = 4 1
B1 = 3 D2 C3 B1 A1 D2 C2 B1 A1 D1 C1 B1 A1 1
C3 = 2 C2 = 1 C1 = 1 1 Round 3 Round 2 Round 1 D2 = 2 D1 = 1 1
Weights : 1:1:1:1 6 5 4 3 2 1 0 A2 C3, A1 Depart at R=4 Time Departs at R=6 A2 = 2 A1 = 4 1
B1 = 3 A2 A2 C3 A1 D2 C3 B1 A1 D2 C2 B1 A1 D1 C1 B1 A1 1
C3 = 2 C2 = 1 C1 = 1 1 6 5 Round 4 Round 3 Round 2 Round 1 D2 = 2 D1 = 1 1
Weights : 1:1:1:1 Sort packets
6 5 4 3 2 1 0 Time Departure order for packet by packet WFQ: Sort by finish round of packets
A2 = 2 A1 = 4 1
B1 = 3 A2 A2 C3 C3 A 1 A 1 A1 A 1 D2 D2 B1 B1 B1 C2 D1 C1 1
C3 = 2 C2 = 1 C1 = 1 1
D2 = 2 D1 = 1 1
Weights : 1:1:1:1 22 WFQ implementation ov erv iew
Assuming we know the current round number R Finish number of packet of length p if arriving to active connection = previous finish number + p if arriving to an inactive connection = R + p Dealing with weights replace p by p/w Computing the current round number Define round number (R) as a real-valued variable R increases at a rate inversely proportional to # of active connections replace # of connections by sum of connection weights WFQ emulates GPS instead of bit-by-bit RR with this definition
from: S. Keshav,23 Cornell WFQ
wi Serves t raffi c at rat e R (t) = i w ∑k∈ACTIVE(t) k 1 If w =1 then the rate is: R(t) = i’s | ACTIVE(t) |
Finish number computation:
Pi (k,t) Fi (k,t) = max{Fi (k −1,t), R(t)}+ wi 1 Round rate: RR(t) ~ w TSR ∑k∈ACTIVE (t) k 24 WFQ E xample
Pi (k,t) Fi (k,t) = max{Fi (k −1,t), R(t)}+ wi =1 link rate 1 unit/sec Queue A Queue B Queue C time # conn RR R F P F P F P ΔR(t) = RR(t)⋅Δtime⋅ LinkRate 0 3 0.33 0 1 1 2 2 2 2 320.51102121 4 3 0.33 1.5 3.5 2 2 0.5 2 0.5 555.5 1 1 2 353.5 151.5 2 0 2 0 7003.53.502020
TSR 25 WFQ E xample
Pi (k,t) Fi (k,t) = max{Fi (k −1,t), R(t)}+ wi link rate 100 bits/s Q A wa=2 Q B wb=5 time (s) # conn RR R F P F P ΔR(t) = RR(t)⋅Δtime⋅ LinkRate 0 2 0 x=100 y=200 1.5 When do packets complete service? 2 What round number? 282.8 What is packet ZZs’s finish number if it arrives at time 1.5s and is 10 bits? 3
TSR 26 Ev alu ation
Pros like GPS, it provides protection can obtain worst-case end-to-end delay bound gives users incentive to use intelligent flow control Cons needs per-connection state comppplex to implement requires a priority queue
27 Problems du e to FIFO qu eu es
1. In order to maximize its chances of ss
ee success, a source has an incentive to maximize the rate at which it transmits. Fairn 2. FIFO queue is “unfair” since it favors the most greedy flow. es 3.ee It i s h ar d to con trol the de lay o f pac ke ts
Delay through a network of FIFO queues. Guarant
28 Delay sensitive applications
Multimedia Applications One-to-many streaming stored audio or video. Interactive streaming audio or video. Others Real-time control . Premium Internet/web access
29 The need for a play back bu ffer
45 45 24Mb/s 45 100 100
e.g. 10fps, 600x500 bytes per frame ~= 24Mb/s
30 The need for a play back bu ffer
Cumulative Source Destination Bytes
24Mb/s 24Mb/s
Total delay
Time
31 In real life: Delay Variation
PbProb
99%
e.g. 200ms Delay/latency Min
32 Model of FIFO router queue
Router Without QoS: A(t)R D(t) B(t)
Model of WFQ router queues
A (t) R(f ) Router 1 1 D(t) A(t) A2(t) With QoS: R(f2)
AN(t)
R(fN)
Fall 2005 33 Let’s say we can bound the arrival process
ρ Cumulative bytes Number of bytes that can arrive in any period of length t A1(()t) is bounded by: σ + ρt
This is called “(σ,ρ) regulation”
σ
time
Fall 2005 34 (σ,ρ) Constrained Arrivals and Minimum Service Rate
dmax
Cumulative Bmax bytes A1(t) D1(t) For no packetρσ loss, B ≥ σ.
If R()f11≥≤ , then dt () /(). R f ρ
σ R(f1) time Theorem [Parekh,Gallager ’93]: If flows are leaky-bucket constrained, and routers use WFQ, then end-to-end delayyg guarantees are p ossible.
35 The leaky bucket “(σ,ρ)” regulator
Tokens at rate, ρ Token bucket size, σ
Packets Packets
One byte (or Packet buffer packet) per token
36 WFQ + Leak y B ucket
Figure from: Kurose37 & Ross Parekh-Gallager Theorem
Let a connection be allocated weights at each of K WFQ schedulers along its path such that the bandwidth allocated at the k-th
schdlheduler is gk
g = smallest gk the connection be leaky-bucket regulated such that # bits sent in
time [t1, t2] <= ρ (t2 -t1) + σ the kth scheduler have a rate ρ(k)
the larggpest packet allowed in the connection be Pc, and in the connection be Pn
KK−1 end _ to _ end _ delay ≤ σ / g + ∑ Pc / g + ∑ Pn / ρ(k) kk==11
38 Significance
Shows that WFQ can provide end-to-end delay bounds WFQ provides both fairness and performance guarantees Bound holds regardless of cross traffic behavior Can be generalized for networks where schedulers are variants of WFQ, and the link service rate changes over time
39 ShdliScheduling w ithDlith a Delay Guaran tee
1. Source asks network for an end-to-end delay guarantee. 2. Source negotiates (σ,ρ) values with each router along the path so at to achieve end-to-end delay guarantee. Routers perform admission control to check whether they have sufficient resources (link data rate and buffers). 3. Each router along path reserves resources. 4. Flow st art s, and source st art s sendi ng pack et s accordi ng t o agreed (σ,ρ) values. 5. Router determines which flow each arriving packet belongs to, and puts it in the right queue. 6. Router serves queues, using WFQ, so as to bound packet delay through the router.
40 Real Time Comm . Architectures
Most basic communication – point to point Three types of communication networks: The field bus The real-time network The backbone network Exchanges non-critical information between the RT cluster and data- ppgyrocessing system Designed to provide timing guarantees Backbone Node Node Gateway
RT Bus
Gateway Gateway
Field bus 41 Real-Time Netw ork
Core of the real-time cluster Provides the following services to the nodes: Reliable and temppyorally predictable message transmission with low latency and minimal latency jitter (<1us) Support for fault-tolerance to handle replicated nodes and replicated communication channels Clock synchronization in the range of microseconds Membership service with low latency for detecting node failures Avoid single point of failure, node containment Periodic state messages with implicit flow control are commonly used Distributed control necessary.
42 Field Bu s
Central controller Sensors and actuators are often controlled by a single- chip microcontroller w standard UART interface Field bus messages Short data field PiPrecise c lklock sync hitihronization nee ddded Fault tolerance not a major issue bottleneck is at the sensors and actuators. Low cost Standard twisted pair wiring common
43 Characteristics
Service Field bus RT network Backbone characteristic network Message State State Event semantics Latency/jitter Yes Yes No control Typical data field 1-6 bytes 6-12 bytes > 100 bytes length Clock Yes Yes Optional synchronization Fault-tolerance Limited Yes Limited Membership Maybe Yes Maybe service Topology Multicast Multicast Point-to-point Communication Multi-master Distributed Central or control distributed Flow control Implicit Implicit Explicit 44 Channel Characteriz ation Bandwidth PtidlPropagation delay: physical signal delay approximately 5 μseconds per 1 km cable length ‘bit length of a channel’ E.g., 100 Mbps over a 200 meter long channel has a bitlength of 5 μs*(200/1000)*100 Mb/s = 100 bits Protocol efficiency Limited by the need to maintain a minimum time interval of one propagation delay between two successive messages. For a channel of bitlength bl bits and the message length of m bits, efficiency < m/(m+bl)m / (m+bl) 1 km bus with 100 Mbps, 100 bit message length Bit length = 500 bits Efficiency < 100 / 600 = 16.6%
45 Media Access Strategies: Event Triggered Protocols
CSMA/CD Carrier Sense Multiple Access/Collision Detection Class of distributed medium access protocols that do not require any central locus of control Example: Ethernet CSMA/CD-LON LON protocol from Echelon Access the channel after a random delay Stochastic back -pressure flow control Industrial, building, home, transportation automation E.g. BART in the bay area, Amtrack monitor more th an 20 syst ems i ncl udi ng b raki ng, doors, HVAC, spee d sensing, speed limitation, fire, integrated truck surveillance, and low- voltage battery charging 46 CSMA/CA-CAN
CAN: Controller Area Network protocol Designed by Boch in 1980s for connecting electronic control units For automotive real-time applications (e.g. in-vehicle data comm.) Very robust in electromagnetically noisy environments 8 byte messages ISO defined for the physical and data link layers serial communication bus with peer-to-peer & multi-server networks 1 Mbps for lengths less than 40m, 125 kbps for 500m Collision avoidance 'arbitration free' transmission CAN message transmitted with highest priority will 'win' the arbitration binary model of "dominant" & "recessive" bits dominant is a logical 0 E.g. racing boats CAN connected units control the engine requires high real-time performance
47 Token Bu s
Token gggives the right to transmit info Two time parameters Token hold time (THT) Longgyest time a node may hold the token Token rotation time (TRT) Longest time for a full rotation of the token Token loss can have serious consequence Example: Profibus more than 500,000 applications to solve all kinds of automation challenges in manufacturing and process control E.g. control of blow-moulding machines for containers with capacities of 0.2 liter up to 10 liter; 2000 to 8000 units per hour
48 Central Master Protocols
Central master controls the access to the bus If it fails, another node takes over the role of central master (CM) Multi-master systems E.g.: Factory Instrumentation Protocol (FIP) 1989 Static list w names and periods of messages generated by CM Master periodically broadcasts the name of a variable on the bus Node responds with a broadcast of the contents of this variable All other nodes listen accept the content if needed If free time remains, the nodes send data after being polled by the master.
49 TDMA-TTP
Time Division Multiple Access Static medium access strategy Right to transmit controlled by real-time progress Needs a (fault-tolerant) global time-base available to all nodes TtlhTotal channe l capac itittilldiidditity is statically divided into o fltf slots A “TDMA Round” gives opportunity to nodes A sequence of TDMA rounds is called a “cluster cycle” The l en gth o f a c luste r cyc l e determines t he pe riod ic ity o f t he TDMA syste m E.g: Time-Triggered Protocol (TTP), Kopetz, 1993 Fault-tolerant real-time systems Has predictable message transmission, acknowledgment in group communitilkication, clock sync hron itiization, rap iddid mode changes, and redundancy management no extra messages to implement and only a small overhead in the message size.
50 TTP TTP/C (cluster communication) or TTP/A (field bus) A cluster of fault-tolerant units is connected by the network Nodes are assumed to be fail silent either deliver a message in their slot or not at all A node is the smallest replaceable unit consists of host and comm. controller; Communication-network interface is formed by a dual-ported RAM Comm. controller has local memory to hold message descriptor list; it determines when a node can send or expect to receive a message.
TTP: Global Time Tick Host Computer
CNI in DPRAM
TTP Control Data Protocol
TTP Node in ROM Processor
BG BG BG: Bus Guardian TTP Bus 51 OSI Protocols for RT
OSI Reference Model Seven layer: Application Presentation Session Transport Network Data Link Physical From conceptual reference to implementation reference A stack of Positive -Acknowledgment-or-Retransmission (PAR) protocols must be executed before a message is delivered at the application process. Generally the assumptions do not match RT needs.
52 Su mmary of RT Commu nication
ShdliScheduling an d a dm iss ion con tro l FCFS RR WRR GPS WFQ Rate control scheduling Real-time communication architectures CSMA/CD-LON, CSMA/CD-CAN, Token Bus, CM, TDMA-TTP
53 Sources and References
Hermann Kopetz, “Real-Time Systems,” Kluwer, 1997. Keshav: “An Engineering Approach to Computer Networking,” Addison-Wesley, 1997 Mani Srivastava @ UCLA Nick McKeown @ Stanford U. Ra jes h Gup ta @ UCSD
54