Group Ratio Round-Robin: O(1) Proportional Share Scheduling for Uniprocessor and Multiprocessor Systems Bogdan Caprita, Wong Chun Chan, Jason Nieh, Clifford Stein,∗ and Haoqiang Zheng Department of Computer Science Columbia University Email: {bc2008, wc164, nieh, cliff, hzheng}@cs.columbia.edu
Abstract
We present Group Ratio Round-Robin (GR3), the first pro- work has been done to provide proportional share schedul- portional share scheduler that combines accurate propor- ing on multiprocessor systems, which are increasingly com- tional fairness scheduling behavior with O(1) scheduling mon especially in small-scale configurations with two or overhead on both uniprocessor and multiprocessor systems. four processors. Over the years, a number of scheduling GR3 uses a simple grouping strategy to organize clients mechanisms have been proposed, and much progress has into groups of similar processor allocations which can be been made. However, previous mechanisms have either su- more easily scheduled. Using this strategy, GR3 combines perconstant overhead or less-than-ideal fairness properties. the benefits of low overhead round-robin execution with a We introduce Group Ratio Round-Robin (GR3), the 3 novel ratio-based scheduling algorithm. GR introduces a first proportional share scheduler that provides constant novel frontlog mechanism and weight readjustment algo- fairness bounds on proportional sharing accuracy with O(1) 3 rithm to operate effectively on multiprocessors. GR pro- scheduling overhead for both uniprocessor and small-scale vides fairness within a constant factor of the ideal general- multiprocessor systems. In designing GR3, we observed ized processor sharing model for client weights with a fixed that accurate, low-overhead proportional sharing is easy to upper bound and preserves its fairness properties on multi- achieve when scheduling a set of clients with equal pro- 3 processor systems. We have implemented GR in Linux cessor allocations, but is harder to do when clients require and measured its performance. Our experimental results very different allocations. Based on this observation, GR3 3 show that GR provides much lower scheduling overhead uses a simple client grouping strategy to organize clients and much better scheduling accuracy than other schedulers into groups of similar processor allocations which can be commonly used in research and practice. more easily scheduled. Using this grouping strategy, GR3 combines the benefits of low overhead round-robin execu- 1 Introduction tion with a novel ratio-based scheduling algorithm. GR3 uses the same basic uniprocessor scheduling al- Proportional share resource management provides a flexible gorithm for multiprocessor scheduling by introducing the and useful abstraction for multiplexing processor resources notion of a frontlog. On a multiprocessor system, a client among a set of clients. Proportional share scheduling has a may not be able to be scheduled to run on a processor be- clear colloquial meaning: given a set of clients with asso- cause it is currently running on another processor. To pre- ciated weights, a proportional share scheduler should allo- serve its fairness properties, GR3 keeps track of a frontlog cate resources to each client in proportion to its respective per client to indicate when the client was already running weight. However, developing processor scheduling mech- but could have been scheduled to run on another processor. anisms that combine good proportional fairness scheduling It then assigns the client a time quantum that is added to its behavior with low scheduling overhead has been difficult to allocation on the processor it is running on. The frontlog achieve in practice. For many proportional share scheduling ensures that a client receives its proportional share alloca- mechanisms, the time to select a client for execution grows tion while also taking advantage of any cache affinity by with the number of clients. For server systems which may continuing to run the client on the same processor. service large numbers of clients, the scheduling overhead of 3 algorithms whose complexity grows linearly with the num- GR provides a simple weight readjustment algo- ber of clients can waste more than 20 percent of system re- rithm that takes advantage of its grouping strategy. On sources [3] for large numbers of clients. Furthermore, little a multiprocessor system, proportional sharing is not fea- sible for some client weight assignments, such as having ∗also in Department of IEOR one client with weight 1 and another with weight 2 on a
USENIX Association 2005 USENIX Annual Technical Conference 337 two-processor system. By organizing clients with similar on some metric of fairness, and has a time complexity that weights into groups, GR3 adjusts for infeasible weight as- grows with the number of clients. The other way is to ad- signments without the need to order clients, resulting in just the size of a client’s time quantum so that it runs longer lower scheduling complexity than previous approaches [7]. for a given allocation, as is done in weighted round-robin We have analyzed GR3 and show that with only O(1) (WRR). This is fast, providing constant time complexity overhead, GR3 provides fairness within O(g2) of the ideal scheduling overhead. However, allowing a client to mo- Generalized Processor Sharing (GPS) model [16], where g, nopolize the resource for a long period of time results in ex- the number of groups, grows at worst logarithmically with tended periods of unfairness to other clients which receive the largest client weight. Since g is in practice a small con- no service during those times. The unfairness is worse with stant, GR3 effectively provides constant fairness bounds skewed weight distributions. with only O(1) overhead. Moreover, we show that GR3 GR3 is a proportional share scheduler that matches uniquely preserves its worst-case time complexity and fair- with O(1) time complexity of round-robin scheduling but ness properties for multiprocessor systems. provides much better proportional fairness guarantees in We have implemented a prototype GR3 processor practice. At a high-level, the GR3 scheduling algorithm scheduler in Linux, and compared it against uniproces- can be briefly described in three parts: sor and multiprocessor schedulers commonly used in prac- tice and research, including the standard Linux sched- 1. Client grouping strategy: Clients are separated into uler [2], Weighted Fair Queueing (WFQ) [11], Virtual-Time groups of clients with similar weight values. The k Round-Robin (VTRR) [17], and Smoothed Round-Robin group of order is assigned all clients with weights 2k 2k+1 − 1 k ≥ 0 (SRR) [9]. We have conducted both simulation studies and between to ,where . kernel measurements on micro-benchmarks and real ap- 2. Intergroup scheduling: Groups are ordered in a list 3 plications. Our results show that GR can provide more from largest to smallest group weight, where the group than an order of magnitude better proportional sharing ac- weight of a group is the sum of the weights of all curacy than these other schedulers, in some cases with more clients in the group. Groups are selected in a round- than an order of magnitude less overhead. These results robin manner based on the ratio of their group weights. 3 demonstrate that GR can in practice deliver better propor- If a group has already been selected more than its pro- tional share control with lower scheduling overhead than portional share of the time, move on to the next group 3 these other approaches. Furthermore, GR is simple to im- in the list. Otherwise, skip the remaining groups in plement and easy to incorporate into existing scheduling the group list and start selecting groups from the be- frameworks in commodity operating systems. ginning of the group list again. Since the groups with This paper presents the design, analysis, and evalua- larger weights are placed first in the list, this allows 3 tion of GR . Section 2 describes the uniprocessor schedul- them to get more service than the lower-weight groups ing algorithm. Section 3 describes extensions for multipro- at the end of the list. cessor scheduling, which we refer to as GR3MP. Section 4 analyzes the fairness and complexity of GR3. Section 5 3. Intragroup scheduling: From the selected group, a presents experimental results. Section 6 discusses related client is selected to run in a round-robin manner that work. Finally, we present some concluding remarks and accounts for its weight and previous execution history. directions for future work. Using this client grouping strategy, GR3 separates scheduling in a way that reduces the need to schedule enti- 2 GR3 Uniprocessor Scheduling ties with skewed weight distributions. The client grouping Uniprocessor scheduling, the process of scheduling a time- strategy limits the number of groups that need to be sched- multiplexed resource among a set of clients, has two basic uled since the number of groups grows at worst logarithmi- steps: 1) order the clients in a queue, 2) run the first client in cally with the largest client weight. Even a very large 32-bit the queue for its time quantum, which is the maximum time client weight would limit the number of groups to no more interval the client is allowed to run before another schedul- than 32. The client grouping strategy also ensures that all ing decision is made. We refer to the units of time quanta clients within a group have weight within a factor of two. As a result, the intragroup scheduler never needs to sched- as time units (tu) rather than an absolute time measure such 3 as seconds. A scheduler can therefore achieve proportional ule clients with skewed weight distributions. GR groups sharing in one of two ways. One way, often called fair are simple lists that do not need to be balanced; they do not queueing [11, 18, 28, 13, 24, 10] is to adjust the frequency require any use of more complex balanced tree structures. that a client is selected to run by adjusting the position of 3 the client in the queue so that it ends up at the front of the 2.1 GR Definitions queue more or less often. However, adjusting the client’s We now define the state GR3 associates with each position in the queue typically requires sorting clients based client and group, and describe in detail how GR3 uses
338 2005 USENIX Annual Technical Conference USENIX Association Cj Client j. (also called ’task’ j) cuss dynamic changes in a client’s run state in Section 2.3. 3 φC The weight assigned to client C. We first focus on the GR intergroup scheduling algorithm, 3 φj Shorthand notation for φCj . then discuss the GR intragroup scheduling algorithm. 3 DC The deficit of C. The GR intergroup scheduling algorithm uses the N The number of runnable clients. ratio of the group weights of successive groups to deter- g The number of groups. mine which group to select. The next group to schedule Gi i’th group in the list ordered by weight. is selected using only the state of successive groups in the |G| The number of clients in group G. group list. Given a group Gi whose weight is x times larger G(C) The group to which C belongs. than the group weight of the next group Gi+1 in the group 3 list, GR will select group Gi x times for every time that it ΦG The group weight of G: PC∈G φC . selects Gi+1 in the group list to provide proportional share Φi Shorthand notation for ΦGi . allocation among groups. σG The order of group G. 3 G σG To implement the algorithm, GR maintains the to- φmin Lower bound for client weights in G: 2 . tal work done by group Gi in a variable Wi. An index i to wC The work of client C. tracks the current group and is initialized to 1. The schedul- wj Shorthand notation for wC . j ing algorithm then executes the following simple routine: WG The group work of group G.
Wi Shorthand notation for WGi . N g INTERGROUP-SCHEDULE() ΦT Total weight: Pj=1 φj = Pi=1 Φi. N g 1 C ← INTRAGROUP-SCHEDULE(Gi) WT Total work: P wj = P Wi. j=1 i=1 2 W ← W +1 φC i i eC Service error of client C: wC − WT W +1 Φ ΦT 3 if i
3 Table 1: GR terminology Let us negate (1) under the form:
that state to schedule clients. Table 1 lists terminology Wi +1 Wi+1 +1 3 ≤ (2) we use. For each client, GR maintains the following Φi Φi+1 three values: weight, deficit, and run state. Each client re- ceives a resource allocation that is directly proportional to We will call this relation the well-ordering condition of two 3 its weight. A client’s deficit tracks the number of remaining consecutive groups. GR works to maintain this condition time quanta the client has not received from previous allo- true at all times. The intuition behind (2) is that we would cations. A client’s run state indicates whether or not it can like the ratio of the work of Gi and Gi+1 to match the ratio be executed. A client is runnable if it can be executed. of their respective group weights after GR3 has finished se- For each group, GR3 maintains the following four lecting both groups. Recall, Φi ≥ Φi+1. Each time a client 3 values: group weight, group order, group work, and current from Gi+1 is run, GR would like to have run Φi/Φi+1 3 client. The group weight is the sum of the corresponding worth of clients from Gi. (1) says that GR should not run weights of the clients in the group run queue. A group with a client from Gi and increment Gi’s group work if it will group order k contains the clients with weights between 2k make it impossible for Gi+1 to catch up to its proportional to 2k+1 − 1.Thegroup work is the total execution time share allocation by running one of its clients once. clients in the group have received. The current client is the To illustrate how intergroup scheduling works, Fig- most recently scheduled client in the group’s run queue. ure 1 shows an example with three clients C1, C2,and 3 C3, which have weights of 5, 2, and 1, respectively. The GR also maintains the following scheduler state: 3 time quantum, group list, total weight, and current group. GR grouping strategy would place each Ci in group Gi, The group list is a sorted list of all groups containing ordering the groups by weight: G1, G2,andG3 have or- runnable clients ordered from largest to smallest group ders 2, 1 and 0 and weights of 5, 2, and 1 respectively. In this example, each group has only one client so there weight, with ties broken by group order. The total weight is 3 the sum of the weights of all runnable clients. The current is no intragroup scheduling. GR would start by selecting group is the most recently selected group in the group list. group G1, running client C1, and incrementing W1.Based on (1), W1+1 =2< Φ1 =2.5,soGR3 would select W2+1 Φ2 G again and run client C . After running C , G ’s work 2.2 Basic GR3 Algorithm 1 1 1 1 would be 2 so that the inequality in (1) would hold and GR3 We initially only consider runnable clients in our dis- would then move on to the next group G2 and run client C2. cussion of the basic GR3 scheduling algorithm. We dis- Based on (1), W2+1 =2≤ Φ2 =2,soGR3 would reset W3+1 Φ3
USENIX Association 2005 USENIX Annual Technical Conference 339 G 5 | 1 5 | 2 5 | 3 5 | 35 | 4 5 | 5 5 | 6 5 | 6 φC φC 1 1)−b G +DC (r−1)c, with DC (0) = G . Thus, in each φmin φmin G2 2 | 1 2 | 1 2 | 12 | 2 2 | 2 2 | 2 2 | 2 2 | 3 round, C is allotted one time quantum plus any additional G 1 | 1 1 | 1 1 | 1 1 | 1 1 | 1 1 | 1 1 | 1 1 | 1 leftover from the previous round, and DC (r) keeps track of 3 the amount of service that C missed because of rounding C C C C C C C C down its allocation to whole time quanta. We observe that 1 1 2 1 1 1 2 3 t 0 ≤ DC (r) < 1 after any round r so that any client C will be allotted one or two time quanta. Note that if a client Figure 1: GR3 intergroup scheduling. At each time step, is allotted two time quanta, it first executes for one time the shaded box contains the pair ΦG | WG +1for the group quantum and then executes for the second time quantum G before it is selected. the next time the intergroup scheduler selects its respective group again (in general, following a timespan when clients
the current group to the largest weight group G1 and run belonging to other groups get to run). 3 client C1. Based on (1), C1 would be run for three time To illustrate how GR works with intragroup quanta before selecting G2 again to run client C2.After scheduling, Figure 2 shows an example with six clients running C2 the second time, W2 would increase such that C1 through C6 with weights 12, 3, 3, 2, 2, and 2, re- W +1 Φ 3 2 =3> 2 =2,soGR would then move on to spectively. The six clients will be put in two groups G1 W3+1 Φ3 the last group G3 and run client C3. The resulting schedule and G2 with respective group order 1 and 3 as follows: would then be: G1, G1, G2, G1, G1, G1, G2, G3. Each G1 = {C2,C3,C4,C5,C6} and G2 = {C1}. The weight 3 group therefore receives its proportional allocation in ac- of the groups are Φ1 =Φ2 =12. GR intergroup schedul- cordance with its respective group weight. ing will consider the groups in this order: G1, G2, G1, 3 The GR intragroup scheduling algorithm selects a G2, G1, G2, G1, G2, G1, G2, G1, G2. G2 will sched- client from the selected group. All clients within a group ule client C1 every time G2 is considered for service since G1 have weights within a factor of two, and all client weights it has only one client. Since φmin =2, the normalized in a group G are normalized with respect to the minimum weights of clients C2, C3, C4, C5,andC6 are 1.5, 1.5, 1, G σG possible weight, φmin =2 , for any client in the group. 1, and 1, respectively. In the beginning of round 1 in G1, GR3 then effectively traverses through a group’s queue each client starts with 0 deficit. As a result, the intragroup in round-robin order, allocating each client its normalized scheduler will run each client in G1 for one time quantum 3 weight worth of time quanta. GR keeps track of subuni- during round 1. After the first round, the deficit for C2, C3, tary fractional time quanta that cannot be used and accumu- C4, C5,andC6 are 0.5, 0.5, 0, 0, and 0. In the beginning of G1 lates them in a deficit value for each client. Hence, each round 2, each client gets another φi/φmin allocation, plus client is assigned either one or two time quanta, based on any deficit from the first round. As a result, the intragroup the client’s normalized weight and its previous allocation. scheduler will select clients C2, C3, C4, C5,andC6 to run More specifically, the GR3 intragroup scheduler con- in order for 2, 2, 1, 1, and 1 time quanta, respectively, dur- siders the scheduling of clients in rounds. A round is one ing round 2. The resulting schedule would then be: C2, C1, pass through a group G’s run queue of clients from begin- C3, C1, C4, C1, C5, C1, C6, C1, C2, C1, C2, C1, C3, C1, ning to end. The group run queue does not need to be sorted C3, C1, C4, C1, C5, C1, C6, C1. in any manner. During each round, the GR3 intragroup al- gorithm considers the clients in round-robin order and exe- 2.3 GR3 Dynamic Considerations cutes the following simple routine: We now discuss how GR3 allows clients to be dynamically created, terminated, or change run state. INTRAGROUP-SCHEDULE(G) Runnable clients can be selected for execution by the sched- 1 C ← G[k] £ k is the current position in the round uler, while clients that are not runnable cannot. With no loss 2 if DC < 1 of generality, we assume that a client is created before it can 3 then k ← (k +1)mod |G| become runnable, and a client becomes not runnable before 4 C ← G[k] it is terminated. As a result, client creation and termination G 3 5 DC ← DC + φC /φmin have no effect on the GR run queues. 6 D ← D − 1 C C When a client C with weight φC becomes runnable, 7 return C it is inserted into group G = G(C) such that φC is between 2σG and 2σG+1 − 1. If the group was previously empty, a For each runnable client C, the scheduler determines new group is created, the client becomes the current client the maximum number of time quanta that the client can be of the group, and g, the number of groups, is incremented. φC selected to run in this round as b G +DC(r−1)c. DC (r), 3 φmin If the group was not previously empty, GR inserts the the deficit of client C after round r, is the time quantum client into the respective group’s run queue right before the φC fraction left over after round r: DC(r)= G + DC (r − current client; it will be serviced after all of the other clients φmin
340 2005 USENIX Annual Technical Conference USENIX Association G 1 G2 G 1 G2 G 1 G2 G 1 G2 G 1 G2 G 1 G2 G 1 G2 G 1 G2 G 1 G2 G 1 G2 G 1 G2 G 1 G2 0 0 0.5 000000000000 0.5 0 0.5 0 0.5 0 0.5 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0.5 0.5 0.5 0.5 0.5 0.5 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
C C C C C C C C C C C C C C C C C C C C C C C C 2 1 3 1 4 1 5 1 6 1 2 1 2 1 3 1 3 1 4 1 5 1 6 1 t
Figure 2: GR3 intragroup scheduling. At each time step, the shaded box contains the deficit of the client before it is run.
in the group have first been considered for scheduling. The by the same rules as for client insertion, depending on the initial deficit DC will be initialized to 0. new position of the group and its next neighbor. After per- When a newly runnable client C is inserted into its forming these removal operations, GR3 resumes schedul- respective group G, the group needs to be moved to its new ing from the largest weight group in the system. position on the ordered group list based on its new group Whenever a client C blocks during round r,weset k G(C) weight. Let this new position be . The corresponding DC (r)=min(DC (r − 1)+ φC /φmin −dwe, 1),wherew group work and group weight of G need to be updated and is the service that the client received during round r until it the client’s deficit needs to be initialized. The group weight blocked. This preserves the client’s credit in case it returns is simply incremented by the client’s weight. We also want by the next round, while also limiting the deficit to 1 so that to scale the group work of G such that the work ratio of a client cannot gain credit by blocking. However, the group consecutive groups will continue to be proportional to their consumes 1 tu (its work is incremented) no matter how long weight ratio: the client runs. Therefore, the client forfeits its extra credit
ΦG whenever it is unable to consume its allocation. (Wk+1 +1) − 1 if k
USENIX Association 2005 USENIX Annual Technical Conference 341 P Number of processors. C C C C C C t p 1 2 2 3 2 2 3 ℘k Processor k. F =1 F =1 C(℘) Client running on processor ℘. C1 C1 F C C Frontlog for client . C 2 C 1 C 2 C 1 C 1 C 3 C 2 C 1 C 2 C 1 C 1 C 3 Table 2: GR3MP terminology C C C C C C t p 2 1 1 1 1 1 1
3 F =0 F =0 processor needs to be scheduled, GR MP selects the client C1 C1 that would run next under GR3, essentially scheduling mul- tiple processors from its central run queue as GR3 sched- Figure 3: GR3 multiprocessor scheduling. The two pro- ules a single processor. However, there is one obstacle to cessors schedule either from the central queue, or use the simply applying a uniprocessor algorithm on a multipro- frontlog mechanism when the task is already running. cessor system. Each client can only run on one processor at any given time. As a result, GR3MP cannot select a CHEDULE(℘k) client to run that is already running on another processor MP-S even if GR3 would schedule that client in the uniproces- 1 C ← C(℘k) £ Client just run sor case. For example, if GR3 would schedule the same 2 if C = NIL client consecutively, GR3MP cannot schedule that client 3 then if N
0 GR3MP introduces the notion of a frontlog. The front- 6 then FC ← FC − 1 k 7 return C log FC for some client C running on a processor ℘ (C = C(℘k)) is defined as the number of time quanta for C accu- 8 C ← INTERGROUP-SCHEDULE() mulated as C gets selected by GR3 and cannot run because 9 while ∃℘ s.t. C = C(℘) k 10 do F ← F +1 it is already running on ℘ . The frontlog FC is then queued C C up on ℘k. 11 C ← INTERGROUP-SCHEDULE() 12 return C Given a client that would be scheduled by GR3 but is already running on another processor, GR3MP uses the 3 frontlog to assign the client a time quantum now but de- To illustrate GR MP scheduling, Figure 3 shows an fer the client’s use of it until later. Whenever a proces- example on a dual-processor system with three clients C1, sor finishes running a client for a time quantum, GR3MP C2,andC3 of weights 3, 2, and 1, respectively. C1 and checks whether the client has a non-zero frontlog, and, if C2 will then be part of the order 1 group (assume C2 is so, continues running the client for another time quantum before C1 in the round-robin queue of this group), whereas 3 and decrements its frontlog by one, without consulting the C3 is part of the order 0 group. The GR schedule is C2, 1 2 central queue. The frontlog mechanism not only ensures C1, C2, C1, C1, C3. ℘ will then select C2 to run, and ℘ 1 3 that a client receives its proportional share allocation, it also selects C1.When℘ finishes, according to GR , it will 2 takes advantage of any cache affinity by continuing to run select C2 once more, whereas ℘ selects C1 again. When 1 3 the client on the same processor. ℘ again selects the next GR client, which is C1, it finds that it is already running on ℘2 and thus we set F =1 When a processor finishes running a client for a time C1 and select the next client, which is C , to run on ℘1.When quantum and its frontlog is zero, we call the processor idle. 3 ℘2 finishes running C for its second time quantum, it finds GR3MP schedules a client to run on the idle processor 1 F =1,setsF =0and continues running C without by performing a GR3 scheduling decision on the central C1 C1 1 any scheduling decision on the GR3 queue. queue. If the selected client is already running on some other processor, we increase its frontlog and repeat the GR3 3.2 GR3MP Dynamic Considerations scheduling, each time incrementing the frontlog of the se- lected client, until we find a client that is not currently run- GR3MP basically does the same thing as the GR3 ning. We assign this client to the idle processor for one time algorithm under dynamic considerations. However, the quantum. This description assumes that there are least P +1 frontlogs used in GR3MP need to be accounted for ap- clients in the system. Otherwise, scheduling is easy: an idle propriately. If some processors have long frontlogs for their processor will either run the client it just ran, or idles un- currently running clients, newly arriving clients may not be til more clients arrive. In effect, each client will simply be run by those processors until their frontlogs are processed, assigned its own processor. Whenever a processor needs to resulting in bad responsiveness for the new clients. Al- perform a scheduling decision, it thus executes the follow- though in between any two client arrivals or departures, ing routine: some processors must have no frontlog, the set of such
342 2005 USENIX Annual Technical Conference USENIX Association processors can be as small as a single processor. In this S1,N = C1,C2,...,CN with φ1 ≥ φ2 ≥ ... ≥ φN . case, newly arrived clients will end up competing with other We call the subsequence Sk,N = Ck,Ck+1,...,.CN Q- clients already in the run queue only for those few proces- ≤ 1 N feasible,ifφk Q Pj=k φj . sors, until the frontlog on the other processors is exhausted. GR3MP provides fair and responsive allocations by Lemma 1. The client mix in the system is feasible if and creating frontlogs for newly arriving clients. Each new only if S1,N is P -feasible. client is assigned a frontlog equal to a fraction of the to- ΦT tal current frontlog in the system based on its proportional Proof. If φ1 > P , C1 is infeasible, so the mix is infea- ≤ ΦT share. Each processor now maintains a queue of frontlog sible. Conversely, if φ1 P , then for any client Cl, ≤ ≤ ΦT clients and a new client with a frontlog is immediately as- φl φ1 P , implying all clients are feasible. The mix signed to one of the processor frontlog queues. Rather than ΦT 1 N is then feasible ⇐⇒ φ1 ≤ = Pj=1 φj , or, equiva- running its currently running client until it completes its P P lently, S1,N is P -feasible. frontlog, each processor now round robins among clients in its frontlog queue. Given that frontlogs are small in prac- Lemma 2. S is Q-feasible ⇒ S is (Q − 1)- tice, round-robin scheduling is used for frontlog clients for k,N k+1,N feasible. its simplicity and fairness. GR3MP balances the frontlog load on the processors by placing new frontlog clients on 1 N Proof. φk ≤ P φj ⇐⇒ Qφk ≤ φk + the processor with the smallest frontlog summed across all Q j=k N φ ⇐⇒ φ ≤ 1 N φ φ ≤ its frontlog clients. Pj=k+1 j k Q−1 Pj=k+1 j.Since k+1 φ , the lemma follows. More precisely, whenever a client C arrives, and k it belongs in group G(C), GR3MP performs the same group operations as in the single processor GR3 algorithm. The feasibility problem is then to identify the least GR3MP finds the processor ℘k with the smallest front- k (denoted the feasibility threshold, f) such that Sk,N is log, then creates a frontlog for client C on ℘k of length (P − k +1)-feasible. If f =1, then the client mix is feasi- φ F = F C ,whereF is the total frontlog on all the ble. Otherwise, the infeasible set S1,f−1 = C1,...,Cf−1 C T ΦT T processors. Let C0 = C(℘k). Then, assuming no further contains the infeasible clients, whose weight needs to be clients arrive, ℘k will round-robin between C and C0 and scaled down to 1/P of the resulting total weight. The car- 0 dinality f − 1 of the infeasible set is less than P . However, run C for FC and C for FC0 time quanta. S When a client becomes not runnable, GR3MP uses the sorted sequence 1,N is expensive to maintain, such that the same lazy removal mechanism used in GR3. If it is traversing it and identifying the feasibility threshold is not removed from the run queue and has a frontlog, GR3MP an efficient solution. GR3MP leverages its grouping strategy to perform simply discards it since each client is assigned a frontlog 3 based on the current state of the system when it becomes fast weight readjustment. GR MP starts with the unmod- runnable again. ified client weights, finds the set I of infeasible clients, and adjust their weights to be feasible. To construct I, the al- 3.3 GR3MP Weight Readjustment gorithm traverses the list of groups in decreasing order of their group order σG, until it finds a group not all of whose Since no client can run on more than one processor clients are infeasible. We denote by |I| the cardinality of I at a time, no client can consume more than a 1/P fraction and by ΦI the sum of weights of the clients in I, PC∈I φC . of the processing in a multiprocessor system. A client C The GR3MP weight readjustment algorithm is as follows: with weight φC greater than ΦT /P is considered infeasi- ble since it cannot receive its proportional share allocation WEIGHT-READJUSTMENT() φC /ΦT without using more than one processor simultane- ously. GR3MP should then give the client its maximum 1RESTORE-ORIGINAL-WEIGHTS ←∅ possible service, and simply assign such a client its own 2 I 3 G ← greatest order group processor to run on. However, since the scheduler uses − − |G|
USENIX Association 2005 USENIX Annual Technical Conference 343 f The correctness of the algorithm is based on Lemma weights of all feasible clients, ΦT =ΦT − ΦI . We can 2. Let some group G span the subsequence Si,j of the se- now compute the new total weight in the system as ΦT = f f σG+1 − ≥ ≥ P | | x quence of ordered clients S1,N .Then2 1 φi P −|I| ΦT , namely the solution to the equation ΦT + I P = σG . ..≥ φj ≥ 2 and it is easy to show: x. Once we have the adjusted ΦT , we change all the weights ΦT − − for the infeasible clients in I to . Lemma 6 in Section 4.2 • 2σG > ΦT ΦI ΦG ⇒ j
3 the service received by client C and its share of the total GR MP can alternatively use a more complicated φC work done by the processor: eC = wC − WT . A pos- but lower time complexity divide-and-conquer algorithm to ΦT itive service time error indicates that a client has received find the infeasible clients in G. In this case, GR3MP par- more than its ideal share over a time interval; a negative titions G around its median C¯ into G ,thesetofG clients S error indicates that it has received less. To be precise, the that have weight less than φC¯ and GB, the set of G clients error eC measures how much time a client C has received that have weight larger than φ ¯ . By Lemma 2, if C¯ is fea- C beyond its ideal allocation. A proportional share scheduler sible, G ∪{C¯} is feasible, and we recurse on G . Other- S B should minimize the absolute value of the allocation error wise, all clients in G ∪{C¯} are infeasible, and we recurse B of all clients with minimal scheduling overhead. on G to find all infeasible clients. The algorithm finishes S We provide bounds on the service error of GR3 and when the set we need to recurse on is empty: GR3MP. To do this, we define two other measures of ser- vice error. The group service time error is a similar mea- INFEASIBLE(G, Q, Φ) sure for groups that quantifies the fairness of allocating the 1 if G = ∅ processor among groups: E = W − W ΦG .Thegroup- G G T ΦT 2 then return ∅ relative service time error represents the service time error ¯ 3 C ← MEDIAN(G) of client C if there were only a single group G = G(C) 4 (GS,GB ) ← PARTITION(G, φC¯ ) in the scheduler and is a measure of the service error of a Φ−ΦG 5 if φ ¯ > B client with respect to the work done on behalf of its group: C Q−|GB | φC ∪{¯}∪ eC,G = wC − WG . We first show bounds on the group 6 then return GB C ΦG service error of the intergroup scheduling algorithm. We INFEASIBLE(GS ,Q−|GB|−1, Φ − ΦGB − φC¯ ) 7 else return INFEASIBLE(GB ,Q,Φ) then show bounds on the group-relative service error of the intragroup scheduling algorithm. We combine these results Once all infeasible clients have been identified, to obtain the overall client service error bounds. We also WEIGHT-READJUSTMENT() determines the sum of the discuss the scheduling overhead of GR3 and GR3MP in
344 2005 USENIX Annual Technical Conference USENIX Association terms of their time complexity. We show that both algo- Dynamic Fairness of GR3 We can consider a client ar- rithms can make scheduling decisions in O(1) time with rival or removal as an operation where a group is first re- O(1) service error given a constant number of groups. Due moved from the group list and added in a different place to space constraints, most of the proofs are omitted. Further with a different weight. We argue that fairness is pre- proof details are available in [5]. served by these operations: when group Gk is removed, then Gk−1, Gk,andGk+1 were well-ordered as defined 3 4.1 Analysis of GR in (2), so after the removal, Gk−1 and Gk+1, now neigh- bors, will be well-ordered by transitivity. When a group, Intergroup Fairness For the case when the weight ratios call it G , is inserted between G and G ,itcanbe of consecutive groups in the group list are integers, we get i+(1/2) i i+1 proven that the work readjustment formula in Section 2.3 the following: ensures Gi+(1/2) and Gi+1 are well-ordered. In the case of Φj Gi and Gi+(1/2), we can show that we can achieve well- Lemma 3. If ∈ N, 1 ≤ j
(g−k)(g−k−1) Φk group in the list, or reverts to the first one, which takes con- Lemma 4. For any group Gk, − − 1 < 2 ΦT stant time. The intragroup scheduler is even simpler, as it E 3 (g−1)(g−2) φC Overall Fairness of GR Based on the identity eC = Theorem 2. − − 4 USENIX Association 2005 USENIX Annual Technical Conference 345 that is not running on any other processor. Since for each SRR [9]. The simulator enabled us to isolate the impact such client, we increase its allocation on the processor it of the scheduling algorithms themselves and examine the runs, the amortized time complexity remains O(1). The up- scheduling behavior of these different algorithms across per bound on the time that any single scheduling decision hundreds of thousands of different combinations of clients takes is given by the maximum length of any scheduling with different weight values. sequence of GR3 consisting of only some fixed subset of Section 5.2 presents detailed measurements of real P − 1 clients. kernel scheduler performance by comparing our prototype GR3 Linux implementation against the standard Linux Theorem 3. The time complexity per scheduling decision scheduler, a WFQ scheduler, and a VTRR scheduler. The in GR3MP is bounded above by (g−k)(g−k+1) +(k + 2 experiments we have done quantify the scheduling over- 1)(g − k +1)P +1 1 ≤ k ≤ g where . head and proportional share allocation accuracy of these Thus, the length of any schedule consisting of at most schedulers in a real operating system environment under a P − 1 clients is O(g2P ). Even when a processor has front- number of different workloads. logs for several clients queued up on it, it will schedule in All our kernel scheduler measurements were per- O(1) time, since it performs round-robin among the front- formed on an IBM Netfinity 4500 system with one or two logged clients. Client arrivals and departures take O(g) 933 MHz Intel Pentium III CPUs, 512 MB RAM, and 9 time because of the need to readjust group weights in the GB hard drive. The system was installed with the Debian saved list of groups. Moreover, if we also need to use GNU/Linux distribution version 3.0 and all schedulers were the weight readjustment algorithm, we incur an additional implemented using Linux kernel version 2.4.19. The mea- O(P ) overhead on client arrivals and departures. surements were done by using a minimally intrusive trac- ing facility that writes timestamped event identifiers into Lemma 6. The complexity of the weight readjustment al- a memory log and takes advantage of the high-resolution gorithm is O(P ). clock cycle counter available with the Intel CPU, providing Proof. Restoring the original weights will worst case touch measurement resolution at the granularity of a few nanosec- a number of groups equal to the number of previously in- onds. Getting a timestamp simply involved reading the feasible clients, which is O(P ). Identifying the infeasible hardware cycle counter register. We measured the times- clients involves iterating over at most P groups in decreas- tamp overhead to be roughly 35 ns per event. ing sequence based on group order, as described in Sec- The kernel scheduler measurements were performed tion 3.3. For the last group considered, we only attempt on a fully functional system. All experiments were per- to partition it into feasible and infeasible clients of its size formed with all system functions running and the system is less than 2P . Since partitioning of a set can be done in connected to the network. At the same time, an effort was linear time, and we recurse on a subset half the size, this made to eliminate variations in the test environment to make operation is O(P ) as well. the experiments repeatable. For small P ,theO(P log(P )) sorting approach to 5.1 Simulation Studies determine infeasible clients in the last group considered is simpler and in practice performs better than the O(P ) re- We built a scheduling simulator that measures the cursive partitioning. Finally, altering the active group struc- service time error, described in Section 4, of a scheduler ture to reflect the new weights is a O(P + g) operation, as on a set of clients. The simulator takes four inputs, the two groups may need to be re-inserted in the ordered list of scheduling algorithm, the number of clients N, the total groups. sum of weights ΦT , and the number of client-weight combi- nations. The simulator randomly assigns weights to clients and scales the weights to ensure that they add up to ΦT . 5 Measurements and Results It then schedules the clients using the specified algorithm We have implemented GR3 uniprocessor and multiproces- as a real scheduler would, assuming no client blocks, and sor schedulers in the Linux operating system and measured tracks the resulting service time error. The simulator runs their performance. We present some experimental data the scheduler until the resulting schedule repeats, then com- quantitatively comparing GR3 performance against other putes the maximum (most positive) and minimum (most popular scheduling approaches from both industrial prac- negative) service time error across the nonrepeating portion tice and research. We have conducted both extensive sim- of the schedule for the given set of clients and weight as- ulation studies and detailed measurements of real kernel signments. This process is repeated for the specified num- scheduler performance on real applications. ber of client-weight combinations. We then compute the Section 5.1 presents simulation results comparing maximum service time error and minimum service time er- the proportional sharing accuracy of GR3 and GR3MP ror for the specified number of client-weight combinations against WRR, WFQ [18], SFQ [13], VTRR [17], and to obtain a “worst-case” error range. 346 2005 USENIX Annual Technical Conference USENIX Association Service Error Service Error Service Error Service Error 800 0 20000 9000 600 -200 10000 6000 400 -400 0 3000 200 -600 0 -10000 0 -800 -3000 65000 65000 65000 65000 130000 130000 130000 130000 195000 195000 195000 195000 260000 4000 2000 260000 4000 2000 260000 4000 2000 260000 4000 2000 8000 6000 8000 6000 8000 6000 8000 6000 Sum of weights Number of clients Sum of weights Number of clients Sum of weights Number of clients Sum of weights Number of clients Figure 4: WRR error Figure 5: WFQ error Figure 6: SFQ error Figure 7: VTRR error Service Error Service Error Service Error Scheduling Decisions per Task Selection 400 10 200 2 2 8 6 0 0 0 4 -200 -2 -2 2 -400 0 65000 65000 65000 65000 130000 130000 130000 130000 195000 195000 195000 195000 260000 4000 2000 260000 4000 2000 260000 4000 2000 260000 4000 2000 8000 6000 8000 6000 8000 6000 8000 6000 Sum of weights Number of clients Sum of weights Number of clients Sum of weights Number of clients Sum of weights Number of clients Figure 8: SRR error Figure 9: GR3 error Figure 10: GR3MP error Figure 11: GR3MP over- head To measure proportional fairness accuracy, we ran time error since it is bounded below at −1. Similarly, GR3 simulations for each scheduling algorithm on 45 different has a much smaller negative service error than SFQ, though combinations of N and ΦT (32 up to 8192 clients and SFQ’s positive error is less since it is bounded above at 1. 16384 up to 262144 total weight, respectively). Since the Considering the total service error range of each scheduler, proportional sharing accuracy of a scheduler is often most GR3 provides well over two orders of magnitude better pro- clearly illustrated with skewed weight distributions, one portional sharing accuracy than WRR, WFQ, SFQ, VTRR, of the clients was given a weight equal to 10 percent of and SRR. Unlike the other schedulers, these results show 3 ΦT . All of the other clients were then randomly assigned that GR combines the benefits of low service time errors weights to sum to the remaining 90 percent of ΦT . For each with its ability to schedule in O(1) time. pair (N,ΦT ), we ran 2500 client-weight combinations and Note that as the weight skew becomes more accentu- determined the resulting worst-case error range. ated, the service error can grow dramatically. Thus, increas- The worst-case service time error ranges for WRR, ing the skew from 10 to 50 percent results in more than WFQ, SFQ, VTRR, SRR, and GR3 with these skewed a fivefold increase in the error magnitude for SRR, WFQ, weight distributions are in Figures 4 to 9. Due to space and SFQ, and also significantly worse errors for WRR and constraints, WF2Q error is not shown since the results sim- VTRR. In contrast, the error of GR3 is still bounded by ply verify its known mathematical error bounds of −1 and small constants: −2.3 and 4.6. 1 tu. Each figure consists of a graph of the error range for We also measured the service error of GR3MP using the respective scheduling algorithm. Each graph shows two this simulator configured for an 8 processor system, where surfaces representing the maximum and minimum service the weight distribution was the same as for the uniprocessor time error as a function of N and ΦT for the same range simulations above. Note that the client given 0.1 of the total of values of N and ΦT . Figure 4 shows WRR’s service 1 weight was feasible, since 0.1 < 8 =0.125. Figure 10 time error is between −12067 tu and 23593 tu. Figure 5 shows GR3MP’s service error is between −2.5 tu and 2.8 shows WFQ’s service time error is between −1 tu and 819 tu, slightly better than for the uniprocessor case, a benefit tu, which is much less than WRR. Figure 6 shows SFQ’s of being able to run multiple clients in parallel. Figure 11 service time error is between −819 tu and 1 tu, which is shows the maximum number of scheduling decisions that almost a mirror image of WFQ. Figure 7 shows VTRR’s an idle processor needs to perform until it finds a client that service error is between −2129 tu and 10079 tu. Figure 8 is not running. This did not exceed seven, indicating that shows SRR’s service error is between −369 tu and 369 tu. the number of decisions needed in practice is well below In comparison, Figure 9 shows the service time er- the worst-case bounds shown in Theorem 3. ror for GR3 only ranges from −2.5 to 3.0 tu. GR3 has a smaller error range than all of the other schedulers mea- 5.2 Linux Kernel Measurements sured except WF2Q. GR3 has both a smaller negative and smaller positive service time error than WRR, VTRR, and To evaluate the scheduling overhead of GR3,we SRR. While GR3 has a much smaller positive service er- compare it against the standard Linux 2.4 scheduler, a WFQ ror than WFQ, WFQ does have a smaller negative service scheduler, and a VTRR scheduler. Since WF2Q has the- USENIX Association 2005 USENIX Annual Technical Conference 347 100 As shown in Figure 12, the increase in scheduling GR3 overhead as the number of clients increases varies a great ) VTRR 3 s WFQ [O(log N)] deal between different schedulers. GR has the smallest u ( t WFQ [O(N)] scheduling overhead. It requires roughly 300 ns to select a s o Linux c client to execute and the scheduling overhead is essentially 10 GR3 MP g n Linux MP constant for all numbers of clients. While VTRR schedul- i l 3 u ing overhead is also constant, GR has less overhead be- d e h cause its computations are simpler to perform than the vir- c s tual time calculations required by VTRR. In contrast, the e 1 g a overhead for Linux and for O(N) WFQ scheduling grows r e v linearly with the number of clients. Both of these sched- A ulers impose more than 200 times more overhead than GR3 0.1 when scheduling a mix of 400 clients. O(log N) WFQ has 0 50 100 150 200 250 300 350 400 much smaller overhead than Linux or O(N) WFQ, but it Number of clients still imposes significantly more overhead than GR3, with 8 times more overhead than GR3 when scheduling a mix of Figure 12: Average scheduling overhead 400 clients. Figure 12 also shows that GR3MP provides the same O(1) scheduling overhead on a multiprocessor, oretically the same time complexity as WFQ (but with although the absolute time to schedule is somewhat higher larger constants, because of the complexity of its steps), we 2 due to additional costs associated with scheduling in mul- present WFQ as a lower bound for the overhead of WF Q. tiprocessor systems. The results show that GR3MP pro- We present results from several experiments that quantify vides substantially lower overhead than the standard Linux how scheduling overhead varies as the number of clients scheduler, which suffers from complexity that grows lin- increases. For the first experiment, we measure scheduling early with the number of clients. Because of the impor- overhead for running a set of clients, each of which exe- tance of constant scheduling overhead in server systems, cuted a simple micro-benchmark which performed a few Linux has switched to Ingo Molnar’s O(1) scheduler in the operations in a while loop. A control program was used to Linux 2.6 kernel. As a comparison, we also repeated this fork a specified number of clients. Once all clients were microbenchmark experiment with that scheduler and found runnable, we measured the execution time of each schedul- that GR3 still runs over 30 percent faster. ing operation that occurred during a fixed time duration of 30 seconds. The measurements required two timestamps As another experiment, we measured the scheduling for each scheduling decision, so measurement error of 70 ns overhead of the various schedulers for hackbench [21], are possible due to measurement overhead. We performed a Linux benchmark used for measuring scheduler perfor- these experiments on the standard Linux scheduler, WFQ, mance with large numbers of processes entering and leav- VTRR, and GR3 for 1 to 400 clients. ing the run queue at all times. It creates groups of readers Figure 12 shows the average execution time required and writers, each group having 20 reader tasks and 20 writer by each scheduler to select a client to execute. Results for tasks, and each writer writes 100 small messages to each of GR3, VTRR, WFQ, and Linux were obtained on uniproces- the other 20 readers. This is a total of 2000 messages sent sor system, and results for GR3MP and LinuxMP were ob- per writer, per group, or 40000 messages per group. We tained running on a dual-processor system. Dual-processor ran a modified version of hackbench to give each reader results for WFQ and VTRR are not shown since MP-ready and each writer a random weight between 1 and 40. We implementations of them were not available. performed these tests on the same set of schedulers for 1 group up to 100 groups. Using 100 groups results in up to For this experiment, the particular implementation 8000 processes running. Because hackbench frequently in- details of the WFQ scheduler affect the overhead, so serts and removes clients from the run queue, the cost of we include results from two different implementations of client insertion and removal is a more significant factor for WFQ. In the first, labeled “WFQ [O(N)]”, the run queue this benchmark. The results show that the simple dynamic is implemented as a simple linked list which must be group adjustments described in Section 2.3 have low over- searched on every scheduling decision. The second, labeled head, since O(g) can be considered constant in practice. “WFQ [O(log N)]”, uses a heap-based priority queue with O(log N) insertion time. To maintain the heap-based pri- Figure 13 shows the average scheduling overhead for ority queue, we used a fixed-length array. If the number each scheduler. The average overhead is the sum of the of clients ever exceeds the length of the array, a costly ar- times spent on all scheduling events, selecting clients to run ray reallocation must be performed. Our initial array size and inserting and removing clients from the run queue, di- was large enough to contain more than 400 clients, so this vided by the number of times the scheduler selected a client additional cost is not reflected in our measurements. to run. The overhead in Figure 13 is higher than the av- 348 2005 USENIX Annual Technical Conference USENIX Association 1000 third experiment, we ran five virtual servers which assigned GR3 a random weight between 1 and 10 to each process. For the VTRR WFQ [O(log N)] Linux scheduler, weights were assigned by selecting nice 100 WFQ [O(N)] values appropriately. Figures 14 to 19 present the results LINUX GR3 MP from the first experiment with one server with weight 10 ) s Linux MP processes and all other servers with weight 1 processes. The u ( total load on the system for this experiment consisted of e 10 m i 600 processes running simultaneously. For illustration pur- T poses, only one process from each server is shown in the 1 figures. Conclusions drawn from the other experiments are the same; those results are omitted due to space constraints. GR3 and GR3MP provided the best overall propor- 0.1 tional fairness for these experiments while Linux provided 0 10 20 30 40 50 60 70 80 90 100 the worst overall proportional fairness. Figures 14 to 17 Number of Groups show the amount of processor time allocated to each client over time for the Linux scheduler, WFQ, VTRR, and GR3. Figure 13: Hackbench weighted scheduling overhead All of the schedulers except GR3 and GR3MP have a pronounced “staircase” effect for the search engine process erage cost per schedule in Figure 12 for all the schedulers with weight 10, indicating that CPU resources are provided measured since Figure 13 includes a significant component in irregular bursts over a short time interval. For the ap- of time due to client insertion and removal from the run plications which need to provide interactive responsiveness GR3 queue. still has by far the smallest scheduling over- to web users, this can result in extra delays in system re- head among all the schedulers measured. The overhead sponse time. It can be inferred from the smoother curves for GR3 remains constant while the overhead for O(log N) of Figure 17 that GR3 and GR3MP provide fair resource O(N) WFQ, WFQ, VTRR, and Linux grows with the num- allocation at a finer granularity than the other schedulers. ber of clients. Client insertion, removal, and selection to run in GR3 are independent of the number of clients. The cost for GR3 is 3 times higher than before, with client selection 6 Related Work to run, insertion, and removal each taking approximately Round robin is one of the oldest, simplest and most widely 300 to 400 ns. For VTRR, although selecting a client to run used proportional share scheduling algorithms. Weighted is also independent of the number of clients, insertion over- round-robin (WRR) supports non-uniform client weights by head grows with the number of clients, resulting in much running all clients with the same frequency but adjusting the higher VTRR overhead for this benchmark. size of their time quanta in proportion to their respective To demonstrate GR3’s efficient proportional sharing weights. Deficit round-robin (DRR) [22] was developed to of resources on real applications, we briefly describe three support non-uniform service allocations in packet schedul- simple experiments running web server workloads using ing. These algorithms have low O(1) complexity but poor the same set of schedulers: GR3 and GR3MP, Linux short-term fairness, with service errors that can be on the 2.4 uniprocessor and multiprocessor schedulers, WFQ, and order of the largest client weight in the system. GR3 uses a VTRR. The web server workload emulates a number of novel variant of DRR for intragroup scheduling with O(1) virtual web servers running on a single system. Each complexity, but also provides O(1) service error by using virtual server runs the guitar music search engine used its grouping mechanism to limit the effective range of client at guitarnotes.com, a popular musician resource web site weights considered by the intragroup scheduler. with over 800,000 monthly users. The search engine is a Fair-share schedulers [12, 14, 15] provide propor- perl script executed from an Apache mod-perl module that tional sharing among users in a way compatible with a searches for guitar music by title or author and returns a UNIX-style time-sharing framework based on multi-level list of results. The web server workload configured each feedback with a set of priority queues. These schedulers server to pre-fork 100 processes, each running consecutive typically had low O(1) complexity, but were often ad-hoc searches simultaneously. and could not provide any proportional fairness guaran- We ran multiple virtual servers with each one hav- tees. Empirical measurements show that these approaches ing different weights for its processes. In the first experi- only provide reasonable proportional fairness over rela- ment, we used six virtual servers, with one server having tively large time intervals [12]. all its processes assigned weight 10 while all other servers Lottery scheduling [26] gives each client a number of had processes assigned weight 1. In the second experiment, tickets proportional to its weight, then randomly selects a we used five virtual servers and processes assigned to each ticket. Lottery scheduling takes O(log N) time and relies server had respective weights of 1, 2, 3, 4, and 5. In the on the law of large numbers for providing proportional fair- USENIX Association 2005 USENIX Annual Technical Conference 349 2 2 2 Weight 10 Weight 10 Weight 10 Weight 1 Weight 1 Weight 1 Weight 1 Weight 1 Weight 1 ) 1.5 Weight 1 ) 1.5 Weight 1 ) 1.5 Weight 1 s s s ( ( ( Weight 1 Weight 1 Weight 1 n n n o Weight 1 o Weight 1 o Weight 1 i i i t t t a a a c 1 c 1 c 1 o o o l l l l l l A A A U U U P P P C 0.5 C 0.5 C 0.5 0 0 0 0 50 100 150 200 250 300 0 50 100 150 200 250 300 0 50 100 150 200 250 300 Time (s) Time (s) Time (s) Figure 14: Linux uniprocessor Figure 15: WFQ uniprocessor Figure 16: VTRR uniprocessor 2 4 4 Weight 10 Weight 10 Weight 10 Weight 1 3.5 Weight 1 3.5 Weight 1 Weight 1 Weight 1 Weight 1 ) ) 1.5 Weight 1 3 Weight 1 ) 3 Weight 1 s s s ( ( ( Weight 1 Weight 1 Weight 1 n n n o o Weight 1 2.5 Weight 1 o 2.5 Weight 1 i i i t t t a a a c c 1 2 c 2 o o o l l l l l l A A A 1.5 1.5 U U U P P P C C 0.5 1 C 1 0.5 0.5 0 0 0 0 50 100 150 200 250 300 0 50 100 150 200 250 300 0 50 100 150 200 250 300 Time (s) Time (s) Time (s) Figure 17: GR3 uniprocessor Figure 18: Linux multiprocessor Figure 19: GR3MP multiprocessor ness. Thus, its allocation errors can be very large, typically skipping some of them at different frequencies without much worse than WRR for clients with smaller weights. having to reorder clients on each schedule. This is done Weighted Fair Queueing (WFQ) [11, 18], was first de- by combining round-robin scheduling with a virtual time veloped for network packet scheduling, and later applied to mechanism. GR3’s intergroup scheduler builds on VTRR uniprocessor scheduling [26]. It assigns each client a virtual but uses weight ratios instead of virtual times to provide bet- time and schedules the client with the earliest virtual time. ter fairness. Smoothed Round Robin (SRR) [9] uses a dif- Other fair queueing variants such as Virtual-clock [28], ferent mechanism for skipping clients using a Weight Ma- SFQ [13], SPFQ [24], and Time-shift FQ [10] have also trix and Weight Spread Sequence (WSS) to run clients by been proposed. These approaches all have O(log N) time simulating a binary counter. VTRR and SRR provide pro- complexity, where N is the number of clients, because the portional sharing with O(1) time complexity for selecting a clients must be ordered by virtual time. It has been shown client to run, though inserting and removing clients from the that WFQ guarantees that the service time error for any run queue incur higher overhead: O(log N) for VTRR and client never falls below −1 [18]. However, WFQ can allow O(k) for SRR , where k =logφmax and φmax is the max- a client to get far ahead of its ideal allocation and accumu- imum client weight allowed. However, unlike GR3, both late a large positive service time error of O(N), especially algorithms can suffer from large service time errors espe- with skewed weight distributions. cially for skewed weight distributions. For example, we can Several fair queueing approaches have been proposed show that the service error of SRR is worst-case O(kN). for reducing this O(N) service time error. A hierar- Grouping clients to reduce scheduling complexity has chical scheduling approach [26] reduces service time er- been used by [20], [8] and [23]. These fair queueing ap- ror to O(log N). Worst-Case Weighted Fair Queueing proaches group clients into buckets based on client virtual (WF2Q) [1] introduced eligible virtual times and can guar- timestamps. With the exception of [23], which uses expo- antee both a lower and upper bound on error of −1 and nential grouping, the fairness of these virtual time bin sort- +1, respectively for network packet scheduling. It has also ing schemes depends on the granularity of the buckets and been applied to uniprocessor scheduling as Eligible Vir- is adversely affected by skewed client weight distributions. tual Deadline First (EEVDF) [25]. These algorithms pro- On the other hand, GR3 groups based on client weights, vide stronger proportional fairness guarantees than other which are relatively static, and uses groups as schedulable approaches, but are more difficult to implement and still re- entities in a two-level scheduling hierarchy. quire at least O(log N) time. The grouping strategy used in GR3 was first intro- Motivated by the need for faster schedulers with good duced by two of the authors for uniprocessor scheduling [6] fairness guarantees, one of the authors developed Virtual- and generalized by three of the authors to network packet Time Round-Robin (VTRR) [17]. VTRR first introduced scheduling [4]. A similar grouping strategy was indepen- the simple idea of going round-robin through clients but dently developed in Stratified Round Robin (StRR) [19] for 350 2005 USENIX Annual Technical Conference USENIX Association network packet scheduling. StRR distributes all clients with been done in proportional share multiprocessor scheduling. −k −(k−1) weights between 2 and 2 into class Fk (F here WRR and fair-share multiprocessor schedulers have been not to be confused with our frontlog). StRR splits time into developed, but have the fairness problems inherent in those scheduling slots and then makes sure to assign all the clients approaches. The only multiprocessor fair queueing algo- in class Fk one slot every scheduling interval, using a credit rithm that has been proposed is Surplus Fair Scheduling and deficit scheme within a class. This is also similar to (SFS) [7]. SFS also adapts a uniprocessor algorithm, SFQ, GR3, with the key difference that a client can run for up to multiple processors using a centralized run queue. No to two consecutive time units, while in GR3, a client is theoretical fairness bounds are provided. If a selected client allowed to run only once every time its group is selected is already running on another processor, it is removed from regardless of its deficit. the run queue. This operation may introduce unfairness if StRR has weaker fairness guarantees and higher used in low overhead, round-robin variant algorithms. In 3 scheduling complexity than GR3. StRR assigns each client contrast, GR MP provides strong fairness bounds with weight as a fraction of the total processing capacity of the lower scheduling overhead. system. This results in weaker fairness guarantees when the SFS introduced the notion of feasible clients along sum of these fractions is not close to the limit of 1. For ex- with a O(P )-time weight readjustment algorithm, which ample, if we have N =2k +1clients, one of weight 0.5 requires however that the clients be sorted by their original 3 and the rest of weight 2−(k+2) (total weight = 0.75), StRR weight. By using its grouping strategy, GR MP performs will run the clients in such a way that after 2k+1 slots, the the same weight readjustment in O(P ) time without the −N need to order clients, thus avoiding SFS’s O(log N) over- error of the large client is 3 , such that this client will then run uniterruptedly for N tu to regain its due service. Client head per maintenance operation. The optimality of SFS’s weights could be scaled to reduce this error, but with addi- and our weight readjustment algorithms rests in preserva- tional O(N) complexity. StRR requires O(g) worst-case tion of ordering of clients by weight and of weight pro- time to determine the next class that should be selected, portions among feasible clients, and not in minimal overall where g is the number of groups. Hardware support can weight change, as [7] claims. hide this complexity assuming a small, predefined maxi- mum number of groups [19], but running an StRR processor 7 Conclusions and Future Work scheduler in software still requires O(g) complexity. We have designed, implemented, and evaluated Group Ra- GR3 also differs from StRR and other deficit round- tio Round-Robin scheduling in the Linux operating system. robin variants in its distribution of deficit. In DRR, SRR, We prove that GR3 is the first and only O(1) uniprocessor and StRR, the variation in the deficit of all the clients af- and multiprocessor scheduling algorithm that guarantees a fects the fairness in the system. To illustrate this, consider service error bound of less than O(N) compared to an ide- N +1clients, all having the same weight except the first alized processor sharing model, where N is the number of one, whose weight is N times larger. If the deficit of all runnable clients. In spite of its low complexity, GR3 offers the clients except the first one is close to 1, the error of the better fairness than the O(N) service error bounds of most first client will be about N = O(N). Therefore, the deficit 2 fair queuing algorithms that need O(log N) time for their mechanism as employed in round-robin schemes doesn’t al- operation. GR3 achieves these benefits due to its group- low for better than O(N) error. In contrast, GR3 ensures ing strategy, ratio-based intergroup scheduling, and highly that a group consumes all the work assigned to it, so that efficient intragroup round robin scheme with good fairness the deficit is a tool used only in distributing work within a bounds. GR3 introduces a novel frontlog mechanism and certain group, and not within the system. Thus, groups ef- and weight readjustment algorithm to schedule small-scale fectively isolate the impact of unfortunate distributions of multiprocessor systems while preserving its good bounds deficit in the scheduler. This allows for the error bounds in on fairness and time complexity. GR3 to depend only on the number of groups instead of the Our experiences with GR3 show that it is simple to much larger number of clients. implement and easy to integrate into existing commodity A rigorous analysis on network packet scheduling operating systems. We have measured the performance [27] suggests that O(N) delay bounds are unavoidable with of GR3 using both simulations and kernel measurements packet scheduling algorithms of less than O(log N) time of real system performance using a prototype Linux im- 3 2 complexity. GR ’s O(g ) error bound and O(1) time com- plementation. Our simulation results show that GR3 can plexity are consistent with this analysis, since delay and ser- provide more than two orders of magnitude better pro- vice error are not equivalent concepts. Thus, if adapted to portional fairness behavior than other popular proportional 3 packet scheduling, GR would worst-case incur O(N) de- share scheduling algorithms, including WRR, WFQ, SFQ, 2 lay while preserving an O(g ) service error. VTRR, and SRR. Our experimental results using our GR3 Previous work in proportional share scheduling has Linux implementation further demonstrate that GR3 pro- focused on scheduling a single resource and little work has vides accurate proportional fairness behavior on real ap- USENIX Association 2005 USENIX Annual Technical Conference 351 plications with much lower scheduling overhead than other [12] R. Essick. An Event-Based Fair Share Scheduler. In Pro- Linux schedulers, especially for larger workloads. ceedings of the Winter 1990 USENIX Conference, pages While small-scale multiprocessors are the most 147–162, Berkeley, CA, Jan. 1990. USENIX. widely available multiprocessor configurations today, the [13] P. Goyal, H. Vin, and H. Cheng. Start-Time Fair Queue- use of large-scale multiprocessor systems is growing given ing: A Scheduling Algorithm for Integrated Services Packet the benefits of server consolidation. Developing accurate, Switching Networks. IEEE/ACM Transactions on Network- low-overhead proportional share schedulers that scale effec- ing, pages 690–704, Oct. 1997. tively to manage these large-scale multiprocessor systems [14] G. Henry. The Fair Share Scheduler. AT&T Bell Laborato- ries Technical Journal, 63(8):1845–1857, Oct. 1984. remains an important area of future work. [15] J. Kay and P. Lauder. A Fair Share Scheduler. Commun. ACM, 31(1):44–55, 1988. Acknowledgements [16] L. Kleinrock. Computer Applications, volume II of Queue- Chris Vaill developed the scheduling simulator and helped ing Systems. John Wiley & Sons, New York, NY, 1976. implement the kernel instrumentation used for our experi- [17] J. Nieh, C. Vaill, and H. Zhong. Virtual-Time Round-Robin: ments. This work was supported in part by NSF grants EIA- An O(1) Proportional Share Scheduler. In Proceedings of 0071954 and DMI-9970063, an NSF CAREER Award, and the 2001 USENIX Annual Technical Conference, pages 245– 259, Berkeley, CA, June 2001. USENIX. an IBM SUR Award. [18] A. Parekh and R. Gallager. A Generalized Processor Shar- ing Approach to Flow Control in Integrated Services Net- References works: The Single-Node Case. IEEE/ACM Transactions on Networking, 1(3):344–357, June 1993. WF2Q [1] J. C. R. Bennett and H. Zhang. : Worst-Case Fair [19] S. Ramabhadran and J. Pasquale. Stratified Round Robin: a Weighted Fair Queueing. In Proceedings of IEEE INFO- Low Complexity Packet Scheduler with Bandwidth Fairness COM ’96 , pages 120–128, San Francisco, CA, Mar. 1996. and Bounded Delay. In Proceedings of ACM SIGCOMM [2] D. Bovet and M. Cesati. Understanding the Linux Kernel. ’03, pages 239–250, Karlsruhe, Germany, Aug. 2003. O’Reilly, Sebastopol, CA, second edition, 2002. [20] J. Rexford, A. G. Greenberg, and F. Bonomi. Hardware- [3] R. Bryant and B. Hartner. Java Technology, Threads, and Efficient Fair Queueing Architectures for High-Speed Net- Scheduling in Linux. IBM developerWorks Library Paper. works. In Proceedings of IEEE INFOCOM ’96, pages 638– IBM Linux Technology Center, Jan. 2000. 646, Mar. 1996. [4] B. Caprita, W. C. Chan, and J. Nieh. Group Round-Robin: [21] R. Russell. Hackbench: A New Multiqueue Sched- Improving the Fairness and Complexity of Packet Schedul- uler Benchmark. http://www.lkml.org/archive/ ing. Technical Report CUCS-018-03, Columbia University, 2001/12/11/19/index.html, Dec. 2001. Message to June 2003. Linux Kernel Mailing List. [5] W. C. Chan. Group Ratio Round-Robin: An O(1) Propor- [22] M. Shreedhar and G. Varghese. Efficient Fair Queueing tional Share Scheduler. Master’s thesis, Columbia Univer- using Deficit Round Robin. In Proceedings of ACM SIG- sity, June 2004. COMM ’95, pages 231–242, Sept. 1995. [6] W. C. Chan and J. Nieh. Group Ratio Round-Robin: [23] D. Stephens, J. Bennett, and H. Zhang. Implementing An O(1) Proportional Share Scheduler. Technical Report Scheduling Algorithms in High Speed Networks. IEEE CUCS-012-03, Columbia University, Apr. 2003. JSAC Special Issue on High Performance Switches/Routers, [7] A. Chandra, M. Adler, P. Goyal, and P. J. Shenoy. Surplus Sept. 1999. Fair Scheduling: A Proportional-Share CPU Scheduling Al- [24] D. Stiliadis and A. Varma. Efficient Fair Queueing Algo- gorithm for Symmetric Multiprocessors. In Proceedings of rithms for Packet-Switched Network. IEEE/ACM Transac- the 4th Symposium on Operating System Design & Imple- tions on Networking, pages 175–185, Apr. 1998. mentation, pages 45–58, San Diego, CA, Oct. 2000. [25] I. Stoica, H. Abdel-Wahab, K. Jeffay, S. Baruah, J. Gehrke, [8] S. Y. Cheung and C. S. Pencea. BSFQ: Bin-Sort Fair and G. Plaxton. A Proportional Share Resource Allocation Queueing. In Proceedings of IEEE INFOCOM ’02, pages Algorithm for Real-Time, Time-Shared Systems. In Pro- 1640–1649, New York, NY, June 2002. ceedings of the 17th IEEE Real-Time Systems Symposium, [9] G. Chuanxiong. SRR: An O(1) Time Complexity Packet pages 288 – 289, Dec. 1996. Scheduler for Flows in Multi-Service Packet Networks. In [26] C. Waldspurger. Lottery and Stride Scheduling: Flexi- Proceedings of ACM SIGCOMM ’01, pages 211–222, San ble Proportional-Share Resource Management. PhD thesis, Diego, CA, Aug. 2001. Dept. of EECS, MIT, Sept. 1995. [10] J. Cobb, M. Gouda, and A. El-Nahas. Time-Shift Schedul- [27] J. Xu and R. J. Lipton. On Fundamental Tradeoffs between ing – Fair Scheduling of Flows in High-Speed Networks. Delay Bounds and Computational Complexity in Packet IEEE/ACM Transactions on Networking, pages 274–285, Scheduling Algorithms. In Proceedings of ACM SIGCOMM June 1998. ’02, pages 279–292, Aug. 2002. [11] A. Demers, S. Keshav, and S. Shenker. Analysis and Sim- [28] L. Zhang. Virtualclock: a New Traffic Control Algorithm ulation of a Fair Queueing Algorithm. In Proceedings of for Packet-Switched Networks. ACM Transactions on Com- ACM SIGCOMM ’89, pages 1–12, Austin, TX, Sept. 1989. puter Systems, 9(2):101–124, May 1991. 352 2005 USENIX Annual Technical Conference USENIX Association