CPU Inheritance Scheduling Bryan Ford Sai Susarla Department of Computer Science University of Utah Salt Lake City, UT 84112 [email protected] http://www.cs.utah.edu/projects/flux/ Abstract such as support for ®xed-priority threads[17], or sev- eral ªscheduling classesº to which threads with different Traditional processor scheduling mechanisms in operat- purposes can be assigned (e.g., real-time, interactive, or ing systems are fairly rigid,often supportingonly one ®xed background)[25]. However, even these variants are gener- scheduling policy, or, at most, a few ªscheduling classesº ally hard-coded intothe system implementationand cannot whose implementations are closely tied together in the OS easily be adapted to the needs of individual applications. kernel. This paper presents CPU inheritance scheduling, a novel processor scheduling framework in which arbitrary In this paper we develop a novel processor schedul- threads can act as schedulers for other threads. Widely dif- ing framework based on a generalized notion of priority ferent scheduling policies can be implemented under the inheritance. In this framework, known as CPU inheri- framework, and many different policies can coexist in a tance scheduling, arbitrary threads can act as schedulers for single system, providing much greater scheduling ¯exibil- other threads by temporarily donatingtheirCPUtime tose- ity. Modular, hierarchical control can be provided over the lected threads while waiting on events of interest such as processor utilization of arbitrary administrative domains, clock/timer interrupts. The receiving threads can further such as processes, jobs, users, and groups, and the CPU donate their CPU time to other threads, and so on, form- resources consumed can be accounted for and attributed ing a logical hierarchy of schedulers as illustrated in Fig- accurately. Applications, as well as the OS, can imple- ure1. A scheduler threadcan benoti®edwhen thethread to ment customized local scheduling policies; the framework which it donated its CPU time no longer needs it (e.g., be- ensures that all the different policies work together logi- cause the target thread has blocked), so that the CPU can be cally and predictably. As a side effect, the framework also reassigned to another target. The basic thread dispatching cleanly addresses priority inversion by providing a gener- mechanism necessary to implement this framework does alized formof priorityinheritancethat automatically works not haveany notionof threadpriority,CPUusage, or clocks within and among diverse scheduling policies. CPU inher- and timers; all of these functions, when needed, are imple- itance scheduling extends naturallyto multiprocessors, and mented by threads acting as schedulers. supports processor management techniques such as proces- Under this framework, arbitrary scheduling policies can sor af®nity[29] and scheduler activations[3]. We show that be implemented by ordinary threads cooperating with each this ¯exibilitycan be provided with acceptable overhead in other through well-de®ned interfaces that may cross pro- typical environments, depending on factors such as context tection boundaries. For example, a ®xed-priority multi- switch speed and frequency. processor scheduling policy can be implemented by main- taining, among a group of scheduler threads (one for each available CPU), a prioritized queue of ªclientº threads to 1 Introduction be scheduled; each scheduler thread successively picks a thread to run and donates its CPU time to the selected tar- Traditional operating systems control the sharing of the get thread while waiting for an interesting event such as machine's CPU resources among threads using a ®xed quantum expiration (e.g., a clock interrupt). See Figure 2 scheduling scheme, typically based on priorities. Some- for an illustration. If the selected thread blocks, its sched- times a few variants on the basic policy are provided, uler thread is noti®ed and the CPU is reassigned. On the This research was supported in part by the Defense Advanced Re- other hand, if a different event causes the scheduler thread search Projects Agency,monitoredby the Departmentof the Army, under to wake up, therunningthread is preempted and theCPU is contract number DABT63±94±C±0058. The opinions and conclusions containedin thisdocumentarethoseof theauthorsandshould not be inter- given back to the scheduler immediately. Other scheduling preted as representing of®cial views or policies of the U.S. Government. policies, such as timesharing[23], ®xed-priority[12, 17], Root Scheduler Scheduler (real-time fixed-priority Real-time threads RR/FIFO scheduler) Scheduler CPU 0 threads Ready queues Timesharing class CPU 1 (lottery scheduler) scheduling Web browser requests Mike CPU donation (BSD scheduler) Java applets port Running thread Ready Jay threads (lottery scheduler) Jay's threads App 1 Waiting thread App 2 Background class (gang scheduler) Figure 2: Example ®xed-priority scheduler Background jobs The framework naturally extends to multiprocessors. Processor af®nity scheduling is supported. Scheduler activations can be implemented easily. Based on a prototype implementation in a user-level Figure 1: Example scheduling hierarchy. The dark circles represent threads package, we demonstrate that this framework be- threads acting as schedulers, while the light circles represent ªordinaryº threads. haves identically to traditional multi-class schedulers in equivalent con®gurations, and that it provides modular re- source control, load insulation, and priority inversion pro- rate monotonic[22], fair share[9, 16, 19], and lottery/stride tection automatically across multiple scheduling policies. scheduling[30, 31, 32], can be implemented in the same The additional scheduling overhead caused by this frame- way. work is negligible in the test environment, and we show This scheduling framework has the following features: that it should also be acceptable in many kernel environ- ments as well despite greater context switch costs. It supports multiple arbitrary scheduling policies on the same or different processors. The rest of this paper is organized as follows. Sec- tion 2 describes motivation and related work, Section 3 Since scheduler threads may run either in the OS ker- presents the CPU inheritance scheduling algorithm in de- nel or in user mode, applications can easily extend or tail, and Section 4 shows how traditional scheduling algo- replace the scheduling policies built into the OS. rithms can be implemented on this framework. Section 5 demonstrates the behavior and performance properties of It provides hierarchical control over the processor re- our framework through experimental results, and Section 6 source usage of different logical or administrative do- concludes with a short re¯ective summary. mains in a system, such as users, groups, individual processes, and threads within a process. 2 Motivation CPU usage accounting can be provided to various de- grees of accuracy depending on the resources one is Traditional operating systems divide a machine's CPU willing to invest. resources among threads using a ®xed scheduling scheme, typically based on priorities. However, the requirements Priority inversion is addressed naturally in the pres- ence of resource contention, without the need for ex- imposed on an operating system's scheduler often vary plicit priority inheritance/ceiling protocols. from application to application. For example, for interac- tive applications, response time is usually the most criti- CPU use is attributedproperly even in the presence of cal factorÐi.e., how quickly the program responds to the priority inheritance. user's commands. For batch jobs, throughput is of primary importance but latency is a minorissue. Hard real-time ap- is possible to obtain the correct behavior for some appli- plications require meeting application-speci®c deadlines, cations. However, this approach has serious, well-known while for soft real-time applications, missing a deadline limitations; in many cases, entirely different non-priority- is unfortunate but not catastrophic. There is no single based scheduling policies are needed[18]. scheduling scheme that works well for all applications. Even in normal interactive and batch-mode computing, Over the years, the importance of providing a variety traditional priority-based scheduling algorithms are show- of scheduling policies on a single machine has waxed and ing their age. For example, these algorithms do not pro- waned, following hardware and application trends. In the vide a clean way to encapsulate a set of processes/threads early years of computing, use of the entire machine was as a single unit to isolate and control their processor usage limited to a single user thread; this evolved into multipro- relative to the rest of the system. This shortcoming opens grammed machines in which a single scheduling policy the system to various denial-of-service attacks, the most managed batch jobs effectively. The advent of timesharing well-known being the creation of a large number of threads on machines still used for batch jobs caused a need for two which overwhelm processor resources and crowd out other scheduling policies. As timesharing gradually gave way to activity. These vulnerabilities generally don't cause seri- single-user workstations and PCs, a single scheduling pol- ous problems for machines only used by one person, or icy was again adequate for a while. when the users of the system fall into one administrative Supportingmultiplescheduling