SCHED DEADLINE: a real-time CPU scheduler for Linux Luca Abeni [email protected]
TuToR 2017 Luca Abeni – 1 / 33 Scheduling Real-Time Tasks
Introduction Real-Time Scheduling in cj cj+1 Linux fj dj fj+1 dj+1 Setting the Scheduling rj rj+1 Policy The Constant Bandwidth Server Consider a set of N real-time tasks SCHED DEADLINE Using Γ= {τ0,...τN−1} SCHED DEADLINE Scheduled on M CPUs Real-Time theory → lot of scheduling algorithms... ...But which ones are available on a commonly used OS? POSIX: fixed priorities Can be used to do RM, DM, etc... Multiple processors: DkC, etc... Linux also provides SCHED DEADLINE: resource
TuToR 2017 reservations + EDF Luca Abeni – 2 / 33 Definitions
Introduction Real-Time Scheduling in Real-time task τ : sequence of jobs Ji =(ri,ci,di) Linux Setting the Scheduling Policy Finishing time fi The Constant Bandwidth Server Goal: fi ≤ di SCHED DEADLINE Using ∀Ji, or control the amount of missed deadlines SCHED DEADLINE Schedule on multiple CPUS: partitioned or global Schedule in a general-purpose OS Open System (with online admission control) Presence of non real-time tasks (do not starve them!)
TuToR 2017 Luca Abeni – 3 / 33 Using Fixed Priorities with POSIX
Introduction Real-Time Scheduling in SCHED FIFO and SCHED RR use fixed priorities Linux Setting the Scheduling Policy They can be used for real-time tasks, to The Constant Bandwidth Server implement RM and DM SCHED DEADLINE Using Real-time tasks have priority over non real-time SCHED DEADLINE (SCHED OTHER) tasks The difference between the two policies is visible when more tasks have the same priority In real-time applications, try to avoid multiple tasks with the same priority
TuToR 2017 Luca Abeni – 4 / 33 Setting the Scheduling Policy
Introduction Real-Time Scheduling in int sched_get_priority_max(int policy); Linux int sched_get_priority_min(int policy); Setting the Scheduling Policy The Constant int sched_setscheduler(pid_t pid, int policy, Bandwidth Server const struct sched_param *param); SCHED DEADLINE int sched_setparam(pid_t pid, Using const struct sched_param *param); SCHED DEADLINE If pid == 0, then the parameters of the running task are changed The only meaningful field of struct sched param is sched priority
TuToR 2017 Luca Abeni – 5 / 33 Issues with Real-Time Priorities
Introduction Real-Time Scheduling in Open Systems → real-time tasks can dynamically Linux Setting the Scheduling arrive (in an unpredictable way) Policy The Constant Bandwidth Server Need to re-arrange priorities to respect RM / DM SCHED DEADLINE Using / ... SCHED DEADLINE Interactions with non real-time tasks? Scheduled in background respect to real-time tasks Suboptimal utilization?
TuToR 2017 Luca Abeni – 6 / 33 Real-Time Priorities vs “Regular Tasks”
Introduction Real-Time Scheduling in In general, “regular” (SCHED OTHER) tasks are Linux Setting the Scheduling scheduled in background respect to real-time ones Policy The Constant Real-time tasks can starve other applications Bandwidth Server SCHED DEADLINE Example: the following task scheduled at high Using SCHED DEADLINE priority can make a CPU / core unusable void bad bad task () { while (1); }
TuToR 2017 Luca Abeni – 7 / 33 Starvation of Non Real-Time Tasks
Introduction Real-Time Scheduling in Starvation of non real-time tasks Linux Setting the Scheduling Policy Real-time computation have to be limited (use The Constant Bandwidth Server real-time priorities only when really needed!) SCHED DEADLINE Using On sane systems, running applications with real-time SCHED DEADLINE priorities requires root privileges (or part of them!) Not usable by everyone
TuToR 2017 Luca Abeni – 8 / 33 Real-Time Throttling
Introduction Real-Time Scheduling in A “bad” high-priority task can make a CPU / core Linux Setting the Scheduling unusable... Policy The Constant ...Linux provides the real-time throttling mechanism Bandwidth Server SCHED DEADLINE to address this problem Using SCHED DEADLINE How does real-time throttling interfere with real-time guarantees? Given a priority assignment, a taskset is guaranteed all the deadlines if no throttling mechanism is used... ...But, what happens in case of throttling? Very useful idea, but something more “theoretically founded” might be needed...
TuToR 2017 Luca Abeni – 9 / 33 Can We Do Better?
Introduction Real-Time Scheduling in Avoid starvation issues by using resource Linux Setting the Scheduling reservations Policy The Constant Use EDF instead of fixed priorities Bandwidth Server SCHED DEADLINE Using CPU Reservations + EDF = SCHED DEADLINE!!! SCHED DEADLINE So, how to implement EDF (or something similar) in Linux? Issue: the kernel is (was?) not aware of tasks deadlines... ...But deadlines are needed in order to schedule the tasks!
TuToR 2017 Luca Abeni – 10 / 33 EDF in the Linux Kernel
Introduction Real-Time Scheduling in EDF assigns dynamic priorities based on absolute Linux Setting the Scheduling deadlines Policy The Constant So, a more advanced API for the scheduler is Bandwidth Server SCHED DEADLINE needed... Using SCHED DEADLINE Assign at least a relative deadline D to the task... We will see that we need a runtime and a period too
Moreover, dj = rj + D...
...However, how can the scheduler know rj? The scheduler is not aware of jobs...
TuToR 2017 Luca Abeni – 11 / 33 Tasks, and Jobs...
Introduction Real-Time Scheduling in EDF → need to know when a job starts / finishes Linux Setting the Scheduling Policy Applications must be modified to signal the The Constant Bandwidth Server beginning / end of a job (some kind of SCHED DEADLINE Using startjob() / endjob() system call)... SCHED DEADLINE ...Or the scheduler can assume that a new job arrives each time a task wakes up! Or, some other algorithm can be used to assign dynamic scheduling deadlines to tasks
TuToR 2017 Luca Abeni – 12 / 33 ...And Scheduling Deadlines!
Introduction Real-Time Scheduling in The scheduler does EDF on scheduling deadlines Linux Setting the Scheduling s Policy Scheduling deadline d : assigned by the kernel The Constant Bandwidth Server to task τ SCHED DEADLINE Using But the task cares about its absolute deadlines SCHED DEADLINE If the scheduling deadline ds matches the absolute deadline dj of a job, then the scheduler can respect dj!!!
TuToR 2017 Luca Abeni – 13 / 33 CBS: The Basic Idea
Introduction Real-Time Scheduling in Constant Bandwidth Server (CBS): algorithm used to Linux s Setting the Scheduling assign a dynamic scheduling deadline d to a task τ Policy The Constant Based on the Resource Reservation paradigm Bandwidth Server SCHED DEADLINE Using Task τ is periodically reserved a maximum SCHED DEADLINE runtime Q every reservation period P Temporal isolation between tasks The worst case finishing time for a task does not depend on the other tasks running in the system... ...Because the task is guaranteed to receive its reserved time
TuToR 2017 Luca Abeni – 14 / 33 CBS: Some More Details
Introduction Real-Time Scheduling in Solves the issue with “bad tasks” trying to consume Linux Setting the Scheduling too much execution time Policy The Constant Based on CPU reservations (Q,P ) Bandwidth Server SCHED DEADLINE Using If τ tries to execute for more than Q every P , the SCHED DEADLINE algorithm decreases its priority, or throttles it τ consumes the same amount of CPU time consumed by a periodic task with WCET Q and period P Q/P : fraction of CPU time reserved to τ
TuToR 2017 Luca Abeni – 15 / 33 CBS: Admission Control
Introduction Real-Time Scheduling in The CBS is based on EDF Linux Setting the Scheduling s Policy Assigns scheduling deadlines d The Constant s Bandwidth Server EDF on d ⇒ good CPU utilization (optimal on SCHED DEADLINE Using UP!) SCHED DEADLINE If EDF is used (based on the scheduling deadlines assigned by the CBS), then τi is guaranteed to receive Qi time units every Pi if Pj Qj/Pj ≤ 1!!! Only on uni-processor / partitioned systems... M CPUs / cores with global scheduling: if Pj Qj/Pj ≤ M each task is guaranteed to receive Qi every Pi with a maximum delay
TuToR 2017 Luca Abeni – 16 / 33 CBS vs Other Reservation Algorithms
Introduction Real-Time Scheduling in The CBS allows to serve non periodic tasks Linux Setting the Scheduling Policy Some reservation-based schedulers have The Constant Bandwidth Server problems with aperiodic job arrivals - due to the SCHED DEADLINE Using (in)famous “deferrable server problem” SCHED DEADLINE The CBS explicitly supports aperiodic arrivals (see the rule for assigning deadlines when a task wakes up) Allows to support “self-suspending” tasks No need to strictly respect the Liu&Layland task model No need to explicitly signal job arrivals / terminations
TuToR 2017 Luca Abeni – 17 / 33 CBS: the Algorithm
Introduction s Real-Time Scheduling in Each task τ is associated a scheduling deadline d Linux Setting the Scheduling and a current runtime q Policy The Constant Bandwidth Server Both initialized to 0 when the task is created SCHED DEADLINE Using SCHED DEADLINE When a job arrives: If the previous job is not finished yet, queue the activation Otherwise, check if the current scheduling deadline can be used (ds >t and q/(ds − t) < Q/P ) s If not, d = t + P , q = Q
When τ executes for a time δ, q = q − δ When q , τ cannot be scheduled (until time ds) TuToR 2017 =0 Luca Abeni – 18 / 33 At time ds, ds = ds + P and q = q + Q SCHED DEADLINE
Introduction Real-Time Scheduling in New SCHED DEADLINE scheduling policy Linux Setting the Scheduling Policy Foreground respect to all of the other policies The Constant Bandwidth Server SCHED DEADLINE Using SCHED DEADLINE
TuToR 2017 Luca Abeni – 19 / 33 SCHED DEADLINE and CBS
Introduction Real-Time Scheduling in Uses the CBS to assign scheduling deadline to Linux Setting the Scheduling SCHED DEADLINE tasks Policy The Constant Bandwidth Server Assign a (maximum) runtime Q and a SCHED DEADLINE Using (reservation) period P to SCHED DEADLINE SCHED DEADLINE tasks Additional parameter: relative deadline D The “check if the current scheduling deadline can be used” rule is used at task wake-up Then uses EDF to schedule them Both global EDF and partitioned EDF are possible Configurable through the cpuset mechanism
TuToR 2017 Luca Abeni – 20 / 33 SCHED DEADLINE Design: Flexibility
Introduction Real-Time Scheduling in Supports both global and partitioned scheduling Linux Setting the Scheduling Policy For partitioned scheduling, use cpusets The Constant Bandwidth Server Flexible utilization-based admission control SCHED DEADLINE Using Qj L SCHED DEADLINE ≤ U Pj Pj U L configurable, ranging from 0 to M
/proc/sys/kernel/sched rt {runtime, period} us Can leave CPU time for non-deadline tasks Bounded tardiness; hard respect of deadlines for partitioned scheduling Even supports arbitrary affinities! But admission control must be disabled... TuToR 2017 Luca Abeni – 21 / 33 Setting the Scheduling Policy
Introduction Real-Time Scheduling in No sched setsched() ← new syscalls (and data Linux Setting the Scheduling structures added to be extensible) Policy The Constant Bandwidth Server Maybe even too extensible! SCHED DEADLINE Using SCHED DEADLINE int sched_setattr(pid_t pid, const struct sched_attr *attr, unsigned int flags); int sched_getattr(pid_t pid, struct sched_attr *attr, unsigned int size, unsigned int flags);
struct sched_attr { __u32 size;
__u32 sched_policy; __u64 sched_flags; ... __u64 sched_runtime; __u64 sched_deadline; __u64 sched_period; };
TuToR 2017 Luca Abeni – 22 / 33 Using sched setattr()
Introduction Real-Time Scheduling in pid: as for sched setscheduler() Linux Setting the Scheduling flags: currently unused (for future extensions!) Policy The Constant attr: scheduling parameters for the task Bandwidth Server SCHED DEADLINE Using size: must be set to sizeof(struct SCHED DEADLINE sched attr) sched policy: set to SCHED DEADLINE! sched runtime: Q sched deadline: D sched period: P sched flags: will see later (set to 0 for now)
TuToR 2017 Luca Abeni – 23 / 33 libdl
Introduction Real-Time Scheduling in So, can we use SCHED DEADLINE in our user Linux Setting the Scheduling programs? Policy The Constant sched setattr() & friends are in the kernel since Bandwidth Server SCHED DEADLINE 3.14... Using SCHED DEADLINE But the user-space side of things is still missing in many Linux distributions No support in glibc, no definition of struct sched attr, etc... Solution: small user-space library providing the sched *attr() system calls and related data structures libdl, released by Juri Lelli under GPL
TuToR 2017 Luca Abeni – 24 / 33 Example
Introduction Real-Time Scheduling in Linux #include "libdl/dl_syscalls.h" Setting the Scheduling Policy ... The Constant Bandwidth Server struct sched_attr attr; SCHED DEADLINE Using attr.size = sizeof(struct attr); SCHED DEADLINE attr.sched_policy = SCHED_DEADLINE; attr.sched_runtime = 30000000; attr.sched_period = 100000000; attr.sched_deadline = 100000000; ... res = sched_setattr(0, &attr, 0); if (res < 0) perror("sched_setattr()"); ... TuToR 2017 Luca Abeni – 25 / 33 Admission Control
Introduction Real-Time Scheduling in sched setattr() might fail if admission control Linux Setting the Scheduling fails Policy The Constant L Bandwidth Server Sum of reserved utilizations exceed the limit U SCHED DEADLINE Using Affinity of the task is different from its root domain SCHED DEADLINE Why the check on the affinity? Q j ≤ M guarantees bounded tardiness for Pj Pj global scheduling! Arbitrary affinities need a different analysis... So, how to use arbitrary affinities? Disable admission control! echo -1 > /proc/sys/kernel/sched rt runtime us
TuToR 2017 Luca Abeni – 26 / 33 Partitioned Scheduling
Introduction Real-Time Scheduling in cpuset: mechanism for assigning a set of CPUs to Linux Setting the Scheduling a set of tasks Policy The Constant Bandwidth Server Exclusive cpuset: CPUs not shared SCHED DEADLINE Using Tasks migrate inside scheduling domains SCHED DEADLINE ⇐ cpusets can bee used to create isolated domains Only one CPU ⇒ partitioned scheduling
mount -t tmpfs cgroup_root /sys/fs/cgroup mkdir /sys/fs/cgroup/cpuset mount -t cgroup -o cpuset cpuset /sys/fs/cgroup/cpuset
mkdir /sys/fs/cgroup/cpuset/Set1 echo 3 > /sys/fs/cgroup/cpuset/Set1/cpuset.cpus echo 0 > /sys/fs/cgroup/cpuset/Set1/cpuset.mems echo 0 > cpuset.sched_load_balance echo 1 > /sys/fs/cgroup/cpuset/Set1/cpuset.cpu_exclusive echo $PID > /sys/fs/cgroup/cpuset/Set1/tasks
TuToR 2017 Luca Abeni – 27 / 33 Warning!
Introduction Real-Time Scheduling in sched setaffinity() on SCHED DEADLINE Linux Setting the Scheduling tasks can fail Policy The Constant Bandwidth Server Again, disable admission control to use SCHED DEADLINE Using something different from global scheduling SCHED DEADLINE SCHED DEADLINE tasks cannot fork Which scheduling parameters would be inherited? Remember: runtimes and periods are in nanoseconds (not microseconds)
TuToR 2017 Luca Abeni – 28 / 33 Using SCHED DEADLINE
Introduction Real-Time Scheduling in ...How to dimension the scheduling parameters? Linux Setting the Scheduling Policy (Maximum) runtime Q: rt runtime (in nsec) The Constant Bandwidth Server (Reservation) period P : rt period (in nsec) SCHED DEADLINE Using SCHED DEADLINE also provides a (relative) SCHED DEADLINE deadline D Obviously, it must be Q j ≤ M X P j j
The kernel can do this admission control Better to use a limit U L smaller than M (so that
TuToR 2017 other tasks are not starved!) Luca Abeni – 29 / 33 Assigning Runtime and Period
Introduction Real-Time Scheduling in Temporal isolation Linux Setting the Scheduling Policy Each task can be guaranteed independently from The Constant Bandwidth Server the others SCHED DEADLINE Using Hard Schedulability property SCHED DEADLINE If Q ≥ WCET and P ≤ MIT (maximum runtime larger than WCET, and server period smaller than task period)... ...Then the scheduling deadlines are equal to the jobs’ deadlines!!! All deadlines are guaranteed to be respected (on UP / partitioned systems), or an upper bound for the tardiness is provided (if global scheduling is used)!!! TuToR 2017 Luca Abeni – 30 / 33 So, can be used to serve hard What About Soft Real-Time?
Introduction Real-Time Scheduling in What happens if Q
TuToR 2017 Luca Abeni – 31 / 33 Changing Parameters...
Introduction Real-Time Scheduling in Tasks’ parameters (execution and inter-arrival times) Linux Setting the Scheduling can change during the tasks lifetime... So, how to Policy The Constant dimension Q and P ? Bandwidth Server SCHED DEADLINE Short-term variations: CPU reclaiming mechanisms Using SCHED DEADLINE (GRUB, ...) If a job does not consume all of the runtime Q, try to reuse the residual Long-term variations: adaptive reservations Generally “slower”, can be implemented by a user-space daemon s Monitor the difference between d and dj s If d − dj increases, Q needs to be increased
TuToR 2017 Luca Abeni – 32 / 33 If s , canbedecreased CPU Reclaiming!
Introduction Real-Time Scheduling in As mentioned, CPU reclaiming can be used to better Linux Setting the Scheduling tolerate short-term variations in the execution Policy The Constant times... Bandwidth Server SCHED DEADLINE ...And a CPU reclaiming mechanism has just been Using SCHED DEADLINE added to SCHED DEADLINE! Available since Linux 4.13 M-GRUB: multi-processor GRUB: per-runqueue reclaiming of unused CPU time Ah... This is what the sched flags field is for! Set SCHED FLAG RECLAIM (2)
TuToR 2017 Luca Abeni – 33 / 33