Adding Generic Process Containers to the Linux Kernel
Paul B. Menage∗ Google, Inc. [email protected]
Abstract Technically resource control and isolation are related; both prevent a process from having unrestricted access to even the standard abstraction of resources provided by the Unix kernel. For the purposes of this paper, we While Linux provides copious monitoring and control use the following defintions: options for individual processes, it has less support for applying the same operations efficiently to related Resource control is any mechanism which can do either groups of processes. This has led to multiple proposals or both of: for subtly different mechanisms for process aggregation for resource control and isolation. Even though some of these efforts could conceptually operate well together, • tracking how much of a resource is being com- merging each of them in their current states would lead sumed by a set of processes to duplication in core kernel data structures/routines. • imposing quantative limits on that consumption, ei- The Containers framework, based on the existing ther absolutely, or just in times of contention. cpusets mechanism, provides the generic process group- ing features required by the various different resource Typically resource control is visible to the processes be- controllers and other process-affecting subsystems. The ing controlled. result is to reduce the code (and kernel impact) required for such subsystems, and provide a common interface Namespace isolation1 is a mechanism which adds an with greater scope for co-operation. additional indirection or translation layer to the nam- ing/visibility of some unix resource space (such as pro- This paper looks at the challenges in meeting the needs cess ids, or network interfaces) for a specific set of pro- of all the stakeholders, which include low overhead, cesses. Typically the existence of isolation itself is in- feature richness, completeness and flexible groupings. visible to the processes being isolated. We demonstrate how to extend containers by writing resource control and monitoring components, we also Resource Control and Isolation are both subsets of the look at how to implement namespaces and cpusets on general model of a subsystem which can apply be- top of the framework. haviour differently depending on a process’ membership of some specific group. Other examples could include: 1 Introduction • basic job control. A job scheduling system needs Over the course of Linux history, there have been and to be able to keep track of which processes are continue to be multiple efforts to provide forms of mod- part of a given running “job,” in the presence of ified behaviour across sets of processes. These ef- fork() and exit() calls from processes in that forts have tended to fall into two main camps, resource job. This is the simplest example of the kind of control/monitoring, and namespace isolation (although some projects contain elements of both). 1We avoid using the alternative term virtualization in this paper to make clear the distinction between lightweight in-kernel virtual- *With additional contributions by Balbir Singh ization/isolation, and the much more heavyweight virtual machine and Srivatsa Vaddagiri, IBM Linux Technology Center, hypervisors such as Xen which run between the kernel and the hard- {balbir,vatsa}@in.ibm.com ware. 46 • Adding Generic Process Containers to the Linux Kernel
process tracking system proposed in this paper, as suitable final name for the concept is currently the sub- the kernel need do nothing more than maintain the ject of debate on various Linux mailing lists, due to its membership list of processes in the job. use by other development groups for a similar concept in userspace. Alternative suggestions have included parti- • tracking memory pressure for a group of processes, tion and process set. and being able to receive notifications when mem- ory pressure reaches some particular level (e.g. as measured by the scanning level reached in try_ to_free_pages()), or reaches the OOM stage. 2 Requirements
Different projects have proposed various basic mech- In this section we attempt to enumerate the properties re- anisms for implementing such process tracking. The quired of a generic container framework. Not all mecha- drawbacks of having multiple such mechanisms include: nisms that depend on containers will require all of these properties.
• different and mutually incompatible user-space and kernel APIs and feature sets. 2.1 Multiple Independent Subsystems • The merits of the underlying process grouping mechanisms and the merits of the actual resource Clients of the container framework will typically be control/isolation system become intertwined. resource accounting/control systems and isolation sys- • systems that could work together to provide syner- tems. In this paper we refer to the generic client as a sub- gistic control of different resources to the same sets system. The relationship between the container frame- of processes are unable to do so easily since they’re work and a subsystem is similar to that between the based on different frameworks. Linux VFS and a specific filesystem—the framework handles many of the common operations, and passes no- • kernel structures and code get bloated with addi- tifications/requests to the subsystem. tional framework pointers and hooks. Different users are likely to want to make use of differ- • writers of such systems have to duplicate large ent subsystems; therefore it should be possible to se- amounts of functionally-similar code. lectively enable different subsystems both at compile time and at runtime. The main function of the container The aim of the work described in this paper is to pro- framework is to allow a subsystem to associate some vide a generalized process grouping mechanism suitable kind of policy/stats (referred to as the subsystem state) for use as a base for current and future Linux process- with a group of processes, without the subsystem hav- control mechanisms such as resource controllers; the in- ing to worry in too much detail about how this is actually tention is that writers of such controllers need not be accomplished. concerned with the details of how processes are being tracked and partitioned, and can concentrate on the par- ticular resource controller at hand, and the resource ab- 2.2 Mobility straction presented to the user. (Specifically, this frame- work does not attempt to prescribe any particular re- source abstraction.) The requirements for a process- It should be possible for an appropriately-privileged tracking framework to meet the needs of existing and fu- user to move a process between containers. Some sub- ture process-management mechanisms are enumerated, systems may require that processes can only move into and a proposal is made that aims to satisfy these require- a container at the point when it is created (e.g. a virtual ments. server system where the process becomes the new init process for the container); therefore it should be possi- For the purposes of this paper, we refer to a tracked ble for mobility to be configurable on a per-subsystem group of processes as a container. Whether this is a basis. 2007 Linux Symposium, Volume Two • 47
2.3 Inescapability 2.5 Nesting
Once a process has been assigned to a container, it For some subsystems it is desirable for there to be mul- shouldn’t be possible for the process (or any of its chil- tiple nested levels of containers: dren) to move to a different container without action by a privileged (i.e. root) user, or by a user to whom that • The existing cpusets system inherently divides and capability has been delegated, e.g. via filesystem per- subdivides sets of memory nodes and CPUs be- missions. tween different groups of processes on the system. • Nesting allows some fraction of the resources 2.4 Extensible User Interface available to one set of processes to be delegated to a subset of those processes. Different subsystems will need to present different con- figuration and reporting interfaces to userspace: • Nested virtual servers may also find hierarchical support useful. • a memory resource controller might want to allow the user to specify guarantees and limits on the Some other subsystems will either be oblivious to the number of pages that processes in a container can concept of nesting, or will actively want to avoid it; use, and report how many pages are actually in use. therefore the container system should allow subsystems to selectively control whether they allow nesting. • a memory-pressure tracker might want to allow the user to specify the particular level of memory pres- 2.6 Multiple Partitions sure which should cause user notifications. • the cpusets system needs to allow users to specify The container framework will allow the user to partition various parameters such as the bitmasks of CPUs the set of processes. Consider a system that is config- and memory nodes to which processes in the con- ured with the cpusets and beancounters subsystems. At tainer have access. any one time, a process will be in one and exactly one cpuset (a cpuset A may also be a child of some other From the user’s point of view it is simplest if there is at cpuset B, in which case in a sense all processes in A are least some level of commonality between the configu- indirectly members of cpuset B as well, but a process is ration of different subsystems. Therefore the container only a direct member of one cpuset). Similarly a process framework should provide an interface that captures the is only a direct member of one beancounter. So cpusets common aspects of different subsystems but which still and beancounters are each a partition function on the set allows subsystems sufficient flexibility. Possible candi- of processes. dates include: Some initial work on this project produced a system that simply used the same partition function for all subsys- • A filesystem interface, where each subsystem can tems, i.e. for every cpuset created there would also be register files that the user can write (for configura- a beancounter created, and all processes moved into the tion) and/or read (for reporting) has the advantages new cpuset would also become part of the new bean- that it can be manipulated by many standard unix counter. However, very plausible scenarios were pre- tools and library routines, has built-in support for sented to demonstrate that this was too limiting. permission delegation, and can provide arbitrary input/output formats and behaviour. A generic container framework should support some way of allowing different partitions for different sub- • An API (possibly a new system call?) that allows systems. Since the requirements suggest that support- the user to read/write named properties would have ing hierarchical partitions is useful, even if not re- the advantages that it would tend to present a more quired/desired for all subsystems, we refer to these par- uniform interface (although possibly too restrictive titions, without loss of generality, as hierarchies. for some subsystems) and potentially have slightly better performance than a filesystem-based inter- For illustration, we consider the following examples of face. dividing processes on a system. 48 • Adding Generic Process Containers to the Linux Kernel
Cpusets Memory Network / / /
sys staff students sys staff students www NFS other
(a) University Server
Cpusets Memory Network Isolation / / / /
R1 R2 R1 R2 R1 R2 R1 R2
VS1 VS2 VS3 VS4 VS1 VS2 VS3 VS4 VS1 VS2 VS3 VS4
(b) Hosting Server
Figure 1: Example container divisions with independent (single-subsystem) hierarchies
A University Timesharing System2 Two possible approaches to supporting these examples A university server has various users—students, are given below; the illustrative figures represent a ker- professors, and system tasks. For CPU and mem- nel with cpusets, memory, network, and isolation sub- ory, it is desired to partition the system according systems. to the process owner’s user ID, whereas for net- work traffic, it is desired to partition the system ac- cording to the traffic content (e.g. WWW browser 2.6.1 Independent hierarchies related traffic across all users shares the same limit of 20%). Any single way of partitioning the sys- In the simplest approach, each hierarchy provides the tem will make this kind of resource planning hard, partition for exactly one subsystem. So to use e.g. potentially requiring creating a container for each cpusets and beancounters on the same system, you tuple of the cross-product of the various config- would need to create a cpuset hierarchy and a beancoun- ured resource subsystems.. Allowing the system ters hierarchy, and assign processes to containers sepa- to be partitioned in multiple ways, depending on rately for each subsystem. resource type, is therefore a desirable feature. This solution can be used to implement any of the A Virtual Server System other approaches presented below. The drawback is that A large commercial hosting server has many it imposes a good deal of extra management work to NUMA nodes. Blocks of nodes are sold to re- userspace in the (we believe likely) common case when sellers (R1, R2) to give guaranteed CPU/memory the partitioning is in fact the same across multiple or resources. The resellers then sell virtual servers even all subsystems. In particular, moving processes (VS1–VS4) to end users, potentially overcommit- between containers can have races—if userspace has to ting their resources but not affecting other resellers move a process as two separate actions on two different on the system. hierarchies, a child process forked during this operation The server owner would use cpusets to restrict each might end up in inconsistent containers in the two hier- reseller to a set of NUMA memory nodes/CPUs; archies. the reseller would then (via root-delegated capa- Figure 1 shows how containers on the university and bilities) use a server isolation subsystem and a re- hosting servers might be configured when each subsys- source control subsystem to create virtual servers tem is an independent hierarchy. The cpusets and mem- with various levels of resource guarantees/limits, ory subsystems have duplicate configurations for the within their own cpuset resources. university server (which doesn’t use the isolation sub- 2Example contributed by Srivatsa Vaddagiri. system), and the network, memory and isolation subsys- 2007 Linux Symposium, Volume Two • 49
Cpusets / Memory Network / /
sys staff students www NFS other
(a) University Server
Memory / Network / Cpusets / Isolation /
R1 R2
VS1 VS2 VS3 VS4
(b) Hosting Server
Figure 2: Example container divisions with multi-subsystem hierarchies tems have duplicate configurations for the hosting server handles or network sockets—with a container, for ac- since they’re the same for each virtual server. counting purposes or in order to affect the kernel’s be- haviour with respect to those objects.
Since such associations are likely to be subsystem- 2.6.2 Multi-subsystem hierarchies specific, the container framework needs primarily to be able to provide an efficient reference-counting mecha- An extension to the above approach allows multiple sub- nism, which will allow references to subsystem state ob- systems to be bound on to the same hierarchy; so e.g. jects to be made in such a way that they prevent the de- if you were using cpusets and beancounters, and you struction of the associated container until the reference wanted the cpuset and beancounter assignments to be has been released. isomorphic for all processes, you could bind cpusets and beancounters to the same hierarchy of containers, and 2.8 Low overhead only have to operate on a single hierarchy when creat- ing/destroying containers, or moving processes between The container framework should provide minimal addi- containers. tional runtime overhead over a system where individ- ual subsystems are hard-coded into the source at all ap- Figure 2 shows how the support for multiple subsystems propriate points (pointers in task_struct, additional per hierarchy can simplify the configuration for the two fork() and exit() handlers, etc). example servers. The university server can merge the configurations for the cpusets and memory subsystems (although not for the network subsystem, since that’s us- 3 Existing/Related Work ing an orthogonal division of processes). The hosting server can merge all four subsystems into a single hier- In this section we consider the existing mechanisms for archy. process tracking and control in Linux, looking at both those already included in the Linux source tree, and those proposed as bases for other efforts. 2.7 Non-process references to containers 3.1 Unix process grouping mechanisms Although the process is typically the object most uni- versally associated with a container, it should also be Linux has inherited several concepts from classical Unix possible to associate other objects—such as pages, file that can be used to provide some form of association be- 50 • Adding Generic Process Containers to the Linux Kernel tween different processes. These include, in order of in- used the independent hierarchies model described in creasing specificity: group id (gid), user id (uid), session Section 2.6.1; Resource Containers bound all schedulers id (sid) and process group (pgrp). (subsystems) into a single hierarchy.
Theoretically the gid and/or uid could be used as the PAGG [5] is part of the SGI Comprehensive System identification portion of the container framework, and Accounting project, adapted for Linux. It provides a a mechanism for configuring per-uid/gid state could be generic container mechanism that tracks process mem- added. However, these have the serious drawbacks that: bership and allows subsystems to be notified when pro- cesses are created or exit. A crucial difference between PAGG and the design presented in this paper is that • Job-control systems may well want to run multiple PAGG allows a free-form association between processes jobs as the same user/group on the same machine. and arbitrary containers. This results in more expensive • Virtual server systems will want to allow processes access to container subsystem state, and more expensive within a virtual server to have different uids/gids. fork()/exit() processing. Resource Groups [2] (originally named CKRM – Class- The sid and/or the pgrp might be suitable as a track- based Kernel Resource Management) and BeanCoun- ing base for containers, except for the fact that tradi- ters [1] are resource control frameworks for Linux. Both tional Unix semantics allow processes to change their provide multiple resource controllers and support addi- sid/pgrp (by becoming a session/group leader); remov- tional controllers. ResGroups’ support for additional ing this ability would be possible, but could resulting in controllers is more generic than the design proposed unexpected breakage in applications. (In particular, re- in this paper—we feel that the additional overheads quiring all processes in a virtual server to have the same that it introduces are unnecessary; BeanCounters is less sid or pgrp would probably be unmanageable). generic, in that additional controllers have to be hard- coded into the existing BeanCounters source. Both frameworks also enforce a particular resource model 3.2 Cpusets on their resource controllers, which may be inappro- priate for resource controllers with different require- The cpusets system is the only major container-like sys- ments from those envisaged—for example, implement- tem in the mainline Linux kernel. Cpusets presents a ing cpusets with its current interface (or an equivalent pseudo-filesystem API to userspace with semantics in- natural interface) on top of either the BeanCounters or cluding: ResGroups abstractions would not be possible.
• Creating a directory creates a new empty cpuset (an 3.4 Linux virtual server systems analog of a container). There have been a variety of virtual server systems de- • Control files in a cpuset directory allow you to con- veloped for Linux; early commercial systems included trol the set of memory nodes and/or CPUs that Ensim’s Virtual Private Server [6] and SWSoft’s Vir- tasks in that cpuset can use. tuozzo [7]; these both provided various resource con- • A special control file, tasks can be read to list the trollers along with namespace isolation, but no support set of processes in the cpuset; writing a pid to the for generic extension with user-provided subsystems. tasks file moves a process into that cpuset. More recent open-sourced virtualization systems have included VServer [8] and OpenVZ [9], a GPL’d subset 3.3 Linux/Unix container systems of the functionality of Virtuozzo.
The Eclipse [3] and Resource Containers [4] projects 3.5 NSProxy-based approaches both sought to add quality of service to Unix. Both supported hierarchical systems, allowing free migration Recent work on Linux virtual-server systems [10] of processes and threads between containers; Eclipse has involved providing multiple copies of the various 2007 Linux Symposium, Volume Two • 51 namespaces (such as for IPC, process ids, etc) within 4.2.1 container the kernel, and having the namespace choice be made on a per-process basis. The fact that different pro- The container structure represents a container ob- cesses can now have different IPC namespaces results ject as described in Section 1. It holds parent/child/ in the requirement that these namespaces be reference- sibling information, per-container state such as flags, fork() exit() counted on and released on . To and a set of subsystem state pointers, one for the state reduce the overhead of such reference counting, and for each subsystem configured in the kernel. It holds to reduce the number of per-process pointers required no resource-specific state. It currently3 holds no refer- struct for all these virtualizable namespaces, the ence to a list of tasks in the container; the overhead of nsproxy was introduced. This is a reference-counted maintaining such a list would be paid whenever tasks structure holding reference-counted pointers to (theoret- fork() or exit(), and the relevant information can ically) all the namespaces required for a task; therefore be reconstructed via a simple walk of the tasklist. at fork()/exit() time only the reference count on the nsproxy object must be adjusted. Since in the common case a large number of processes are expected 4.2.2 container_subsys to share the same set of namespaces, this results in a reduction in the space required for namespace pointers and in the time required for reference counting, at the The container_subsys structure represents a sin- cost of an additional indirection each time one of the gle resource controller or isolation component, e.g. a namespaces is accessed. memory controller or a CPU scheduler.
The most important fields in a container_subsys are the callbacks provided to the container framework; 4 Proposed Design these are called at the appropriate times to allow the sub- system to learn about or influence process events and container events in the hierarchy to which this subsys- This section presents a proposed design for a container tem is bound, and include: framework based on the requirements presented in Sec- tion 2 and the existing work surveyed above. A pro- totype implementation of this design is available at create is called when a new container is created the project website [11], and has been posted to the [email protected] mailing list and destroy is called when a container is destroyed other relevant lists. can_attach is called to determine whether the subsys- tem wants to allow a process to be moved into a 4.1 Overview given container
attach is called when a process moves from one con- The proposed container approach is an extension of tainer to another the process-tracking design used for cpusets. The cur- fork is called when a process forks a child rent design favours the multiple-hierarchy approach de- scribed in Section 2.6.2. exit is called when a process exits
populate is called to populate the contents of a con- 4.2 New structure types tainer directory with subsystem-specific control files
The container framework adds several new structures bind is called when a subsystem is moved between hi- to the kernel. Figure 3 gives an overview of the rela- erarchies. tionship between these new structures and the existing 3The possibility of maintaining such a task list just for those sub- task_struct and dentry. systems that really need it is being considered. 52 • Adding Generic Process Containers to the Linux Kernel
Apart from create and destroy, implemention of This function simply dereferences the given subsystem these callbacks is optional. pointer in the task’s css_group (see next Section).
Other fields in container_subsys handle house- keeping state, and allow a subsystem to find out to which 4.2.4 css_group hierarchy it is attached.
Subsystem registration is done at compile time— For the same reasons as described in Section 3.5, main- subsystems add an entry in the header file include/ taining large numbers of pointers (per-hierarchy or per- linux/container_subsys.h. This is used in con- subsystem) within the task_struct object would re- junction with pre-processor macros to statically allocate sult in space wastage and reference-counting overheads, an identifier for each subsystem, and to let the container particularly in the case when container systems com- system locate the various container_subsys ob- piled into the kernel weren’t actually used. jects. Therefore, this design includes the css_group (con- The compile-time registration of subsystems means that tainer subsystem group) object, which holds one it is not possible to build a container subsystem purely container_subsys_state pointer for each reg- as a module. Real-world subsystems are expected to re- istered subsystem. A reference-counted pointer field quire subsystem-specific hooks built into other locations (called containers) to a css_group is added to in the kernel anyway; if necessary, space could be left in task_struct, so the space overhead is one pointer per the relevant arrays for a compile-time configurable num- task, and the time overhead is one reference count op- ber of “extra” subsystems. eration per fork()/exit(). All tasks with the same set of container memberships across all hierarchies will share the same css_group. 4.2.3 container_subsys_state A subsystem can access the per-subsystem state for a task by looking at the slot in the task’s css_group A container_subsys_state represents the base indexed by its (statically defined) subsystem id. Thus type from which subsystem state objects are derived, the additional indirection is the only subsystem-state ac- and would typically be embedded as the first field in the cess overhead introduced by the css_group; there’s subsystem-specific state object. It holds housekeeping no overhead due to the generic nature of the container information that needs to be shared between the generic framework. The space/time tradeoff is similar to that container system and the subsystem. In the current de- associated with nsproxy. sign this state consists of: It has been proposed (by Srivatsa Vaddagiri and oth- container – a reference to the container object with ers) that the css_group should be merged with the which this state is associated. This is primarily nsproxy to form a single per-task object containing useful for subsystems which want to be able to ex- both namespace and container information. This would amine the tree of containers (e.g. a hierarchical re- be a relatively straightforward change, but the current source manager may propagate resource informa- design keeps these as separate objects until more expe- tion between subsystem state objects up or down rience has been gained with the system. the hierarchy of containers). 4.3 Code changes in the core kernel refcnt – the reference count of external non-process ob- jects on this subsystem state object, as described in Section 2.7. The container framework will refuse The bulk of the new code required for the container to destroy a container whose subsystems have non- framework is that implementing the container filesys- zero states, even if there are no processes left in the tem and the various tracking operations. These are container. driven entirely by the user-space API as described in Section 4.5.
To access its state for a given task, a subsystem can Changes in the generic kernel code are minimal and con- call task_subsys_state(task,
container / dentry / container_subsys_state
A1 /foo /bar B1 C1
A2 A3 B2 B3
CG1 CG2 css_group
T1 T2 T3 task_struct
Figure 3: Three subsystems (A, B, C) have been compiled into the kernel; a single hierarchy has been mounted with A and B bound to it. Container directories foo and bar have been created, associated with subsystem states A2/B2 and A3/B3 respectively. Tasks T1 and T2 are in container foo; task T3 is in container bar. Subsystem C is unbound so all container groups (and hence tasks) share subsystem state C1.
• A hook early in the fork() path to take an ad- include creating/destroying containers, moving pro- ditional reference count on the task’s css_group cesses between containers, and reading/writing sub- object. system control files. Performance-sensitive operations should not require this lock. • A hook late in the fork() path to invoke subsystem-specific fork callbacks, if any are re- When modifying the containers pointer in a task, quired. the container framework surrounds this operation with task_lock()/task_unlock(), and follows with • A hook in the exit() path to invoke any a synchronize_rcu() operation before releasing subsystem-specfic exit callbacks, if any, and to the container_mutex. release the reference count on the task’s css_ group object (which will also free the css_group Therefore, in order for a subsystem to be sure that it is if this releases the last reference.) accessing a valid containers pointer, it suffices for at least one of the following three conditions to be true of the current task: 4.4 Locking Model • it holds container_mutex The current container framework locking model re- volves around a global mutex (container_mutex), • it holds its own alloc_lock. each task’s alloc_lock (accessed via task_ • it is in an RCU critical section. lock() and task_unlock()) and RCU critical sec- tions. The final condition, that of being in an RCU critical sec- The container_mutex is used to synchronize per- tion, doesn’t prevent the current task being concurrently container operations driven by the userspace API. These moved to a different container in some hierarchy—it 54 • Adding Generic Process Containers to the Linux Kernel simply tells you that the current task was in the spec- is released and all subsystems are available for reuse. ified set of containers at some point in the very recent If the hierarchy still has child containers, the hierarchy past, and that any of the subsystem state pointers in that (and superblock) remain active even though not actively css_group object won’t be candidates for deletion attached to a mounted filesystem.4 until after the end of the current RCU critical section. Creating a directory in a container mount creates a new When an object (such as a file or a page) is accounted child container; containers may be arbitrarily nested, to the value that has been read as the current subsystem within any restrictions imposed by the subsystems state for a task, it may actually end up being accounted bound to the hierarchy being manipulated. to a container that the task has just been moved from; provided that a reference to the charged subsystem state Each container directory has a special control file, is stored somewhere in the object, so that at release tasks. Reading from this file returns a list of pro- time the correct subsystem state can be credited, this is cesses in the container; writing a pid to this file moves typically sufficient. A subsystem that wants to reliably the given process into the container (subject to success- migrate resources between containers when the process ful can_attach() callbacks on the subsystems bound that allocated those resources moves (e.g. cpusets, when to the hierarchy). the memory_migrate mode is enabled) may need ad- ditional subsystem-specific locking, or else use one of Other control files in the container directory may be cre- the two other locking methods listed above, in order to ated by subsystems bound to that hierarchy; reads and ensure that resources are always accounted to the correct writes on these files are passed through to the relevant containers. subsystems for processing.
The subsystem state pointers in a given css_group are Removing a directory from a container mount destroys immutable once the object has been created; therefore the container represented by the directory. If tasks re- as long as you have a pointer to a valid css_group, it main within the container, or if any subsystem has a non- is safe to access the subsystem fields without additional zero reference count on that container, the rmdir() locking (beyond that mandated by subsystem-specific operation will fail with EBUSY. (The existence of sub- rules). system control files within a directory does not keep it busy; these are cleared up automatically.) 4.5 Userspace API The file /proc/PID/container lists, for each ac- The userspace API is very similar to that of the existing tive hierarchy, the path from the root container to the cpusets system. container of which process PID is a member.5
A new hierarchy is created by mounting an instance of The file /proc/containers gives information the container pseudo-filesystem. Options passed to about the current set of hierarchies and subsystems in the mount() system call indicate which of the avail- use. This is primarily useful for debugging. able subsystems should be bound to the new hierarchy. Since each subsystem can only be bound to a single hier- archy at once, this will fail with EBUSY if the subsystem 4.6 Overhead is already bound to a different hierarchy.
Initially, all processes are in the root container of this The container code is optimized for fast access by sub- new hierarchy (independently of whatever containers systems to the state associated with a given task, and a fork() exit() they might be members of in other hierarchies). fast / path.
If a mount request is made for a set of subsystems that Depending on the synchronization requirements of a exactly match an existing active hierarchy, the same su- particular subsystem, the first of these can be as simple as: perblock is reused. 4They remain visible via a /proc reporting interface. At the time when a container hierarchy is unmounted, if 5An alternative proposal is for this to be a directory holding con- the hierarchy had no child containers then the hierarchy tainer path files, one for each subsystem. 2007 Linux Symposium, Volume Two • 55 struct task_struct *p = current; • declaration of a subsystem in include/linux/ rcu_read_lock() container_subsys.h struct state *st = task_subsys_state(p, • Kconfig/Makefile additions my_subsys_id); ... • the code in kernel/cpuacct.c to implement
The cpuacct subsystem is a simple demonstration of For backwards compatibility the existing cpuset a useful container subsystem. It allows the user to eas- filesystem type remains; any attempt to mount it gets ily read the total amount of CPU time (in milliseconds) redirected to a mount of the container filesystem, used by processes in a given container, along with an with a subsystem option of cpuset. estimate of the recent CPU load for that container.
The 250-line patch consists of: 5.3 ResGroups
• callback hooks added to the account_*_ ResGroups (formerly CKRM [2]) is a hierarchical re- time() functions in kernel/sched.c to in- source control framework that specifies the resource form the subsystem when a particular process is limits for each child in terms of a fraction of the re- being charged for a tick. sources available to its parent. Additionally resources 56 • Adding Generic Process Containers to the Linux Kernel may be borrowed from a parent if a child has reached its when performing work in the kernel on behalf of another resource limits. process. This aspect of BeanCounters is maintained un- changed in the containers port—if a process has an over- The abstraction provided by the generic containers ride context set then that is used for accounting, else the framework is low-level, with free-form control files. context reached via the css_group pointer is used. As an example of how to provide multiple subsys- tems sharing a common higher-level resource abstrac- tion, ResGroups is implemented as a container subsys- 5.5 NSProxy / Container integration tem library by stripping out the group management as- pects of the code and adding container subsystem call- backs. A resource controller can use the ResGroups A final patch in the series integrates the NSProxy system abstraction simply by declaring a container subsystem, with the container system by making it possible to track with the subsystem’s private field pointing to a Res- processes that are sharing a given namespace, and (po- Groups res_controller structure with the relevant tentially in the future) create custom namespace sets for resource-related callbacks. processes. This patch is a (somewhat speculative) ex- ample of a subsystem that provides an isolation system The ResGroups library registers control files for that rather than a resource control system. subsystem, and translates the free-form read/write in- terface into a structured and typed set of callbacks. This The namespace creation paths (via fork() or has two advantages: unshare()) are hooked to call container_ clone(). This is a function provided by the container • it reduces the amount of parsing code required in a framework that creates a new sub-container of a task’s subsystem container (in the hierarchy to which a specified subsys- tem is bound) and moves the task into the new con- • it allows multiple subsystems with similar (re- tainer. The ns container subsystem also makes use of source or other) abstractions to easily present the the can_attach container callback to prevent arbi- same interface to userspace, simplifying userspace trary manipulation of the process/container mappings. code. When a task changes its namespaces via either of these One of the ResGroups resource controllers two methods, it ends up in a fresh container; all of its (numtasks, for tracking and limiting the number children (that share the same set of namespaces) are in of tasks created in a container) is included in the patch. the same container. Thus the generic container frame- work provides a simple way to export to userspace the 5.4 BeanCounters sets of namespaces in use, their hierarchical relation- ships, and the processes using each namespace set. BeanCounters [1] is a single-level resource accounting framework that aims to account and control consump- tion of kernel resources used by groups of processes. Its 6 Conclusion and Future Work resource model allows the user to specify soft and hard limits on resource usage, and tracks high and low water- In this paper we have examined some of the existing marks of resource usage, along with resource allocations work on process partitioning/tracking in Linux and other that failed due to limits being hit. operating systems, and enumerated the requirements for The port to use the generic containers framework con- a generic framework for such tracking; a proposed de- verts between the raw read/write interface and the struc- sign was presented, along with examples of its use. tured get/store in a similar way to the ResGroups port, although presenting a different abstraction. As of early May 2007, several other groups have proposed resource controllers based on the containers Additionally, BeanCounters allows the accounting con- framework; it is hoped that the containers patches can text (known as a beancounter) to be overridden in par- be trialled in Andrew Morton’s -mm tree, with the aim ticular situations, such as when in an interrupt handler or of reaching the mainline kernel tree in time for 2.6.23. 2007 Linux Symposium, Volume Two • 57
7 Acknowledgements
Thanks go to Paul Jackson, Eric Biederman, Srivatsa Vaddagiri, Balbir Singh, Serge Hallyn, Sam Villain, Pavel Emelianov, Ethan Solomita, and Andrew Mor- ton for their feedback and suggestions that have helped guide the development of these patches and this paper.
References
[1] Kir Kolyshkin, Resource management: the Beancounters, Proceedings of the Linux Symposium, June 2007
[2] Class-based Kernel Resource Management, http://ckrm.sourceforge.net
[3] J. Blanquer, J. Bruno, E. Gabber, M. Mcshea, B. Ozden, and A. Silberschatz, Resource management for QoS in Eclipse/BSD. In Proceedings of FreeBSD’99 Conf., Berkeley, CA, USA, Oct. 1999
[4] Gaurav Banga, Peter Druschel, and Jeffrey C. Mogul. Resource containers: A new facility for resource management in server systems. In Proceedings of the Third Symposium on Operating Systems Design and Implementation (OSDI ’99), pages 45–58, 1999
[5] SGI. Linux Process Aggregates (PAGG), http://oss.sgi.com/projects/pagg/
[6] Ensim Virtual Private Server, http://www.ensim.com
[7] SWSoft Virtuozzo, http: //www.swsoft.com/en/virtuozzo/
[8] Linux VServer, http://www.linux-vserver.org/
[9] OpenVZ, http://www.openvz.org/
[10] Eric W. Biederman, Multiple Instances of the Global Linux Namespaces, Proceedings of the Linux Symposium, July 2006.
[11] Linux Generic Process Containers, http:// code.google.com/p/linuxcontainers 58 • Adding Generic Process Containers to the Linux Kernel Proceedings of the Linux Symposium
Volume Two
June 27th–30th, 2007 Ottawa, Ontario Canada Conference Organizers
Andrew J. Hutton, Steamballoon, Inc., Linux Symposium, Thin Lines Mountaineering C. Craig Ross, Linux Symposium
Review Committee
Andrew J. Hutton, Steamballoon, Inc., Linux Symposium, Thin Lines Mountaineering Dirk Hohndel, Intel Martin Bligh, Google Gerrit Huizenga, IBM Dave Jones, Red Hat, Inc. C. Craig Ross, Linux Symposium
Proceedings Formatting Team
John W. Lockhart, Red Hat, Inc. Gurhan Ozen, Red Hat, Inc. John Feeney, Red Hat, Inc. Len DiMaggio, Red Hat, Inc. John Poelstra, Red Hat, Inc.