THE ADVANCED COMPUTING SYSTEMS ASSOCIATION The following paper was originally published in the Proceedings of the USENIX Annual Technical Conference Monterey, California, USA, June 6-11, 1999 The UVM Virtual Memory System _ _ Charles D. Cranor and Gurudatta M. Parulkar Washington University © 1999 by The USENIX Association All Rights Reserved Rights to individual papers remain with the author or the author's employer. Permission is granted for noncommercial reproduction of the work for educational or research purposes. This copyright notice must be included in the reproduced paper. USENIX acknowledges all trademarks herein. For more information about the USENIX Association: Phone: 1 510 528 8649 FAX: 1 510 548 5738 Email: [email protected] WWW: http://www.usenix.org The UVM Virtual Memory System Charles D. Cranor Gurudatta M. Parulkar Department of Computer Science Washington University St. Louis, MO 63130 g fchuck,guru @arl.wustl.edu Abstract virtual memory system that was written for Carnegie We introduce UVM, a new virtual memory system for Mellon University’s Mach operating system [18]. The the BSD kernel that has an improved design that in- BSD VM system features a clean separation of machine- creases system performance over the old Mach-based dependent functions, support for mmap, and fine-grain 4.4BSD VM system. In this paper we present an data structure locking suitable for multiprocessor sys- overview of both UVM and the BSD VM system. We tems. focus our discussion on the design decisions made when creating UVM and contrast the UVM design with the 1.1 Why Replace BSD VM? less efficient BSD VM design. Topics covered in- clude mapping, memory object management, anony- The BSD VM system has four main drawbacks that con- mous memory and copy-on-write mechanisms, and tributed to our decision to replace it: complex data struc- pager design. We also present an overview of virtual tures, poor performance, no virtual memory based data memory based data movement mechanisms that have movement mechanisms, and poor documentation. been introduced in BSD by UVM. We believe that the The data structures and functions used by BSD to lessons we learned from designing and implementing manage memory are complex and thus difficult to main- UVM can be applied to other kernels and large software tain. This is especially true of the structures used to systems. Implemented in the NetBSD operating system, represent copy-on-write mappings of memory objects. UVM will completely replace BSD VM in NetBSD 1.4. As memory objects are copied using the copy-on-write mechanism [2] (e.g., during a fork) they are linked to- gether in lists called object chains. If left unchecked, 1 Introduction use of the copy-on-write mechanism can cause object chains to grow quite long. Long object chains are a In this paper we introduce UVM1, a new virtual memory problem for two reasons. First, long object chains slow system that replaces the 4.4BSD virtual memory system memory search times by increasing the number of ob- in BSD-based operating systems. UVM is the third gen- jects that need to be checked to locate a requested page. eration BSD virtual memory system that improves the Second, long object chains are likely to contain inacces- performance and reduces the complexity of the BSD ker- sible redundant copies of the same page of data, thus nel. UVM’s improved performance is particularly useful wasting memory. If the BSD VM system allows too to applications that make heavy use of VM features such many pages of memory to be wasted this way the sys- as memory-mapped files and copy-on-write memory. tem’s swap area will eventually become filled with re- Versions of BSD prior to 4.4BSD used the old BSD- dundant data and the system will deadlock. This condi- VAX virtual memory system that was tightly bound to tion is known as a swap memory leak deadlock. To avoid the VAX architecture and lacked support for memory- problems associated with long object chains, the BSD mapped files (mmap) and fine-grain data structure lock- VM system attempts to reduce their length by using a ing for multiprocessors. To address these issues, the old complex collapse operation. To successfully collapse an VAX-based VM system was replaced with a new VM object chain, the VM system must search for an object system for 4.4BSD [12]. The 4.4BSD virtual mem- that contains redundant data and is no longer needed in ory system (BSD VM) is a modified version of the the chain. If a redundant object is found, then it is either 1Note that “UVM” is a name, not an acronym bypassed or discarded. Note that even a successfully col- lapsed object chain can still contain inaccessible redun- combined with I/O and IPC system changes currently dant pages. The collapse operation can only repair swap under development, these mechanisms can reduce the memory leaks after they occur, it cannot prevent them kernel’s data copying overhead. from happening. UVM’s source code is freely available under the stan- BSD VM exhibits poor performance due to several dard BSD license. UVM was merged into the NetBSD factors. First, the overhead of object chain management operating system’s master source repository in early slows down common operations such as page faults and 1998 and has proven stable on the architectures sup- memory mapping. Second, I/O operations in BSD VM ported by NetBSD including the Alpha, I386, M68K, are performed one page at a time rather than in more effi- MIPS, Sparc, and VAX. UVM will appear as the official cient multi-page clusters, thus slowing paging response virtual memory system of NetBSD in NetBSD release time. Third, BSD VM’s integration into the BSD ker- 1.4. A port of UVM to OpenBSD is also in progress. nel has not been optimized. For example, unreferenced In this paper we present the design and implementa- memory-mapped file objects are cached at the I/O sys- tion of the UVM virtual memory system, highlighting its tem (vnode) layer and redundantly at the virtual memory improvements over the BSD VM design. In Section 2 layer as well. Finally, several virtual memory operations we present an overview of BSD VM and UVM. In Sec- are unnecessarily performed multiple times at different tions 3, 4, 5, 6, and 7 we present the design of UVM’s layers of the BSD kernel. map, object, anonymous memory, pager, and data move- Another drawback of the BSD VM system is its lack ment facilities, respectively. In Section 8 we present the of virtual memory based data movement mechanisms. overall performance of UVM. Section 9 contains related While BSD VM does support the copy-on-write mecha- work, and Section 10 contains our concluding remarks nism, it is not possible in BSD VM for the virtual mem- and directions for future research. ory system to safely share memory it controls with other kernel subsystems such as I/O and IPC without perform- ing a costly data copy. It is also not possible for pro- 2VMOverview cesses to easily exchange, copy, or share chunks of their Both BSD VM and UVM can be divided into two layers: virtual address space between themselves. a small machine-dependent layer, and a larger machine- Finally, the BSD VM system is poorly documented. independent layer. The machine-dependent layer used While some parts of the BSD kernel such as the network- by both BSD and UVM is called the pmap layer. The ing and IPC system are well documented [20], the BSD pmap layer handles the low level details of program- VM system lacks such detailed documentation and is ming a processor’s MMU. This task consists of adding, poorly commented. This has made it difficult for devel- removing, modifying, and querying the mappings of a opers of free operating systems projects such as NetBSD virtual address or of a page of physical memory. The [17] to understand and maintain the 4.4BSD VM code. pmap layer has no knowledge of higher-level operat- ing system abstractions such as files. Each architecture 1.2 The UVM Approach supported by the BSD kernel must have its own pmap module. Note that UVM was designed to use the same One way to address the shortcomings of the BSD virtual machine-dependent layer that BSD VM and Mach use. memory system is to try and evolve the data structures This allows pmap modules from those systems to be and functions BSD inherited from Mach into a more effi- reused with UVM, thus reducing the overhead of port- cient VM system. This technique has been successfully ing a UVM-based kernel to a new architecture. applied by the FreeBSD project to the BSD VM sys- The machine-independent code is shared by all BSD- tem. However, the improved VM system in FreeBSD supported processors and contains functions that per- still suffers from the object chaining model it inherited form the high-level operations of the VM system. Such from BSD VM. An alternative approach is to reimple- functions include managing a process’ file mappings, ment the virtual memory system, reusing the positive requesting data from backing store, paging out mem- aspects of the BSD VM design, replacing the parts of ory when it becomes scarce, managing the allocation the design that do not work well, and adding new fea- of physical memory, and managing copy-on-write mem- tures on top of the resulting VM system. This is the ory. Figure 1 shows the five main abstractions that cor- approach we used for UVM.