Protection Strategies for Direct Access to Virtualized I/O Devices Paul Willmann, Scott Rixner, and Alan L. Cox Rice University {willmann, rixner, alc}@rice.edu Abstract chine virtualization [1, 2, 5, 8, 9, 10, 14, 19, 22]. How- Commodity virtual machine monitors forbid direct ac- ever, virtualization can impose performance penalties up cess to I/O devices by untrusted guest operating systems to a factor of 5 on I/O-intensive workloads [16, 19]. in order to provide protection and sharing. However, These penalties stem from the overhead of providing both I/O memory management units (IOMMUs) and re- shared and protected access to I/O devices by untrusted cently proposed software-based methods can be used to guest operating systems. Commodity virtualization ar- reduce the overhead of I/O virtualization by providing chitectures provide protection in software by forbidding untrusted guest operating systems with safe, direct ac- direct access to I/O hardware by untrusted guests— cess to I/O devices. This paper explores the performance instead, all I/O accesses, both commands and data, are and safety tradeoffs of strategies for using these mecha- routed through a single software entity that provides both nisms. protection and sharing. The protection strategies presented in this paper pro- Preferably, guest operating systems would be able to vide equivalent inter-guest protection among operating directly access I/O devices without the need for the data system instances. However, they provide varying levels to traverse an intermediate software layer within the vir- of intra-guest protection from driver software and incur tual machine monitor [17, 23]. However, if a guest can varying levels of overhead. A simple direct-map strat- directly access an I/O device, then it can potentially di- egy incurs the least overhead, providing native-level per- rect the device to access memory that it is not entitled formance but offering no enhanced protection from mis- to via direct memory access (DMA). Therefore, the vir- behaving device drivers within the guest operating sys- tual machine monitor must be able to ensure that guest tem. Additional protection against guest drivers can be operating systems do not access each other’s memory achieved by limiting IOMMU page-table mappings to indirectly through the shared I/O devices in the system. memory buffers that are actually used in I/O transfers. Both I/O memory management units (IOMMUs) [7] and Furthermore, the cost incurred by this limitation can be recently proposed software-based methods [23] can pro- minimized by aggressively reusing these mappings. Sur- vide DMA memory protection for the virtual machine prisingly, a software-only strategy that does not use an monitor. They do so by preventing guest operating sys- IOMMU at all performs competitively, and sometimes tems from directing I/O devices to access memory that it better than, hardware-based strategies while maintaining is not entitled to access, while still allowing the guest to strict inter-guest isolation. directly access the device. These DMA protection mechanisms can also be used 1 Introduction by a guest operating system to enhance safety and iso- lation among its own drivers and processes. The state- In many organizations, the economics of supporting a of-the-art single-use IOMMU-based protection strategy growing number of Internet-based application services employed by many existing non-virtualized operating has created a demand for server consolidation. Conse- systems provides just such a level of enhanced safety. quently, there has been a resurgence of interest in ma- This strategy creates a mapping for each I/O transaction and then destroys that mapping as soon as the transac- This work was supported in part by the National Science Foundation tion completes. In conjunction with IOMMU hardware, under Grant Nos. CCF-0546140 and CNS-0720878 and by gifts from Advanced Micro Devices and Hewlett-Packard. Paul Willmann was the operating system’s protection strategy can exert fine- supported in part by SFE Technology, Inc. grained control over what portions of memory may be USENIX Association USENIX ’08: 2008 USENIX Annual Technical Conference 15 used in an I/O transaction at any given time. tion strategy performs competitively with several of the This paper explores and experimentally compares five better-performing IOMMU-based strategies while main- different strategies for ensuring memory isolation of un- taining strong inter-guest protection guarantees and en- trusted, virtualized operating systems that each have di- abling intra-guest protection capabilities. rect access to I/O hardware. These strategies all ensure The next section provides background on how I/O de- isolation among OS instances and the virtual machine vices access main memory and the possible memory pro- monitor, but they vary in the level of protection within a tection violations that can occur when doing so. Sec- particular guest they can support and the overheads they tions 3 and 4 discuss the four IOMMU-based protection incur. Though upcoming commodity platforms will fea- strategies and the one software-based protection strat- ture IOMMUs for efficient I/O virtualization, there ex- egy. Section 5 then describes the protection proper- ists no comprehensive study about how to best leverage ties afforded by the five strategies. Section 6 discusses IOMMUs, what the tradeoffs are among efficiency and IOMMU hardware architectures. Section 7 describes the protection for different possible strategies, and what the experimental methodology and Section 8 evaluates the comparative costs associated with the various protection protection strategies. Section 9 then describes related strategies are. work and Section 10 concludes the paper. The first IOMMU-based strategy is based on state-of- the-art strategies and uses single-use I/O memory map- 2 Background pings that are created before each I/O operation and im- mediately destroyed after each I/O operation. The sec- Modern server I/O devices, including disk and network ond IOMMU-based strategy is introduced in this pa- controllers, utilize direct memory access (DMA) to move per and uses shared I/O memory mappings that can be data between the host’s main memory and the device’s reused by multiple, concurrent I/O operations. The third on-board buffers. The device uses DMA to access mem- IOMMU-based strategy is also introduced in this paper ory independently of the host CPU, so such accesses and uses persistent I/O memory mappings that can be must be controlled and protected. To initiate a DMA reused. The fourth IOMMU-based strategy uses a static operation, the device driver within the operating system direct map of the guest’s physical memory to isolate creates DMA descriptors that refer to regions of mem- that guest’s I/O transactions. Finally, the software-based ory. Each DMA descriptor typically includes an address, strategy is based on previous work [23] and uses VMM- a length, and a few device-specific flags. In commodity managed DMA descriptors that can only be used for one x86 systems, devices lack support for virtual-to-physical I/O operation. address translation, so DMA descriptors always contain The comparison of these five strategies yields sev- physical addresses for main memory. Once created, the eral insights. First, all five strategies provide equiva- device driver passes the descriptors to the device, which lent inter-guest protection among OS instances. How- will later use the descriptors to transfer data to or from ever, the strategies support differing levels of protection the indicated memory regions autonomously. When the within a particular guest (intra-guest protection). For ex- requested I/O operations have been completed, the de- ample, the direct-map strategy incurs almost no perfor- vice raises an interrupt to notify the device driver. mance overhead but supports no intra-guest protection. For example, to transmit a network packet, the net- Conversely, the single-use strategy provides the maxi- work interface’s device driver might create two DMA de- mum possible intra-guest protection, but it imposes the scriptors. The first descriptor might point to the packet largest performance penalty. Second, there is signifi- headers and the second descriptor might point to the cant opportunity to reuse IOMMU mappings, which can packet payload. Once created, the device driver would reduce protection overheads. Multiple concurrent I/O then notify the network interface that there are new DMA operations are able to share the same mappings often descriptors available. The precise mechanism of that no- enough that there is a noticeable decrease in the over- tification depends on the particular network interface, head of providing protection. Sharing mappings only but typically involves a programmed I/O operation to among concurrent I/O operations provides the same level the device telling it the location of the new descriptors. of intra-guest protection as the single-use strategy but The network interface would then retrieve the descriptors with less overhead. Relaxing this intra-guest protection from main memory using DMA—if they were not writ- guarantee by allowing mappings to persist so that they ten to the device directly by programmed I/O. The net- can be reused in future I/O operations can significantly work interface would then retrieve the two memory re- decrease this overhead, allowing the guest to achieve gions that compose the network packet and transmit them performance levels very close to that of the direct-map over the network. Finally, the network interface would strategy while still maintaining some amount of intra- interrupt the host to indicate that the packet has been guest protection. Finally, the software-based protec- transmitted. In practice, notifications from the device 16 USENIX ’08: 2008 USENIX Annual Technical Conference USENIX Association driver and interrupts from the network interface would to protect against a misbehaving device, as will be dis- likely be aggregated to cover multiple packets for effi- cussed in Section 5).