Making I/O Virtualization Easy with Device Files
Total Page:16
File Type:pdf, Size:1020Kb
Making I/O Virtualization Easy with Device Files Technical Report 2013-04-13, Rice University Ardalan Amiri Sani?, Sreekumar Nairy, Lin Zhong?, Quinn Jacobsonz ?Rice University, yDynavisor, Inc., zVibrado Technologies Abstract different versions of the same OS. As a result, we tar- get our solutions for scenarios like having multiple vir- Personal computers have diverse and fast-evolving I/O tual machines in the same personal computer or reusing devices, making their I/O virtualization different from legacy code. that of servers and data centers. In this paper, we present our recent endeavors in simplifying I/O virtu- While good solutions exist for CPU and memory vir- alization for personal computers. Our key insight is tualization [23,33], virtualizing I/O devices of personal that many operating systems, including Unix-like ones, computers has proven to be much harder due to their abstract I/O devices as device files. There is a small diversity in function and implementation. To support and stable set of operations on device files, therefore, our targeted scenarios (above), the I/O virtualization I/O virtualization at the device file boundary requires solution must (i) require low development effort to sup- a one-time effort to support various I/O devices. port various I/O devices; (ii) allow for sharing the I/O We present devirtualization, our design of I/O virtu- device between the host and the guests; (iii) support alization at the device file boundary and its implemen- legacy devices that are not specialized for virtualization; tation for Linux/x86 systems. We are able to virtualize and (iv) be portable to support virtualization across dif- various GPUs, input devices, cameras, and audio de- ferent versions of the same OS. The solution should also vices with fewer than 4900 LoC, of which only about provide adequate performance for personal computers. 300 are specific to I/O device classes. Our measure- Unfortunately, available solutions do not provide one or ments show that devirtualized devices achieve interac- more of these properties. tive performance indistinguishable from native ones by In this paper, we study a novel boundary, device files, human users, even when running 3D HD games. for I/O virtualization that meets all the aforementioned properties for personal computers. Modern OSes, such 1 Introduction as Unix-like ones employ device files to abstract I/O devices [1]. To virtualize an I/O device, our solution The value of virtualization is increasingly recognized creates a virtual device file in the guest OS for the cor- for personal computers1 [5, 15{17, 19, 30, 32]. As per- responding device file in the host. Threads of guest sonal computers are used for diverse purposes, virtual- processes issue file operations to this virtual device file ization allows a user to have multiple virtual machines as if it were the real device file. A thin indirection layer, (or guests) inside the same computer, each for a dedi- called Common Virtual Driver (CVD), forwards such arXiv:1304.3771v1 [cs.OS] 13 Apr 2013 cated purpose: one for work, one for personal use, and file operations to the host to be executed by the unmod- one for sharing with others [36]. Also, as hardware and ified host device driver. software of personal computers evolve rapidly, virtu- Our use of device files as the boundary for I/O vir- alization allows the legacy code to be reused in new tualization is motivated by four properties: (i) Low de- systems. velopment effort: device files are common to many im- We are particularly interested in whole system virtu- portant classes of I/O devices in personal computers, alization, which allows multiple guest operating systems including GPUs, input devices, camera, and audio de- to reside in the same computer and provides strong iso- vices. Moreover, the device file boundary is narrow due lation between them. In this paper, we assume a hosted to the small set of file operations. For example, Linux hypervisor, i.e., a hypervisor running inside a host OS, has about 30 file operations, and only about 10 of them and assume the guests and the host use the same OS or are used by most I/O devices. Finally, since device files 1By personal computer, we refer to desktops and mobile are at a higher layer than device drivers, virtualization computers of diverse form factors including laptops, smart- at this boundary allows for reuse of the device drivers in phones, and tablets. the host. (ii) Sharing: virtualization at the device file boundary readily supports sharing the device between Process Thread User Space the host and the guests. If multiple host applications can use the same device file in the host OS, then guest applications can use the same device file as well. (iii) System Call/Signal Interface Kernel Legacy support: device files are used by existing devices Device File in personal computers; therefore, virtualization at this (/dev/dri/card0) boundary can support these devices. (iv) Portability: Virtual Filesystem Switch the device file boundary has been quite stable across different versions of mature OSes such as Linux. Device File Operations Interface We present our design of the CVD and its imple- Device Driver mentation for Linux/x86 computers, called devirtual- ization, which realizes the theoretical benefits analyzed above and achieves performance adequate for personal I/O Device computing. We have addressed a fundamental challenge that the guest and the host reside in different virtual- Figure 1: The simplified I/O stack in Linux ization domains, creating a barrier for forwarding the file operations from the guest to the host. Our solution to this challenge contributes two novel techniques: a Unfortunately, devirtualization is not universal and virtual memory technique, called hybrid address space, cannot virtualize all devices, such as network and block that enables efficient cross-domain memory operations, devices. This is because applications interface to these and the dual thread technique that efficiently leverages devices is sockets and file systems, and not device files. hypercalls to forward the operations and improve con- Fortunately, good solutions already exist for virtualiz- currency in cross-domain operations. ing these devices [37], as they have been critical for data Devirtualization currently supports four important centers and basic use of VMs in personal computers. classes of I/O devices for personal computers using the Devirtualization introduces the device file interface same CVD implementation with fewer than 4900 LoC, between the guest and the host, which may be abused of which about 300 are specific to each class: GPU, in- by malicious guest applications. We are currently em- put devices, such as mouse and keyboard, camera, and ploying techniques to guarantee isolation between the audio devices, such as speaker. We note that GPU has system core, e.g., the host and the hypervisor, and the not been amenable to virtualization due to its functional guests. §9 elaborates more on this issue. and implementation complexity. Yet, devirtualization easily virtualizes GPU of various makes in laptop and 2 Background desktop computers with full functionality and adequate Devirtualization targets virtualizing I/O devices for whole performance for multiple guests. system virtualization. It currently supports hosted hy- We report a comprehensive evaluation of devirtual- pervisors, where the hypervisors runs inside a host OS, ization. Our evaluation shows that devirtualization re- such as VMware Workstation. The principle and de- quires low development effort to support various I/O de- sign of devirtualization, however, apply to bare-metal vices, easily shares the device between the host and the hypervisors equally well. guests, supports legacy devices, and is portable across different versions of Linux. For interactive devices, such 2.1 I/O Stack and Devices Files as input devices, camera, and speaker, devirtualiza- Devirtualization virtualizes I/O devices by virtualizing tion achieves performance indistinguishable from that device files. Figure 1 shows a simplified I/O stack in of native by human user. For GPU, devirtualization Linux. A process thread issues a file operation by call- achieves close to or even higher than 60 frames per sec- ing the right system calls to operate on the device file; ond (the display refresh rate) on average for 3D HD these system calls are handled by the Virtual Filesys- games (1152×864), even under stress test by standard tem Switch (VFS), which invokes the file operations im- test engines. plemented by the device driver, e.g., read and memory We designed devirtualization for Unix-like OSes, such map. The kernel exports device files to user space through as Linux distributions, Mac OS X, Android, and iOS, a special filesystem, e.g., devfs (/dev) in Linux. Im- because they constitute a large number of the installa- portant file operations for I/O devices include read, tions on modern personal computers, especially smart- write, poll, notification, memory map, page fault, phones and tablets. However, we believe that with and I/O control. proper engineering, devirtualization can also be useful Threads are the execution units in the OS, issuing for other OSes, such as Windows, that also abstract the file operations. All threads of a process share the several I/O devices with device files. process address space. Therefore, we use \thread" when 2 discussing the execution of file operations, but use \pro- Process Thread Guest cess" when discussing memory operations. User Space To correctly access an I/O device, an application may Guest need to know the exact model or functional capabilities System Call/Signal Interface Kernel of the device. For example, the X Server needs to know Device File (/dev/dri/card0) the exact model of the GPU in order to load the cor- rect libraries. As such, the device driver and the kernel Virtual Filesystem Switch collect this information and export it to the user space, e.g., through special file systems of procfs and sysfs Device File Operation Interface CVD frontend in Linux.