Network I/O Fairness in Virtual Machines

Network I/O Fairness in Virtual Machines Muhammad Bilal Anwer, Ankur Nayak, Nick Feamster, Ling Liu School of Computer Science, Georgia Tech ABSTRACT tive, they fall into two categories: the driver-domain model We present a mechanism for achieving network I/O fair- and those that provide direct I/O access for high speed net- ness in virtual machines, by applying flexible rate limiting work I/O. mechanisms directly to virtual network interfaces. Con- A key problem in virtual machinesis network I/O fairness, ventional approaches achieve this fairness by implementing which guarantees that no virtual machine can have dispro- rate limiting either in the virtual machine monitor or hyper- portionate use of the physical network interface. This paper visor, which generates considerable CPU interrupt and in- presents a design that achieves network I/O fairness across struction overhead for forwarding packets. In contrast, our virtual machines by applying rate limiting in hardware on design pushes per-VM rate limiting as close as possible to virtual interfaces. We show that applying rate limiting in the physical hardware themselves, effectively implementing hardware can reduce the CPU cycles required to implement per-virtual interface rate limiting in hardware. We show that per-virtual machine rate limiting for network I/O. Our de- this design reduces CPU overhead (both interrupts and in- sign applies generally to network I/O fairness for network structions) by an order of magnitude. Our design can be interfaces in general, making our contributions applicable to applied either to virtual servers for cloud-based services, or any setting where virtual machines need network I/O fair- to virtual routers. ness (i.e., either for virtual servers [9] or virtual routers [20]). Our goals for the implementation of network I/O fair- Categories and Subject Descriptors: C.2.1 [Computer- ness for virtual machines are four-fold. First, the rate lim- Communication Networks]: Network Architecture and De- iter should be fair: it should be able to enforce network sign C.2.5 [Computer-Communication Networks]: Local I/O fairness across different virtual machines. Second, the and Wide-Area Networks C.2.6 [Computer-Communication mechanism must be scalable: the mechanism must scale as Networks]: Internetworking the number of virtual machines increases. Third, it must be flexible: because virtual machines may be continually General Terms: Algorithms, Design, Experimentation, Per- remapped to the underlying hardware, the mechanism must formance operate in circumstances where virtual ports may be fre- Keywords: Network Virtualization, NetFPGA, Xen quently remapped to underlying physical ports; this associa- tion should also be easily programmable. Finally, the mechanism must be robust, so that neither benign users nor mali- 1. Introduction cious attackers can subvert the rate-control mechanism. An important aspect of virtual machine design and im- We achieve these design goals with a simple principle: plementation is fairness in resource allocation across virtual push rate limiting as close as possible to the underlying machines. Although it is relatively well understood how to physical hardware, suppressing as many software interrupts fairly allocate computing resources like CPU cycles, the no- as possible. Our implementation builds on our previous tion of fairnessbecomesless clear whenit is applied to I/O— work in fast data planes for virtual routers that are built on and, in particular, network I/O. A common approach for con- commodity hardware [8]. The design suppresses interrupts trolling I/O resource utilization is to implement scheduling by implementing rate limiting for each virtual queue in hard- in the virtual machine monitor or hypervisor. Unfortunately, ware, preventing the hypervisor or operating system from this approach induces significant CPU overhead due to I/O ever needing to process the packets or implement any fair- interrupt processing. ness mechanisms. Our evaluation shows that implementing Various approaches to increase the performanceof virtual- rate limiting directly in hardware, rather than relying on the machine I/O have been proposed. Some try to provide new virtual machine hypervisor, can reduce both CPU interrupts scheduling algorithms for virtual machines for better net- and the required number of instructions to forward packets work I/O performance. Other techniques use existing sched- at a certain rate by an order of magnitude. ulers and try to provide system optimizations, both in soft- The rest of this paper proceeds as follows. Section 2 pro- ware and in hardware. From the operating system perspec- vides background and related work on network virtualization in virtual machine’s context. Section 4 presents the basic Permission to make digital or hard copies of all or part of this work for design of a virtualized network interface card to enable net- personal or classroom use is granted without fee provided that copies are work resource isolation in hardware and freeing CPU from not made or distributed for profit or commercial advantage and that copies unwanted packets overhead to do better virtual machine re- bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific source scheduling; this design is agnostic to any specific pro- permission and/or a fee. grammable hardware platform. Section 5 presents an imple- VISA’10, September 3, 2010, New Delhi, India. mentation of our design using the NetFPGA platform. Sec- Copyright 2010 ACM 978•1•4503•0199•2/10/09_..$10.00.. 73 tion 6 presents a preliminary evaluation of the virtualized network interface card; Section 7 discusses limitations and possible extensions of the current implementation. Section 8 concludes with a summary and discussion of future work. 2. Background and Related Work Many CPU scheduling algorithms in virtual machines fol- low process scheduling algorithms from operating systems. Figure 1: Xen Separate Driver Domain. For example, Xen [9] has implementations of Borrowed Vir- tual Time (BVT) and Simple Earliest Deadline First (sEDF). Recent work [10,15] has shown that CPU schedulers do not perform well in the presence of a combination of I/O intensive and CPU intensive applications running together on a single server. It can result in unfair scheduling of resources for CPU intensive or network I/O intensive virtual machines. Virtual machine monitors must maintain both network I/O fairness and CPU resource fairness in a virtualized environment while providing the best performance for both CPU- intensive and network-intensive applications. To enable fair sharing of CPU and network resources, we offload the task Figure 2: Dom0 as Driver Domain. of network virtualization and fairness from the CPU and enforce it in the virtualized network interface card. It also assigns each virtual machine to a queue and bypasses Network I/O virtualization has been used in both OS vir- the driver domain to provide direct network traffic I/O. tualization and a non-OS virtualization context. Virtual In- Many virtual machine monitors (e.g., Xen [9], Microsoft terface Architecture (VIA) [12] is an abstract model that is Hyper-V [2] and L4 [16]) use the driver-domain model to targeted towards system area networks and tries to reduce provide virtualization of I/O devices. In Xen, the driver do- the amount of software overhead compared to traditional main can be run separately or in domain 0, as shown in Fig- communication models. In traditional models, the OS ker- ures 1 and 2, respectively. This model provides a safe ex- nel multiplexes the access to hardware peripherals, so all ecution environment for the device drivers, which enables communication must trap within the kernel. VIA tries to improved fault isolation. remove this communication overhead by removing the ker- Both of these approaches have various problems. Al- nel operations in each communication operation. Similarly, though direct I/O provides better performance, it can cause Remote Direct Memory Access (RDMA) [4] is a data com- reliability problems [11]. The driver-domain model is pre- munication model that allows the network interface card di- ferred because a separate domain can handle device failures rect memory access to application memory without copying without affecting the guest domains [24]. Direct I/O also re- data between kernel and user space. quires the guest OS to have the device-specific driver, which In OS virtualization, the driver-domain model and the di- increases the complexity of the guest OS and thus makes rect I/O model are two approaches for achieving network porting and migration more difficult. The driver-domain I/O performance while maintaining fairness. The driver- model, on the other hand attempts to keep the guest OS sim- domain model [21,23] uses a separate virtual machine for ple, but it does suffer from performance overhead, as the device drivers. This domain runs a mostly unmodified oper- driver domain becomes a bottleneck, since every incoming ating system and simplifies the problem of providing device packet has to be inside the driver domain before it can be drivers to many virtual machines. The driver domain pro- copied to the guest domain. vides better fault tolerance by isolating driver failures to a In both driver-domain model and direct I/O model, inter- separate domain compared to maintaining device drivers in rupts are received by hypervisor, which dispatches them to hypervisor. On the other hand, the direct I/O model gives the driver domain in case of driver domain model or to the direct hardware access to the guest domains running on the guest domains in case of direct I/O model. In this work we server [6,7,17,22]. This approach provides near-native per- preserve the driver-domain model and assign virtual queues formance, but it sacrifices fault isolation and device trans- to every virtual machine running on the server.

Load more