The Case Against User-Level Networking Kostas Magoutis Margo Seltzer Eran Gabber IBM T. J. Watson Research Center Harvard University Google, Inc. 19 Skyline Drive 33 Oxford Street 1440 Broadway, 21st Floor Hawthorne, NY 10532 Cambridge, MA 02138 New York, NY 10018 Abstract— Extensive research on system support for enabling mote direct data placement (RDDP) [16] enables an operating I/O-intensive applications to achieve performance close to the system (OS) structure that supports user-level file caching [25] limits imposed by the hardware suggests two main approaches: and user-level network buffering [3]. User-level caching of- Low overhead I/O protocols and the flexibility to customize I/O policies to the needs of applications. One way to achieve fers full control over I/O policies, an essential requirement both is by supporting user-level access to I/O devices, enabling for resource-intensive applications, such as databases. This user-level implementations of I/O protocols. User-level networking paper describes such an OS structure that blends ideas from is an example of this approach, specific to network interface traditional and novel/radical OS architectures: In accordance controllers (NICs). In this paper, we argue that the real key to with standard OS principles, resource abstractions, such as high-performance in I/O-intensive applications is user-level file caching and user-level network buffering, both of which can be files and network sockets, and global system policies, are achieved without user-level access to NICs. implemented in the kernel. However, similarly in spirit to Avoiding the need to support user-level networking carries more radical system designs, such as the Exokernel [20], full two important benefits for overall system design: First, a NIC control over application-specific I/O policies is possible with exporting a privileged kernel interface is simpler to design and caching implemented in user space. This caching, however, implement than one exporting a user-level interface. Second, the kernel is re-instated as a global system resource controller and is performed over the file or network socket abstractions arbitrator. We develop an analytical model of network storage rather than over the raw hardware. The key advantages of this applications and use it to show that their performance is not approach over Exokernel’s are improved security and safety affected by the use of a kernel-based API to NICs. due to a larger common kernel code-base, support for global resource policies, and improved portability. I. INTRODUCTION In summary, the main arguments presented in this paper The need to reduce networking overhead in system-area are: First, NICs exporting a user-level host interface pose networks in the early 1990’s motivated a flurry of research on implementation challenges not present in NICs exporting an user-level networking protocols. Projects such as SHRIMP [2], interface to a trusted host entity such as the kernel. Second, Hamlyn [4], and U-Net [36] proposed user-level access to a with the transport protocol offloaded to the NIC, the key to network interface controller (NIC) as an approach that offered lowering networking overhead is remote direct data placement. two primary benefits: First, it enabled host-based implemen- RDDP enables a new OS structure that supports user-level file tations of new, lightweight networking protocols with lower caching and user-level network buffering. The performance of overhead compared to kernel-based TCP/IP protocol stacks. a user-level API to file and network services should be similar Second, for applications requiring use of the TCP/IP protocol to the performace of a kernel-based API, since the difference is stack, there is a potential for application-specific customization only in user-kernel boundary crossing, which is not a dominant of user-level libraries. In recent years, the need for scalable cost. Third, kernel involvement is necessary in order to enforce and more manageable services, such as HTTP servers and global policies. Systems that advocate implementing all system network storage systems, leads to increased amounts of data policies in user-space libraries [20] do not offer a solution to transferred through the network. This trend links overall I/O the problem of implementing global policies without involving performance to the efficiency of the network subsystem and a privileged server. In addition, security and safety are other makes network interface controllers (NICs) a key system I/O benefits of our proposed design: For example, common kernel- device. The large-scale deployment of high-speed (1 Gb/s and based networking code may be more robust against attacks soon 10 Gb/s) Ethernet networks stimulated interest in the than a multitude of user-level networking stacks. A benefit of design of systems that offload TCP/IP to a new generation of user-level networking is better portability due to bypassing the NICs [27] and can transfer data directly between the network OS. However, a kernel interface that is simple to implement and application buffers. Many such TCP-offload NICs are should be rapidly incorporated into most mainstream operating currently being designed to export user-level interfaces to host systems. applications. The layout of this paper is as follows: In Section III we In this paper, we show that a user-level interface to NICs compare the benefits of user-level networking to those of a offering transport offload is not necessary. A kernel host kernel interface to a NIC offering transport offload and RDDP. interface to the NIC in combination with NIC support for re- In Section IV we describe an OS structure that takes advantage of RDDP to support user-level file caching and user-level artifact, rather than to the use of hardware address space pro- network buffering. In Section V we develop an analytical tection. Research on systems such as L4 [23] and Pebble [14] model of network storage applications to study the effect of the showed that the overhead of switching protection domains user-kernel protection boundary crossing on performance. Our could be as low as 150 machine cycles, allowing for efficient motivation to use an analytical model backed by experimental modular designs using hardware address spaces. In addition, measurements of key parameters, instead of a full experimental we believe that increased cache and TLB sizes in modern system comparison, is based on the fact that the latter depends CPUs will help to alleviate cache and TLB miss overheads. significantly on the quality of the implementations and focuses Another significant factor contributing to network overhead on a particular point of the design space. An analytical model is hardware interrupts, as shown in a recent study of network enables the examination of the entire design space, provides protocol performance over high speed networks [6]. Detecting significant qualitative results, and points to the key parameters I/O completions by polling is an alternative that is offered by affecting system performance. Earlier studies of operating many user-level networking protocols [9, 36]. This mechanism system structure, such as the classical work by Lauer and can reduce network overhead in network storage applications Needham [21], have also used analytical modeling to reason as shown by Zhou and her colleagues [37]. Polling I/O about system performance. Finally, in Section VI we present completions, however, does not necessarily depend on user- experimental results. level access to the NIC and is equally applicable within the kernel. II. RELATED WORK User-level NICs enable the construction of user-level op- Support for user-level networking was first offered in su- erating system libraries1 (libOSes) such as proposed by Ex- percomputers such as the Thinking Machines CM-5 [22], okernel [15, 20]. In the same spirit, the Direct Access File Meiko CS-2 [17], and IBM SP-2 [35], and later introduced System [25] (DAFS) is an example of a network attached in commodity networks by U-Net [36], Application Device storage system that proposes a user-level file system client. Channels (ADC) [13], Hamlyn [4], SHRIMP [2], DART [31] Just like in Exokernel, application-specific extensibility was and others. User-level networking was recently standardized one of the main motivations behind user-level networking. For with the Virtual Interface [9] (VI) and InfiniBand [18] architec- example, application-specific customization of the networking tures. In addition to the ability for user-level access to the NIC, stack in a user-level networking system was shown to be which is the defining feature of user-level NICs, most of these beneficial by enabling application-controlled flow control and projects also advocated a new host programming interface to feedback [36]. As the debate on extensibility technologies the NIC. This programming interface is based on a queue-pair eventually pointed out, however, the key is often in finding abstraction and requires pre-posting of receive buffers [3, 9]. It the right API to kernel services rather than supporting user- is important to note that this new programming interface is not level implementations of kernel subsystems. In many cases, a defining feature of user-level NICs and can be implemented parametrized kernel interfaces are sufficient from a perfor- without special NIC support [5]. In contrast, user-level access mance standpoint [12]. to the NIC necessarily requires special NIC support, increasing Another motivation for user-level networking was the ob- the complexity of NIC