User Customization of Virtual Network Interfaces with U-Net/SLE David Opp enheimer and Matt Welsh fdavidopp,[email protected] Decemb er 9, 1997 Abstract We describ e U-Net/SLE Safe Language Extensions, a user-level network interface architecture which enables p er-application customization of communication semantics through downloading of user extension applets, imple- mented as Java class les, into the network interface. This architecture p ermits application s to safely sp ecify co de to b e executed within the NI on message transmission and reception. By leveraging the existing U-Net mo del, applications may implement proto col co de at the user level, within the NI, or using some combination of the two. Our current implementation, using the Myricom Myrinet interface and a small Java Virtual Machine subset, obtains go o d p erformance, allowing host communication overhead to b e reduced and improving the overlap of communication and computation during proto col pro cessing. 1 Intro duction Recentwork in high-sp eed interconnects for distributed and parallel computing architectures, particularly workstation clusters, has fo cused on developmentofnetwork interfaces enabling low-latency and high-bandwidth communication. Often, these systems bypass the op erating system kernel to achieve high p erformance; however the features and functionality provided by these di erent systems vary widely. Several systems, such as U-Net [26] and Active Messages [27], virtualize the network interface to provide multiple applications on the same host with direct, protected network access. Other systems, including Fast Messages [16] and BIP [17], eschew sharing the network in lieu of design simplicity and high p erformance. In addition, fast network interfaces often di er with resp ect to the communication semantics they provide, ranging from \raw" access U-Net to token-based ow-control Fast Messages to a full- featured RPC mechanism Active Messages. Complicating matters further is the sp ectrum of network adapter hardware up on which these systems are built, ranging from simple, fast NICs which require host intervention to multiplex the hardware [30] to advanced NICs incorp orating a programmable co-pro cessor [26]. Application programmers are faced with a wide range of functionalitychoices given the many fast networking layers currently available: some applications may b e able to take advantage of, say, the ow-control strategy imple- mented in Berkeley Active Messages, while others such as continuous media applications may wish to implement their own communication semantics entirely at user level. Additionally, the jury is still out on where certain features suchas ow-control and retransmission are b est implemented. It is tempting to base design choices on the results of microb enchmarks, such as user-to-user round-trip latency, but recent studies [12] has hinted that other factors, such as host overhead, are far more imp ortant in determining application-level p erformance. Given the myriad of p otential application needs, it may seem attractive to design for the lowest common denom- inator of network interface options, namely, the interface which provides only fast protected access to the network without implementing other features, such as RPC or ow-control, b elow the user level U-Net is one such design. This design enables applications to implement proto cols entirely at user level and do es not restrict communication semantics to some arbitrary set of \built-in" features. However, exp erience has shown [10] that in the interest of reducing host overhead, interrupts, and I/O bus transfers, it may b e b ene cial to p erform some proto col pro cessing 0 For more information on this pro ject, please see http://www.cs.berkeley.edu/~mdw/projects/unet-sle/. 1 within the network interface itself, for example on a dedicated network co-pro cessor [5]. Such a system could b e used to implementa multicast tree directly on the NI, allowing data to b e retransmitted down branches of the tree without intervention of the user application, overheads for I/O bus transfer, and pro cess or thread context switch. Another p otential application is packet-sp eci ed receive bu ers, in which the header of an incoming packet contains the bu er address in which the payload should b e stored. Being able to determine the packet destination bu er address b efore any I/O DMA o ccurs enables true zero-copy as long as the sender is trusted to sp ecify bu er addresses. Anumb er of systems have incorp orated these NI-side features in an ad hoc manner, however, it would seem desirable to have a consistent and universal mo del for fast network access which subsumes all of these features. We have designed an implemented U-Net/SLE Safe Language Extensions, a system which couples the U-Net user-level network interface architecture with user extensibilityby allowing the user to download customized packet-pro cessing co de, in the form of Java applets, into the NI. With this design, it is p ossible for multiple user applications to indep endently customize their interface with the U-Net architecture without compromising protection or p erformance. Applications which are content with the standard mo del provided by U-Net are able to use \direct" U-Net access and are not p enalized for features provided by the underlying system which they do not use. With the U-Net/SLE mo del, for example, it is p ossible for an application to implement sp ecialized ow-control and retransmission co de as a Java applet which is executed on the network interface. For instance, the semantics of the Active Messages layer could b e implemented as a combination of Java and user-level library co de. Those applications which require the use of Active Messages may use those features without requiring all applications on the same host to go through this interface, while still b eing able to take advantage of the ability to do NI-level pro cessing rather than pushing all proto col co de to user level. Anumb er of questions are raised by U-Net/SLE and other similar designs, including: How can p erformance b e maintained while protection b etween user extensions is enforced? How should resource management for user extensions b e handled? At what p oint do es it b ecome advantageous to move packet-pro cessing co de from user level into the NI, and vice versa? What tradeo s should b e considered when providing a certain feature in the network interface rather than on the host? In this pap er we describ e the design and implementation of U-Net/SLE, fo cusing on the considerations of using Java as a safe language for user extensions executed in the critical path of communication on a network interface. Section 2 of this pap er describ es the U-Net/SLE design in more detail. Section 3 describ es JIVE, our implemen- tation of the Java Virtual Machine used in U-Net/SLE. Sections 4 and 5 summarize p erformance, while section 6 describ es related work. Section 7 concludes and raises issues for future work to consider. 2 Design and Implementation U-Net/SLE is based on U-Net [26], a user-level network interface architecture whichmultiplexes the NI hardware between multiple applications such that each application has transparent, direct, protected access to the network. U- Net may b e implemented either in hardware, software, or a combination of b oth, and do es not presume any particular NIC design. On NICs with a programmable co-pro cessor, for instance, U-Net multiplexing/demultiplexing functions may b e implemented directly on the co-pro cessor, while on a non-programmable NIC a protected co-routine on the host may b e used to enforce protection. In the U-Net mo del an endpoint serves as an application's interface to the network and consists of a bu er area and transmit, receive, and free bu er queues see Figure 1. The bu er area is a pinned region of physical host RAM mapp ed into the user's address space; in order to ensure that the NI may p erform network bu er DMA at any time, 2 Tx FreeBuffer area Rx Figure 1: U-Net endp oint data structure 1 all transmit and receive bu ers are lo cated in this region. In order to transmit data, the user constructs the data in the bu er area and pushes a descriptor on the transmit queue indicating the lo cation and size of the data to b e transmitted as well as a channel tag, which indicates the intended recipient of the data. U-Net transmits the data and sets a ag in the transmit queue entry when complete. When data arrives from the network, U-Net determines the recipient endp oint for the packet, p ops a bu er address from that endp oint's free bu er queue and transfers the data into the bu er. Once the entire PDU has arrived whichmay span multiple receive bu ers, U-Net pushes a descriptor onto the user receive queue indicating the size, bu er addresses, and source channel tag of the data. As an optimization, small messages may tentirely within a receive queue descriptor. The user may p oll the receive queue or register an up call e.g. a signal handler to b e invoked when new data arrives. U-Net/SLE allows each user endp oint to b e asso ciated with a Java class le implementing the user extension 2 applet for that endp oint.