Bachelor’s Thesis Nr. 198b Systems Group, Department of Computer Science, ETH Zurich Device Queues for USB by Joël Busch Supervised by Prof. Timothy Roscoe, Reto Achermann, Roni Häcki Systems Group, ETH Zurich November 2017–May 2018 Abstract The motivation behind this thesis is that we want to provide the research operating system Barrelfish with a useful feature and in the course of implementing it we want to prove that some of Barrelfish’s newer systems work well. Therefore the goal we chose was to develop a USB mass storage service. The implementation of the service provides an opportunity to use the newly introduced Device Queues for inter-process bulk communication. Demonstrating their functionality and effi- ciency is part of our aim. Furthermore, in the course of developing the USB mass storage driver we can show how well the system knowledge base and the device manager cooperate to enable event based driver startup. Our approach was to first reorganize the existing USB subsystem to conform to the new driver model for Barrelfish and to change its initialization to be based around events dispatched by the system knowledge base. Next we added a USB mass storage driver to the subsystem and used Device Queues for providing clients with access to the service over a zero-copy channel. Finally Barrelfish’s FAT implemen- tation was extended to include write support and its back-end was modified to use the aforementioned communication channel. The resulting mass storage service achieves better performance than Linux when both are run virtualized on the same host with the same USB hardware being passed through. Some stability issues, particularly based around FAT, still need to be ad- dressed and performance can still be improved, but the core functionality has been demonstrated. We further provide evidence that the Device Queues do not add any measurable communication overhead. Finally we show that the device manager and the system knowledge base robustly solve the problem of device initialization on a discoverable bus. 1 2 Contents 1 Introduction 5 1.1 Thesis Outline . .7 2 Background 9 2.1 Barrelfish . .9 2.1.1 Inter-domain communication . 10 2.1.2 System initialization . 11 2.2 Protocols . 14 2.2.1 USB . 14 2.2.2 SCSI . 17 2.2.3 FAT . 18 2.3 Previous USB subsystem . 19 3 Design 21 3.1 New driver model . 21 3.2 Queue setup . 22 3.3 Initialization . 24 3.4 USB mass storage driver . 25 3.5 FAT ................................. 26 4 Implementation 27 4.1 New driver model and Initialization . 27 4.2 USB mass storage driver . 29 4.3 Queue setup . 30 4.4 FAT ................................. 30 5 Evaluation 33 5.1 Aim . 33 5.2 Testing Setup . 34 5.2.1 Systems and Hardware . 34 5.2.2 Testing the full stack . 34 5.2.3 Isolating FAT . 35 5.2.4 Isolating the Device Queue . 36 3 4 CONTENTS 5.2.5 Generating a baseline . 37 5.3 Test failures . 37 5.4 Results . 38 5.5 Interpretation . 40 6 Conclusion and Future Work 41 6.1 Conclusion . 41 6.2 Future Work . 42 Chapter 1 Introduction The Barrelfish research operating system [16] runs many of its services, includ- ing its drivers, in user-space processes. This necessitates efficient communication between processes, especially when facing large amounts of data to transfer. Previ- ously Barrelfish had a variety of small tools to enable bulk transfer of data between processes, but they all had various drawbacks. As a formal solution to the problem Device Queues [6] were recently added to Barrelfish. They allow for sending mem- ory buffers between processes without copying the data. Instead, ownership of a buffer that both processes have access to is passed back and forth. Only the current owner of a location is allowed to access the buffer. Since there is no effective safe- guard against access by the non-owner, the queue is intended to be used between cooperative processes. When using a hardware specific back-end, the same func- tionality can be used to write directly into hardware buffers of the device, enabling zero-copy usage. File systems communicating with user programs are a common use case that in- volves shifting large amounts of data between a driver in one process to a user application in another. This project specifically targets a file system on a USB mass storage device [10], because it is a good opportunity to demonstrate the use of Device Queues in a real world scenario. Another new addition to Barrelfish is the specification of a more unified driver interface [5] that also allows for drivers to share a process and employ local direct communication within the same virtual memory space. This is of interest to the USB subsystem, because there are various drivers typically running that are tightly coupled and could benefit from sharing a process. At a minimum three drivers must be running and working together if any functionality of a USB device needs to be used. The situation before the project was that one application called the USB man- ager [1] was running driver code for the host controller, who is responsible for 5 6 CHAPTER 1. INTRODUCTION enabling communication with all other USB devices, and for the hubs, which are port multipliers for USB. It was directly responsible for bringing up the drivers of USB devices that were connected. Startup of the USB manager was hard coded into the device manager [4] and the USB manager had it’s own stub mechanism for locating USB driver binaries that was working only for the case of the keyboard driver. A better solution can be achieved with two mechanisms in tandem: Firstly the common interface in the new driver model [5] allows the device manager to start various drivers in a more unified way. Secondly the publish-subscribe server Oc- topus [20] can asynchronously inform the device manager of new device records appearing in the system knowledge base [7]. Based on these we can design a system where the device manager reacts to events signifying the appearance of new devices. It can then easily and dynamically load drivers for the device, because they conform to the new interface. This also means that the discovery code running in the hub driver can be made agnostic of driver startup routines. It only needs to put a device record into the system knowledge base. For these reasons the project is aimed at transforming the existing USB implemen- tation in Barrelfish in multiple ways. First of all the existing driver is ported to the new device driver model, including the addition of new instantiation routines by the device manager. The second change is allowing communication with the USB driver over Device Queues for efficient bulk data transfers. And finally, to enable testing of the queues a USB mass storage driver is added, including supporting in- frastructure such as a partial SCSI [15] implementation to encapsulate in the USB transport and changes to the existing FAT file system [14]. The contributions of the thesis are: • Splitting all the USB code into driver modules conforming to the new driver interface • Having the device manager perform event based initialization of devices in the USB subsystem • Producing a USB mass storage driver that allows clients to connect over a Device Queue interface • Merging write support into the FAT library and making adjustments allowing it to connect to the aforementioned Device Queue 1.1. THESIS OUTLINE 7 1.1 Thesis Outline Beyond this introduction the thesis is split into five chapters. Chapter 2 contains explanations of various relevant Barrelfish subsystems and some of the protocols and standards used. Chapter 3 provides the new design of the USB subsystem as it relates to initialization and inter domain communication, including the reasoning behind design choices. Chapter 4 describes various implementation details that may be relevant primarily in system maintenance. Chapter 5 presents the testing setup used for performance evaluation and the results obtained from it. Finally, chapter 6 draws conclusions from the data and experiences and casts a gaze forward towards possible future improvements. 8 CHAPTER 1. INTRODUCTION Chapter 2 Background 2.1 Barrelfish Barrelfish is a research operating system developed by researchers in the Systems Group of ETH Zürich. The main goal behind Barrelfish is exploring designs for better scalability as well as system heterogeneity. It finds further use as a platform which to base related research and teaching in the area of operating system design on [16]. The base design chosen for Barrelfish is that of the multikernel [8], where each core independently runs a small kernel. The kernels do not share memory and do not need to be identical. In fact they are regarded simply as CPU drivers within Barrelfish. They do not enjoy special treatment and are simply started by the device manager like other device drivers. This allows kernels for different architectures to run within a single system. The kernels are minimal by design, much like those of microkernel [12] operating systems. They only provide access to kernel objects and memory, access to the core system hardware, a channel for local communication within a CPU core, and scheduling to user space dispatchers. Such a dispatcher is, from the kernel’s perspective, akin to a process in other op- erating systems, but with the limitation of being fixed to the single core the kernel that controls it lives on. One or multiple dispatchers, interacting over various means of message passing, form a domain together. A domain is the abstraction that is mostly equivalent to a process from the user’s perspective.