NetTLP: A Development Platform for PCIe devices in Software Interacting with Hardware Yohei Kuga and Ryo Nakamura, The University of Tokyo; Takeshi Matsuya, Keio University; Yuji Sekiya, The University of Tokyo https://www.usenix.org/conference/nsdi20/presentation/kuga This paper is included in the Proceedings of the 17th USENIX Symposium on Networked Systems Design and Implementation (NSDI ’20) February 25–27, 2020 • Santa Clara, CA, USA 978-1-939133-13-7 Open access to the Proceedings of the 17th USENIX Symposium on Networked Systems Design and Implementation (NSDI ’20) is sponsored by NetTLP: A Development Platform for PCIe devices in Software Interacting with Hardware Yohei Kuga1, Ryo Nakamura1, Takeshi Matsuya2, Yuji Sekiya1 1The Univeristy of Tokyo, 2Keio University Abstract Table 1: Comparison of platforms for prototyping PCIe de- Observability on data communication is always essential vices from the viewpoints of software and hardware. for prototyping, developing, and optimizing communication systems. However, it is still challenging to observe transac- PCIe device tions flowing inside PCI Express (PCIe) links despite them Software Hardware being a key component for emerging peripherals such as smart Software QEMU Root Complex − NICs, NVMe, and accelerators. To offer the practical observ- Hardware NetTLP FPGA/ASIC ability on PCIe and for productively prototyping PCIe devices, we propose NetTLP, a development platform for software PCIe devices that can interact with hardware root complexes. By contrast to the spread of PCIe, it is still difficult for On the NetTLP platform, software PCIe devices on top of researchers and software developers to observe PCIe and pro- IP network stacks can send and receive Transaction Layer totype PCIe devices, although they are crucial for optimizing Packets (TLPs) to and from hardware root complexes or other performance and developing future PCIe devices. Observ- devices through Ethernet links, an Ethernet and PCIe bridge ing PCIe transactions is difficult because PCIe transactions called a NetTLP adapter, and PCIe links. This paper describes are confined in hardware. PCIe is not just a simple fat-pipe the NetTLP platform and its implementation: the NetTLP between hardware elements; it also has several features for adapter and LibTLP, which is a software implementation of achieving high-performance communication, i.e., hardware the PCIe transaction layer. Moreover, this paper demonstrates interrupt, virtualization, and CPU cache operations. Utilizing the usefulness of NetTLP through three use cases: (1) observ- these features is important for exploiting PCIe efficiently; ing TLPs sent from four commercial PCIe devices, (2) 400 however, the concrete behaviors of the transactions in PCIe LoC software Ethernet NIC implementation that performs links cannot be determined unless special capture devices for an actual NIC for a hardware root complex, and (3) physical observing PCIe hardware are used. memory introspection. In addition to the observation, prototyping PCIe devices lacks productivity. Field Programmable Gate Array (FPGA) 1 Introduction is a major platform for prototyping PCIe devices [26,40,44, 45, 50]. However, developing all parts of a PCIe device on PCI Express (PCIe) is a widely used I/O interconnect for an FPGA still requires significant effort, such as the great storage, graphic, network, and accelerator devices [16,24,25]. devotion of the NetFPGA project [52] for networking devices. Not limited to connect the peripheral devices, some high- Another approach is to adopt virtualization or simulation, e.g., performance interconnects adopt the PCIe protocol [5,22,35]. GEM5 [11,23] and RTL simulators [4,31]. QEMU [9], which Moreover, specifications of future interconnects are designed is a famous virtualization platform, can be used for proto- by extending the PCIe protocol [10, 14]. Such versatility of typing PCIe devices from the software perspective. QEMU PCIe is derived from the packet-based data communication enables researchers and developers to prototype new hardware and the flexibility of PCIe topology. The PCIe specification architecture; however, its environment is fully softwarized. defines building blocks comprising PCIe topologies: endpoint, QEMU devices can communicate with only the emulated switches, bridges, and root complexes. PCIe packets flow root complex and cannot communicate with the physical root through point-to-point PCIe links between the blocks, and complex and other hardware connected to the root complex. motherboard manufacturers can expand the PCIe topologies The goal of this paper is to bridge the gap between software with these blocks depending on the use cases. and hardware for PCIe, as shown in Table 1. Our proposed USENIX Association 17th USENIX Symposium on Networked Systems Design and Implementation 141 platform, called NetTLP, offers softwarized PCIe endpoints CPU-to-Device/Device-to-CPU that can interact with hardware root complexes. By using Net- Device-to-Device Remote DMA TLP, researchers and software developers can prototype their CPU Memory Memory CPU PCIe devices as software PCIe endpoints and test the soft- ware devices with actual hardware root complexes through PCIe Switch Root Complex Root Complex the PCIe protocol. This hybrid platform of software and hard- PCIe PCIe PCIe PCIe PCIe PCIe ware simultaneously improves both the observability of PCIe Device Device Device Device Device Device transactions and the productivity of prototyping PCIe devices. The key technique for connecting softwarized PCIe end- Figure 1: A PCIe topology and three communication models. points to hardware root complexes is to separate the PCIe transaction layer into software and put the software transac- tion layer on top of IP network stacks. Our FPGA-based add- We show a prototype of a nonexistent Ethernet NIC • in card, called NetTLP adapter, delivers Transaction Layer with NetTLP. This prototyping demonstrates the high Packets (TLPs) to a remote host over Ethernet and IP net- productivity of the NetTLP platform; the NIC is certainly works. The substance of the NetTLP adapter is implemented implemented in software, but it performs as an actual in software on the IP network stack of the remote host with Ethernet NIC for a physical root complex. LibTLP, which is a software implementation of the PCIe trans- action layer. The NetTLP platform consisting of the adapter We demonstrate the possibility of developing memory • and library enables software PCIe devices on IP network introspection methods on NetTLP without implement- stacks to interact with hardware root complex through the ing dedicated devices. As a proof-of-concept, we imple- NetTLP adapter. Moreover, TLPs delivered over Ethernet mented two applications that gather process information links can be easily observed by IP networking techniques from a remote host by DMA through NetTLP. such as tcpdump and Wireshark. In this paper, we describe NetTLP, the novel platform for All source codes for hardware and software and captured software PCIe devices. To achieve the platform, we investi- data described in this paper are publicly available [1]. gate PCIe from the perspective of packet-based communica- tion (§2) and then describe the approach to connect the soft- 2 Background ware PCIe devices with hardware root complexes and process TLPs in software (§3) and describe its implementation (§4). PCIe is not only an interconnect, but also a packet-based data In addition to micro-benchmarks (§5), we demonstrate three communication network. As with IP networks, PCIe has a use cases of NetTLP (§6): observing behaviors of a root com- layering model composed of a physical layer, a data link layer, plex and commercial devices at a TLP-level, prototyping an and a transaction layer. The data link layer delivers PCIe Ethernet NIC in software interacting with a physical root packets across one hop over a PCIe link, while the transaction complex, and physical memory introspection using NetTLP. layer is responsible for delivering TLPs from a PCIe endpoint The contributions of this paper include the following: to a PCIe endpoint across the PCIe links. PCIe interconnect is We propose a novel platform for prototyping PCIe de- composed of the following elements that are capable of sup- • vices in software, while the software devices can commu- porting the layer functionalities: endpoints, switches, bridges, nicate with physical hardware such as root complexes, and root complexes. PCIe switches and root complexes route CPU, memory, and other PCIe devices. This platform and forward PCIe packets in accordance with the addresses offers high productivity for prototyping PCIe devices in memory-mapped I/O (MMIO) space or requester IDs. Any with actual interactions with hardware. functionalities of PCIe stand at the packet-based communi- cation, e.g., MSI-X for hardware interrupts is implemented We provide observability of PCIe transactions confined • by memory writes to specific memory addresses. Because of in hardware by the softwarized PCIe endpoints on the being such a packet-based network, PCIe topologies and their IP network stack. Our modified tcpdump can distinguish communication models are flexible, as depicted in Figure 1. the encapsulated TLPs in Ethernet by NetTLP, enabling In IP networks, we can easily prototype and implement any us to easily capture TLPs in an IP networking manner. part of the networks, such as end hosts, switches, and routers, We present detailed observation results of PCIe transac- and observe packets flowing