Computer Science Tobias Rang NFV performance benchmarking with OVS and Linux containers Bachelor’s Project NFV performance benchmarking with OVS and Linux containers © 2017 The author(s) and Karlstad University This report is submitted in partial fulfillment of the requirements for the Bachelor’s degree in Computer Science. All material in this report which is not my own work has been identified and no material is included for which a degree has previously been conferred. Approved, 170605 Advisor: Andreas Kassler Examiner: Stefan Alfredsson iii Abstract One recent innovation in the networking industry, is the concept of Network Function Virtualization (NFV). NFV is based on a networking paradigm in which network functions, which have typically been implemented in the form of dedicated hardware appliances in the past, are implemented in software and deployed on commodity hardware using modern virtualization techniques. While the most common approach is to place each virtual network function in a virtual machine - using hardware-level virtualization – the growing influence and popularity of Docker and other container-based solutions has naturally led to the idea of containerized deployments. This is a promising concept, as containers (or operating system level virtualization) can offer a flexible and lightweight alternative to hardware-level virtualization, with the ability to use the resources of the host directly. The main problem with this concept, is the fact that the default behavior of Docker and similar technologies is to rely on the networking stack of the host, which typically isn’t performant enough to handle the performance requirements associated with NFV. In this dissertation, an attempt is made to evaluate the feasibility of using userspace networking to accelerate the network performance of Docker containers, bypassing the standard Linux networking stack by moving the packet processing into userspace. v Contents 1 Introduction............................................................................................................................1 1.1 Motivation......................................................................................................................2 1.2 Scope..............................................................................................................................2 1.3 Outline............................................................................................................................3 2 Related Work..........................................................................................................................3 2.1 The traditional Linux networking stack.........................................................................3 2.1.1 High-level overview 2.1.2 Performance limitations and potential optimizations 2.2 Containers......................................................................................................................6 2.2.1 Historical overview 2.2.2 Container isolation 2.2.3 Container networking 2.2.4 Containers and traditional virtual machines 2.3 Network function virtualization.....................................................................................9 2.3.1 NFV and its relation to Software Defined Networking 2.4 DPDK...........................................................................................................................10 2.5 CloudLab......................................................................................................................11 2.6 Open vSwitch...............................................................................................................11 2.6.1 Vanilla OVS 2.6.2 OVS-DPDK 2.7 MoonGen.....................................................................................................................14 2.7.1 Overview 2.7.2 Timestamping and latency measurements 3 Methodology.........................................................................................................................15 3.1 Approach.....................................................................................................................15 3.2 Test environment..........................................................................................................17 3.3 Experiments.................................................................................................................20 4 Results...................................................................................................................................21 4.1 Baseline performance...................................................................................................22 4.1.1 Throughput 4.1.2 Latency 4.2 Test case 1....................................................................................................................24 4.2.1 Throughput 4.2.2 Latency 4.3 Test case 2....................................................................................................................26 4.3.1 Throughput 4.3.2 Latency 4.4 Test case 3....................................................................................................................28 4.4.1 Throughput 4.4.2 Latency vi 5 Conclusion.............................................................................................................................30 5.1 Future work..................................................................................................................31 vii List of Figures Figure 1: Linux networking stack, packet RX overview [10]....................................................4 Figure 2: Linux networking stack overview, categorized according to kernel space / userspace [21]..............................................................................................................................................6 Figure 3: Containers and VMs, architectural differences [3].....................................................9 Figure 4: Standard OVS architecture and OVS-DPDK architecture [2].................................13 Figure 5: OVS-DPDK, switching table overview [36]............................................................13 Figure 6: MoonGen architecture, overview [39].....................................................................14 Figure 7: Benchmark setup, overview.....................................................................................17 Figure 8: Standard OVS - throughput for different packet sizes.............................................22 Figure 9: Standard OVS - latency - cumulative distribution function, 64 byte packets..........23 Figure 10: Throughput for different packet sizes.....................................................................24 Figure 11: Latency - cumulative distribution function, 64 byte packets..................................25 Figure 12: Throughput for different packet sizes with increasingly limited CPU access........26 Figure 13: Latency for different packet sizes with increasingly limited CPU access..............27 Figure 14: Throughput for different packet sizes.....................................................................28 Figure 15: Latency - cumulative distribution function............................................................29 Figure 16: Latency for different packet sizes...........................................................................36 Figure 17: Low latency configuration - throughput for different packet sizes........................37 Figure 18: Low latency configuration - latency, cumulative distribution function, 64 byte packets.......................................................................................................................................38 viii List of Tables Table 1: Software versions........................................................................................................16 Table 2: Balanced DPDK configuration...................................................................................19 Table 3: Standard OVS - merged result from multiple invocation of the latency benchmark (microseconds)..........................................................................................................................23 Table 4: Merged result from multiple invocation of the latency benchmark (microseconds)..25 Table 5: Throughput and standard deviation for 64 byte packets with increasingly limited CPU access................................................................................................................................26 Table 6: Latency and standard deviation for different packet sizes with increasingly limited CPU access................................................................................................................................27 Table 7: Merged result from multiple invocation of the latency benchmark (microseconds)..29 9 List of Scripts Script 1: snippet from start_single_container.py - container command line arguments...........17 Script 2: setup_ovs.sh...............................................................................................................18 Script 3: Kernel options............................................................................................................18 x 1 Introduction The concept of Network Function Virtualization (NFV) [1] is based on a networking paradigm in which Network Functions (NFs) are implemented in software and deployed on commodity hardware – leveraging modern virtualization techniques [2]. NFV has generated a lot of interest