Programmer’s Guide Release 2.2.0 January 16, 2016 CONTENTS 1 Introduction 1 1.1 Documentation Roadmap...............................1 1.2 Related Publications..................................2 2 Overview 3 2.1 Development Environment..............................3 2.2 Environment Abstraction Layer............................4 2.3 Core Components...................................4 2.4 Ethernet* Poll Mode Driver Architecture.......................6 2.5 Packet Forwarding Algorithm Support........................6 2.6 librte_net........................................6 3 Environment Abstraction Layer7 3.1 EAL in a Linux-userland Execution Environment..................7 3.2 Memory Segments and Memory Zones (memzone)................ 11 3.3 Multiple pthread.................................... 12 3.4 Malloc.......................................... 14 4 Ring Library 19 4.1 References for Ring Implementation in FreeBSD*................. 20 4.2 Lockless Ring Buffer in Linux*............................ 20 4.3 Additional Features.................................. 20 4.4 Use Cases....................................... 21 4.5 Anatomy of a Ring Buffer............................... 21 4.6 References....................................... 28 5 Mempool Library 31 5.1 Cookies......................................... 31 5.2 Stats.......................................... 31 5.3 Memory Alignment Constraints............................ 31 5.4 Local Cache...................................... 32 5.5 Use Cases....................................... 33 6 Mbuf Library 34 6.1 Design of Packet Buffers............................... 34 6.2 Buffers Stored in Memory Pools........................... 36 6.3 Constructors...................................... 36 6.4 Allocating and Freeing mbufs............................. 36 6.5 Manipulating mbufs.................................. 36 6.6 Meta Information.................................... 36 i 6.7 Direct and Indirect Buffers.............................. 38 6.8 Debug......................................... 39 6.9 Use Cases....................................... 39 7 Poll Mode Driver 40 7.1 Requirements and Assumptions........................... 40 7.2 Design Principles................................... 41 7.3 Logical Cores, Memory and NIC Queues Relationships.............. 42 7.4 Device Identification and Configuration....................... 42 7.5 Poll Mode Driver API................................. 44 8 IVSHMEM Library 47 8.1 IVHSHMEM Library API Overview.......................... 48 8.2 IVSHMEM Environment Configuration........................ 48 8.3 Best Practices for Writing IVSHMEM Applications................. 49 8.4 Best Practices for Running IVSHMEM Applications................ 49 9 Link Bonding Poll Mode Driver Library 50 9.1 Link Bonding Modes Overview............................ 50 9.2 Implementation Details................................ 51 9.3 Using Link Bonding Devices............................. 58 10 Timer Library 61 10.1 Implementation Details................................ 61 10.2 Use Cases....................................... 62 10.3 References....................................... 62 11 Hash Library 63 11.1 Hash API Overview.................................. 63 11.2 Implementation Details................................ 64 11.3 Entry distribution in hash table............................ 65 11.4 Use Case: Flow Classification............................ 66 11.5 References....................................... 67 12 LPM Library 68 12.1 LPM API Overview.................................. 68 12.2 Implementation Details................................ 68 13 LPM6 Library 72 13.1 LPM6 API Overview.................................. 72 13.2 Use Case: IPv6 Forwarding............................. 76 14 Packet Distributor Library 77 14.1 Distributor Core Operation.............................. 78 14.2 Worker Operation................................... 79 15 Reorder Library 80 15.1 Operation........................................ 80 15.2 Implementation Details................................ 80 15.3 Use Case: Packet Distributor............................. 81 16 IP Fragmentation and Reassembly Library 82 16.1 Packet fragmentation................................. 82 ii 16.2 Packet reassembly................................... 82 17 Multi-process Support 85 17.1 Memory Sharing.................................... 85 17.2 Deployment Models.................................. 86 17.3 Multi-process Limitations............................... 88 18 Kernel NIC Interface 89 18.1 The DPDK KNI Kernel Module............................ 89 18.2 KNI Creation and Deletion.............................. 91 18.3 DPDK mbuf Flow................................... 91 18.4 Use Case: Ingress................................... 92 18.5 Use Case: Egress................................... 92 18.6 Ethtool......................................... 92 18.7 Link state and MTU change.............................. 92 18.8 KNI Working as a Kernel vHost Backend...................... 92 19 Thread Safety of DPDK Functions 96 19.1 Fast-Path APIs..................................... 96 19.2 Performance Insensitive API............................. 97 19.3 Library Initialization.................................. 97 19.4 Interrupt Thread.................................... 97 20 Quality of Service (QoS) Framework 98 20.1 Packet Pipeline with QoS Support.......................... 98 20.2 Hierarchical Scheduler................................ 99 20.3 Dropper......................................... 123 20.4 Traffic Metering.................................... 132 21 Power Management 134 21.1 CPU Frequency Scaling................................ 134 21.2 Core-load Throttling through C-States........................ 135 21.3 API Overview of the Power Library.......................... 135 21.4 User Cases....................................... 135 21.5 References....................................... 135 22 Packet Classification and Access Control 136 22.1 Overview........................................ 136 22.2 Application Programming Interface (API) Usage.................. 141 23 Packet Framework 144 23.1 Design Objectives................................... 144 23.2 Overview........................................ 144 23.3 Port Library Design.................................. 145 23.4 Table Library Design.................................. 146 23.5 Pipeline Library Design................................ 160 23.6 Multicore Scaling................................... 162 23.7 Interfacing with Accelerators............................. 163 24 Vhost Library 164 24.1 Vhost API Overview.................................. 164 24.2 Vhost Implementation................................. 165 24.3 Vhost supported vSwitch reference......................... 166 iii 25 Port Hotplug Framework 167 25.1 Overview........................................ 167 25.2 Port Hotplug API overview.............................. 167 25.3 Reference....................................... 168 25.4 Limitations....................................... 168 26 Source Organization 169 26.1 Makefiles and Config................................. 169 26.2 Libraries........................................ 169 26.3 Drivers......................................... 170 26.4 Applications...................................... 170 27 Development Kit Build System 172 27.1 Building the Development Kit Binary......................... 172 27.2 Building External Applications............................ 174 27.3 Makefile Description.................................. 174 28 Development Kit Root Makefile Help 179 28.1 Configuration Targets................................. 179 28.2 Build Targets...................................... 179 28.3 Install Targets..................................... 180 28.4 Test Targets...................................... 180 28.5 Documentation Targets................................ 180 28.6 Deps Targets...................................... 181 28.7 Misc Targets...................................... 181 28.8 Other Useful Command-line Variables........................ 181 28.9 Make in a Build Directory............................... 181 28.10Compiling for Debug.................................. 182 29 Extending the DPDK 183 29.1 Example: Adding a New Library libfoo........................ 183 30 Building Your Own Application 185 30.1 Compiling a Sample Application in the Development Kit Directory........ 185 30.2 Build Your Own Application Outside the Development Kit............. 185 30.3 Customizing Makefiles................................ 185 31 External Application/Library Makefile help 187 31.1 Prerequisites...................................... 187 31.2 Build Targets...................................... 187 31.3 Help Targets...................................... 187 31.4 Other Useful Command-line Variables........................ 188 31.5 Make from Another Directory............................. 188 32 Performance Optimization Guidelines 189 32.1 Introduction....................................... 189 33 Writing Efficient Code 190 33.1 Memory......................................... 190 33.2 Communication Between lcores........................... 191 33.3 PMD Driver....................................... 192 33.4 Locks and Atomic Operations............................ 192 33.5 Coding Considerations................................ 193 iv 33.6 Setting the Target CPU Type............................. 193 34 Profile Your Application 194 35 Glossary 195 v CHAPTER ONE INTRODUCTION