Onload User Guide Copyright © 2017 SOLARFLARE Communications, Inc. All rights reserved. The software and hardware as applicable (the “Product”) described in this document, and this document, are protected by copyright laws, patents and other intellectual property laws and international treaties. The Product described in this document is provided pursuant to a license agreement, evaluation agreement and/or non‐disclosure agreement. The Product may be used only in accordance with the terms of such agreement. The software as applicable may be copied only in accordance with the terms of such agreement. Onload is licensed under the GNU General Public License (Version 2, June 1991). See the LICENSE file in the distribution for details. The Onload Extensions Stub Library is Copyright licensed under the BSD 2‐Clause License. Onload contains algorithms and uses hardware interface techniques which are subject to Solarflare Communications Inc patent applications. Parties interested in licensing Solarflare's IP are encouraged to contact Solarflare's Intellectual Property Licensing Group at: Director of Intellectual Property Licensing Intellectual Property Licensing Group Solarflare Communications Inc, 7505 Irvine Center Drive Suite 100 Irvine, California 92618 You will not disclose to a third party the results of any performance tests carried out using Onload or EnterpriseOnload without the prior written consent of Solarflare. The furnishing of this document to you does not give you any rights or licenses, express or implied, by estoppel or otherwise, with respect to any such Product, or any copyrights, patents or other intellectual property rights covering such Product, and this document does not contain or represent any commitment of any kind on the part of SOLARFLARE Communications, Inc. or its affiliates. The only warranties granted by SOLARFLARE Communications, Inc. or its affiliates in connection with the Product described in this document are those expressly set forth in the license agreement, evaluation agreement and/or non‐disclosure agreement pursuant to which the Product is provided. EXCEPT AS EXPRESSLY SET FORTH IN SUCH AGREEMENT, NEITHER SOLARFLARE COMMUNICATIONS, INC. NOR ITS AFFILIATES MAKE ANY REPRESENTATIONS OR WARRANTIES OF ANY KIND (EXPRESS OR IMPLIED) REGARDING THE PRODUCT OR THIS DOCUMENTATION AND HEREBY DISCLAIM ALL IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NON‐INFRINGEMENT, AND ANY WARRANTIES THAT MAY ARISE FROM COURSE OF DEALING, COURSE OF PERFORMANCE OR USAGE OF TRADE. Unless otherwise expressly set forth in such agreement, to the extent allowed by applicable law (a) in no event shall SOLARFLARE Communications, Inc. or its affiliates have any liability under any legal theory for any loss of revenues or profits, loss of use or data, or business interruptions, or for any indirect, special, incidental or consequential damages, even if advised of the possibility of such damages; and (b) the total liability of SOLARFLARE Communications, Inc. or its affiliates arising from or relating to such agreement or the use of this document shall not exceed the amount received by SOLARFLARE Communications, Inc. or its affiliates for that copy of the Product or this document which is the subject of such liability. The Product is not intended for use in medical, life saving, life sustaining, critical control or safety systems, or in nuclear facility applications. A list of patents associated with this product can be found at: http://www.solarflare.com/patent SF‐104474‐CD Last Revised: February 2017 Issue 22 Issue 22 © Solarflare Communications 2017 i Onload User Guide Trademark OpenOnload® and EnterpriseOnload® are registered trademarks of Solarflare Communications Inc in the United States and other countries. Issue 22 © Solarflare Communications 2017 ii Onload User Guide Table of Contents 1 What’s New . .1 1.1 New features in OpenOnload 201601‐u1 . .1 1.2 New features in OpenOnload 201606 . .2 1.3 Change history. .4 2 Low Latency Quickstart Guide . .5 2.1 Introduction . .5 2.2 Software Installation . .5 2.3 Test Setup. .6 2.4 Latency . .8 2.5 Testing Without Onload . .12 2.6 Further Information . .13 3 Background . .14 3.1 Introduction . .14 4 Installation. .18 4.1 Introduction . .18 4.2 Onload Distributions . .18 4.3 Hardware and Software Supported Platforms . .19 4.4 Onload and the Network Adapter Driver . .20 4.5 Removing Previously Installed Drivers. .20 4.6 Migrating Between Onload Versions ‐ Upgrade/Downgrade. .21 4.7 EnterpriseOnload ‐ Build and Install from SRPM . .23 4.8 EnterpriseOnload ‐ Debian Source Packages. .24 4.9 OpenOnload DKMS Installation . .25 4.10 Build OpenOnload Source RPM . .25 4.11 OpenOnload ‐ Installation. .25 4.12 Onload Kernel Modules . .27 4.13 Configuring the Network Interfaces. .28 4.14 Installing Netperf. .28 4.15 How to run Onload . .28 4.16 Testing the Onload Installation. .28 4.17 Apply an Onload Patch . .28 Issue 22 © Solarflare Communications 2017 iii Onload User Guide Table of Contents 5 Tuning Onload . .30 5.1 Introduction . .30 5.2 System Tuning . .31 5.3 Standard Tuning. .33 5.4 Onload Deployment on NUMA Systems . .36 5.5 Interrupt Handling ‐ Kernel Driver . .38 5.6 Performance Jitter. .44 5.7 Advanced Tuning . .47 6 Onload Functionality. .54 6.1 Onload Transparency. .54 6.2 Onload Stacks. .54 6.3 Virtual Network Interface (VNIC) . .55 6.4 Functional Overview . .55 6.5 Onload with Mixed Network Adapters . .55 6.6 Maximum Number of Network Interfaces . .56 6.7 Whitelist and Blacklist Interfaces . .56 6.8 Onloaded PIDs . .56 6.9 Onload and File Descriptors, Stacks and Sockets . .57 6.10 System calls intercepted by Onload . .57 6.11 Linux Sysctls . .57 6.12 Changing Onload Control Plane Table Sizes . .60 6.13 SO_BINDTODEVICE . .60 6.14 Multiplexed I/O . .61 6.15 Wire Order Delivery . .65 6.16 Stack Sharing . .67 6.17 Application Clustering. .68 6.18 Bonding, Link aggregation and Failover. .69 6.19 Teaming . .70 6.20 VLANS. .71 6.21 Accelerated pipe() . .72 6.22 Zero‐Copy API . .73 6.23 Debug and Logging . .73 7 TimeStamps . .74 7.1 Introduction . .74 7.2 Software Timestamps . .74 7.3 Hardware Timestamps . .75 7.4 Timestamping ‐ Example Applications . .77 Issue 22 © Solarflare Communications 2017 iv Onload User Guide Table of Contents 8 Onload ‐ TCP . .80 8.1 TCP Operation . .80 8.2 TCP Handshake ‐ SYN, SYNACK . ..