Yardstick Release Latest Jan 26, 2019 Contents 1 Yardstick Release Note 1 1.1 Yardstick Release Notes.........................................1 2 Yardstick User Guide 7 2.1 Introduction...............................................7 2.2 Methodology...............................................8 2.3 Architecture............................................... 11 2.4 Yardstick Installation........................................... 19 2.5 Yardstick Usage............................................. 29 2.6 Installing a plug-in into Yardstick.................................... 35 2.7 Store Other Project’s Test Results in InfluxDB............................. 37 2.8 Grafana dashboard............................................ 39 2.9 Yardstick Restful API.......................................... 43 2.10 Yardstick User Interface......................................... 53 2.11 Network Services Benchmarking (NSB)................................. 54 2.12 Yardstick - NSB Testing -Installation.................................. 57 2.13 Yardstick - NSB Testing - Operation................................... 81 2.14 Update Spirent Landslide TG configuration in pod file......................... 94 2.15 Update NSB test case definitions.................................... 95 2.16 Yardstick Test Cases........................................... 96 2.17 NSB Sample Test Cases......................................... 199 2.18 Glossary................................................. 230 2.19 References................................................ 231 3 Yardstick Developer Guide 233 3.1 Introduction............................................... 233 3.2 Yardstick developer areas........................................ 234 3.3 How Todos?............................................... 234 3.4 Backporting changes to stable branches................................. 241 3.5 Development guidelines......................................... 242 3.6 Plugins.................................................. 244 3.7 Introduction............................................... 244 3.8 Prerequisites............................................... 245 3.9 Sample Prox Test Hardware Architecture................................ 246 3.10 Prox Test Architecture.......................................... 246 3.11 NSB Prox Test.............................................. 249 3.12 How to run NSB Prox Test on an baremetal environment........................ 273 i 3.13 How to run NSB Prox Test on an Openstack environment....................... 274 3.14 Frequently Asked Questions....................................... 274 ii CHAPTER 1 Yardstick Release Note The Yardstick framework, the Yardstick test cases are open-source software, licensed under the terms of the Apache License, Version 2.0. 1.1 Yardstick Release Notes 1.1.1 Abstract This document compiles the release notes for the Gambia release of OPNFV Yardstick. 1.1.2 Version History Date Version Comment November 9, 2018 7.0.0 Yardstick for Gambia release December 14, 2018 7.1.0 Yardstick for Gambia release January 25, 2019 7.2.0 Yardstick for Gambia release 1.1.3 Important Notes The software delivered in the OPNFV Yardstick Project, comprising the Yardstick framework, and the Yardstick test cases is a realization of the methodology in ETSI-ISG NFV-TST001. The Yardstick framework is installer, infrastructure and application independent. 1.1.4 OPNFV Gambia Release This Gambia release provides Yardstick as a framework for NFVI testing and OPNFV feature testing, automated in the OPNFV CI pipeline, including: 1 Yardstick, Release Latest • Documentation generated with Sphinx – User Guide – Developer Guide – Release notes (this document) – Results • Automated Yardstick test suite (daily, weekly) – Jenkins Jobs for OPNFV community labs • Automated Yardstick test results visualization – Dashboard using Grafana (user:opnfv/password: opnfv), influxDB is used as backend • Yardstick framework source code • Yardstick test cases yaml files • Yardstick plug-in configuration yaml files, plug-in install/remove scripts For Gambia release, the Yardstick framework is used for the following testing: • OPNFV platform testing - generic test cases to measure the categories: – Compute – Network – Storage • OPNFV platform network service benchmarking (NSB) – NSB • Test cases for the following OPNFV Projects: – Container4NFV – High Availability – IPv6 – KVM – Parser – StorPerf – VSperf The Yardstick framework is developed in the OPNFV community, by the Yardstick team. Note: The test case description template used for the Yardstick test cases is based on the document ETSI-ISG NFV- TST001; the results report template used for the Yardstick results is based on the IEEE Std 829-2008. 2 Chapter 1. Yardstick Release Note Yardstick, Release Latest 1.1.5 Release Data 1.1.6 Deliverables Documents • User Guide: <yardstick:userguide> • Developer Guide: <yardstick:devguide> Software Deliverables • The Yardstick Docker image: https://hub.docker.com/r/opnfv/yardstick (tag: opnfv-7.0.0) List of Contexts Context Description Heat Models orchestration using OpenStack Heat Node Models Baremetal, Controller, Compute Standalone Models VM running on Non-Managed NFVi Kubernetes Models VM running on Non-Managed NFVi List of Runners Note: Yardstick Gambia 7.0.0 adds 1 new Runner, “IterationIPC”. Runner Description Arithmetic Steps every run arithmetically according to specified input value Duration Runs for a specified period of time Iteration Runs for a specified number of iterations IterationIPC Runs a configurable number of times before it returns. Each iteration has a configurable timeout. Sequence Selects input value to a scenario from an input file and runs all entries sequentially Dynamictp A runner that searches for the max throughput with binary search Search A runner that runs a specific time before it returns 1.1. Yardstick Release Notes 3 Yardstick, Release Latest List of Scenarios Category Delivered Availability Attacker: • baremetal, process HA tools: • check host, openstack, process, service • kill process • start/stop service Monitor: • command, process Compute • cpuload • cyclictest • lmbench • lmbench_cache • perf • unixbench • ramspeed • cachestat • memeoryload • computecapacity • SpecCPU2006 Networking • iperf3 • netperf • netperf_node • ping • ping6 • pktgen • sfc • sfc with tacker • networkcapacity • netutilization • nstat • pktgenDPDK Parser Tosca2Heat Storage • fio • bonnie++ • storagecapacity StorPerf storperf NSB vFW thoughput test case New Test cases Note: Yardstick Gambia 7.2.0 adds no new test cases. 4 Chapter 1. Yardstick Release Note Yardstick, Release Latest • Generic NFVI test cases • (e.g.) OPNFV_YARDSTICK_TCO84 - SPEC CPU 2006 for VM • HA Test cases • (e.g.) OPNFV_YARDSTICK_TC093 - SDN Vswitch resilience in non-HA or HA configuration 1.1.7 Version Change Module Version Changes This is the seventh tracked release of Yardstick. It is based on following upstream versions: • OpenStack Queens • OpenDayLight Oxygen Document Version Changes This is the seventh tracked version of the Yardstick framework in OPNFV. It includes the following documentation updates: • Yardstick User Guide: – Remove vTC chapter; • Yardstick Developer Guide • Yardstick Release Notes for Yardstick: this document Feature additions • Simplify Yardstick installation to use a single ansible playbook (nsb_setup.yaml). • Spirent support. • vEPC testcases. • Agnostic VNF tests cases for reuse of standard RFC-2544 test case. • PROX enhancements and the addition of Standalone test case using SRIOV and OVS-DPDK. • Ixia enhancements for vBNG and PPPoE traffic. • Improvements of unit tests and gating. • Add DPDK pktgen traffic generator. • Kubernetes context enhancements. • Kubernetes sample test cases of fio and lmbench added. 1.1.8 Scenario Matrix For Gambia 7.2.0, Yardstick was tested on the following scenarios: 1.1. Yardstick Release Notes 5 Yardstick, Release Latest Scenario Apex Compass Fuel-arm Fuel os-nosdn-nofeature-noha X os-nosdn-nofeature-ha X os-odl-bgpvpn-noha X os-nosdn-calipso-noha X os-nosdn-kvm-ha X os-odl-nofeature-ha X X os-odl-sfc-noha X os-nosdn-ovs-ha X k8-nosdn-nofeature-ha X k8-nosdn-stor4nfv-noha X k8-nosdn-stor4nfv-ha X 1.1.9 Test results Test results are available in: • jenkins logs on CI: https://build.opnfv.org/ci/view/yardstick/ Known Issues/Faults Corrected Faults Gambia 7.2.0: JIRA REFER- DESCRIPTION ENCE YARDSTICK-1512 [dovetail] split the sla check results into process recovery and service recovery for HA test cases. 1.1.10 Gambia 7.2.0 known restrictions/issues 1.1.11 Useful links • wiki project page: https://wiki.opnfv.org/display/yardstick/Yardstick • wiki Yardstick Gambia release planning page: https://wiki.opnfv.org/display/yardstick/Release+Gambia • Yardstick repo: https://git.opnfv.org/yardstick • Yardstick CI dashboard: https://build.opnfv.org/ci/view/yardstick • Yardstick grafana dashboard: http://testresults.opnfv.org/grafana/ • Yardstick IRC channel: #opnfv-yardstick 6 Chapter 1. Yardstick Release Note CHAPTER 2 Yardstick User Guide 2.1 Introduction Welcome to Yardstick’s documentation ! Yardstick is an OPNFV Project. The project’s goal is to verify infrastructure compliance, from the perspective of a Virtual Network Function (VNF). The Project’s scope is the development of a test framework, Yardstick, test cases and test stimuli to enable Network Function Virtualization Infrastructure (NFVI) verification. Yardstick is used in OPNFV for verifying the OPNFV infrastructure and some of the OPNFV features. The Yardstick framework is deployed in several