Networking the Fd.Io/VPP Example

#CLUS Open Source for Networking The fd.io/VPP Example

Giles Heron – Principal Engineer BRKSDN-2262

#CLUS Agenda

• Why Open-Source?

• The Open-Source Networking Landscape

• Open-Source and Standards Bodies

• OpenDaylight & ONAP

• fd.io VPP - The Universal Fast Data Plane

• Using VPP with Virtual Machines and Containers

• The Future of VPP

Questions? Use Cisco Webex Teams to chat with the speaker after the session How 1 Find this session in the Cisco Live Mobile App 2 Click “Join the Discussion” 3 Install Webex Teams or go directly to the team space 4 Enter messages/questions in the team space

Webex Teams will be moderated cs.co/ciscolivebot#BRKSDN-2262 by the speaker until June 16, 2019.

Use Open Source Libraries to Solve Problems Open Source Code (~70%) Write Custom Code Custom Code (~10%)

Choose a Framework Open Source Code (~20%)

• Concern: OSS = Many contributors and contributions = Large attack surface for malicious contributions; Community response is “best effort”

• Reality: It largely depends on community of a project and associated tooling • Healthy projects (large/active community) fix issues really quickly • Healthy community means a larger and more diverse sets of eyes on the code: More likely to detect issues than in proprietary code • Healthy projects leverage tooling to continuously assess the code quality • Commercial distributions of OSS offer SLAs – Example security scan results: FD.io/VPP matching SLAs of proprietary software Source: https://scan.coverity.com/projects/fd-io-vpp

Components Platforms Open Systems

Enable and drive new Compose a framework of tightly Integrate components into technologies integrated components. systems. by creating and evolving key solution components.

Projects that address a narrowly Projects whose scope Projects that focus on the defined problem whose output encompasses multiple integration of platforms and may be consumed as an atomic components to yield a framework components, and are primarily entity. Examples: VPP (virtual that can be adapted to meet a used to test, demonstrate, and switch), a platform plug-in to range of different user needs. validate broader solutions. integrate new hardware or Examples: OpenDaylight, FD.io, Examples: OPNFV NFVI software. PNDA and OpenStack. scenarios, MEF OpenLSO reference platform, ONAP.

Components Platforms Open Systems

• FD.io/VPP - the industry’s • Ligato – a platform for cloud • Cloud native networking – highest performing and most native networking – service Integration of FD.io/VPP, Contiv- versatile software forwarder chaining VPP, K8s, Ligato • Contiv-VPP – container • Kubernetes – Container cluster (demonstrated at ONS 2018, networking plugin for VPP management ships with CCP), Network Service • IPv6 for Kubernetes – enhance • OpenDaylight – Swiss-army Mesh (NSM) (demonstrated at K8s with IPv6 support knife network controller platform Kubecon NA 2018) • Istio/Envoy – service mesh • OpenStack – Virtualization • High-speed cloud networking – Multi-cluster support for management Integration of FD.io/VPP, Kubernetes • PNDA – platform for network Networking-VPP, OpenStack • Calipso (in OPNFV) – Real time data analytics (“FastDataStacks”: Shipping as operations support/monitoring • Network Service Mesh (NSM) – part of OPNFV releases and for Kubernetes and OpenStack a platform for doing Service Cisco’s NFVI solution) • NFVbench (in OPNFV) NFVI Mesh for L2/L3 payloads – think • Versatile network analytics – benchmarking SFC for the Cloud-native world. Integration of PNDA.io with ONAP DCAE (in development)

Open Commercial Components Platforms Systems Offering

Products and Test, harden, Build foundational Build foundational solutions which Equipment Suppliers sub-system focused Components Platforms integrate OSS integration components Services for Evolve System composition enhanced/hardened Systems Integrators foundational from components and OSS systems platforms platforms or platforms Create reference Services on-top Users/Operators system/architecture of a system composed to reduce integration of OSS and and test effort commercial products

#CLUS BRKSDN-2262 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 10 Integration: Open Source Projects Fuel Products Examples from Cisco Open Components Platforms Products & Solutions Systems

• ACI (future release) • VPP • Ligato • VTS • CNBR • XRv 9000, ..

• Contiv-VPP • Ligato • Cisco Container Platform • VPP • Kubernetes (CCP) • Istio/Envoy • Network Service Mesh

• VPP • OpenStack • OPNFV • Cisco VIM FastDataStacks Cisco leverages OSS components to build products and/or offer associated services Cisco participates in OSS system level integration to test/harden associated platforms/solutions

Orchestration

SDN Controllers

Analytics Routing Control/Management Plane Stacks

Data-Plane Test & Integration

• Seven incorporated projects including fd.io

Zebra ONAP Orchestration OSM Quagga OpenDaylight bird SDN Controllers PNDA ONOS Routing Analytics Stacks Routing Honeycomb Sweetcomb ExaBGP Control/Management Plane Stacks Networking-VPP Ligato

GoBGP OpenFlow OVS OPNFV Data-Plane Test & Integration P4 VPP

Design and Orchestration LFN

Data Analytics LFN

Jaeger

Monitoring, Logging, Tracing Tracing

LFN CNCF

Virtualization Control

CNCF Cloud Cloud Infra & Tooling

Service Overlays NSM CNCF

Network & Connectivity Ligato Contiv VPP LFN

Operating Systems

Software Data Plane

LFN Infrastructure Hardware

Integrate/Deploy/Test & Tooling

LFN CI/CD

OSS Decomposition Planning (ONAP D-release):

Design and Orchestration LFN cn-VNFs, Service Mesh Planning: Data Analytics LFN cn-PNDA, PNDA open community

Jaeger

Monitoring, Logging, Tracing Tracing

LFN CNCF

Virtualization Control

CNCF Cloud Cloud Infra & Tooling

Service Overlays NSM CNCF

Ligato Network & Connectivity Network Contiv VPP LFN ServiceMesh

Operating Systems

Software Data Plane

LFN Infrastructure Hardware

Integrate/Deploy/Test & Tooling Already evolving: CI/CD, Tools, LFN CI/CD CN-solution stacks – CN & Edge

Project Description

Hyperledger Business Blockchain Framework & Tool - https://www.hyperledger.org/

Open Compute Project (OCP) Shared hardware designs of data center products - https://www.opencompute.org/

Open nFAPI Implementation of the Small Cell Forum's network functional API (nFAPI); https://github.com/cisco/open-nFAPI

VMI-Linux Virtualization interface for cross hypervisor compatibility; https://sourceforge.net/projects/vmi-linux/

Thor Video Codec Implementation of draft-fuldseth-netvc-thor; https://github.com/cisco/thor

Anaconda Distribution of the Python and R programming languages for data science and machine learning related applications https://github.com/Anaconda-Platform

Scikit Scikit-learn is a Python module for machine learning built on top of SciPy; https://github.com/scikit-learn/scikit-learn

Joy Capture and analyze network flow data and intraflow data, for network research, forensics, and security monitoring https://github.com/cisco/joy

Kubeflow Cloud Native platform for machine learning; https://github.com/kubeflow/kubeflow

Calipso Monitoring and analysis of VM/Container networking (OpenStack/Kubernetes) http://calipso.io/, https://git.opnfv.org/calipso/tree/

Pravega Stream as a new storage abstraction; http://pravega.io/index.html

Scikit-learn Machine learning in Python; tools for data mining and data analysis; http://scikit-learn.org/stable/

Magen Platform that integrates encrypted file sharing, identity management and policy enforcement. https://github.com/magengit

OpenNARS General-purpose AI system, designed in the framework of a reasoning system; https://github.com/opennars/opennars

PaaS/Application Layer` < empty > OSS defines Orchestration`` < empty >

Network Data Analytics SG12 E.INADF OSS leads, SDOs complement Service Models • TOSCA

NFV Architecture • NFV ISG

Network Telemetry • IPFIX SDOs lead, • BGP Monitoring Protocol OSS for rapid GTM Network Models YANG • YANG models catalog

Network • Segment Routing • In-Situ/In-Band OAM

Claiming standardization responsibility for technology that has already been “defacto” standardized by an OSS community.

• OSS as strategic market and tech development tool by large organizations is unstoppable • Interdependency requires SDOs to develop competencies, cultures and communities and outreach of their own • Liaison Mechanism Failure

#CLUS BRKSDN-2262 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 22 OpenDaylight and ONAP #CLUS BRKSDN-2262 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 24 ONAP - Architecture

• Project at Linux Foundation fd.io Scope: • Multi-party • Network IO - NIC/vNIC <-> cores/threads • Multi-project • Packet Processing (Classify, Transform, Prioritize, Forward, Terminate) • Software Dataplane • Dataplane Management Agents (Control and • Terabit-class throughput Management Plane) • Low Latency • Feature Rich Bare Metal/VM/Container • Resource Efficient Dataplane Management Agent • Bare Metal/VM/Container • Multi-platform Packet Processing

Network IO

Application Layer/App Server

Orchestration

Network Controller

Data Plane Services Dataplane Packet Network IO Management Agent Processing

Operating System

Hardware

Qiniu Yandex

JVPP Dataplane Management Agent GoVPP Testing/Support Honeycomb hc2vpp Sweetcomb VPP Sandbox

P4vpp Packet Processing DMM CSIT NSH SFC ONE TLDK puppet-fdio Cicn HICN ODP4VPP TRex

VPP ci-management

Network IO Pma tools deb_dpdk rpm_dpdk

Bare-metal / VM / Container Packet Processing Software Platform • High performance Dataplane Management Agent • Linux user space • Runs on compute CPUs: Packet Processing - And “knows” how to run them well !

Shipping at volume in server & embedded Network IO products

Packet processing is decomposed … packets move through … graph nodes are optimized 1 2 3 into a directed graph of nodes … graph nodes in vector … to fit inside the instruction cache … Packet 0 vhost-user- af-packet- dpdk-input input input Packet 1 Microprocessor ethernet- Packet 2 input Packet 3 3 Instruction Cache Packet 4 arp-inputcdp-input l2-input ip4-input ip6-input lldp-input ...-no- mpls-input Packet 5 checksum Packet 6 4 Data Cache ip4-lookup- ip4-lookup* mulitcast Packet 7

Packet 8 mpls-policy- ip4-load- ip4-rewrite- ip4- encap balance transit midchain Packet 9 … packets are pre-fetched 4 Packet 10 into the data cache. interface- output

* Each graph node implements a “micro-NF”, a “micro-NetworkFunction” processing packets. Makes use of modern Intel® Xeon® Processor micro-architectures. Instruction cache & data cache always hot  Minimized memory latency and usage.

fd.io VPP 0 1 2 3 …n Vector Packet Processing Vector of n packets

dpdk-input vhost-user-input af-packet-input … Packet Processing Graph Input Graph Node ethernet-input Graph Node

ip6-input ip4-input mpls-input … arp-input

ip6-lookup ip4-lookup ip6-rewrite ip6-local ip4-local ip4-rewrite

fd.io VPP 0 1 2 3 …n Splitting the Vector Vector of n packets

dpdk-input vhost-user-input af-packet-input … Packet Processing Graph Input Graph Node ethernet-input Graph Node

ip6-input ip4-input mpls-input … arp-input

ip6-lookup ip4-lookup ip6-rewrite ip6-local ip4-local ip4-rewrite

fd.io VPP 0 1 2 3 …n Plugins Vector of n packets Hardware Plugin

hw-accel-input dpdk-input vhost-user-input af-packet-input … Packet Processing Graph Input Graph Node ethernet-input Graph Node Skip s/w nodes where work is ip6-input ip4-input mpls-input arp-input done by … Plugins are: hardware Plugin First class citizens /usr/lib/vpp_plugins/foo.so already That can: ip4-lookup Add graph nodes ip6-lookup custom-1 Add API Rearrange graph ip6-rewrite ip6-local ip4-local ip4-rewrite custom-2 Can be built independently of VPP source tree #CLUS BRKSDN-2262 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 36 Using VPP with VMs and Containers VPP with VMs and Containers

Application Virtualized Network Connectivity Functions

Honeycomb Networking-vpp VMs Sweetcomb

Ligato Containers Contiv-VPP Network Service Mesh

Neutron Node ML2Server VPP journaling Mechanism Driver

VM VM VM HTTP/json VM VM VM

vhostuser vhostuser VPP VPP

dpdk VPP Agent dpdk VPPAgent Node Node vlan / flat network

• HC core functionality is split into 2 layers:

1) Data processing layer • Pipeline processing data from northbound interfaces down to translation layer

2) Translation layer • Invoked by above layer to handle configuration updates or when polling operational state from VPP • Specific translation code lives in this layer in a form of extensions/plugins

#CLUS BRKSDN-2262 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 40 • Can deliver complete container networking solution entirely from userspace • Legacy apps can still use the kernel host stack in Contiv-VPP Architecture the same architecture • Replace all eth/kernel interfaces with memif/userspace interfaces. • Apps can add VCL library for Higher Performance (bypass Kernel host stack and use VPP TCP stack) K8s Master

High Performance Legacy Cloud- K8s State Cloud- Legacy High Performance Apps Apps Native VNFs Reflector Native VNFs Apps Apps

PodPod PodPod PodPod PodPod PodPod PodPod Pod Pod Pod Kubelet Kubelet Pod Pod Pod Envoy Sidecar App App VNF Contiv-VPP VNF App App Envoy Sidecar Etcd

VPP VPP Kernel Host stack CNI K8s policy & state CNI Kernel Host stack TCP TCP memif distribution memif Stack tapv2/veth tapv2/veth Stack

VPP Agent Agent VPP Contiv-VPP vswitch … Contiv-VPP vswitch IPv4/IPv6/SRv6 Network

Load-Balancing between PODs defined using POD label selectors

Rendered into NAT config on VPP (instead of using kube-proxy)

MyApp

VPP Agent 9376

• K8s Service Processor • matches k8s service metadata (k8s API) with endpoints (labels to PODs)

• K8s Service Renderers • render k8s service data into VPP & Linux (POD) configuration • Different renderers for different use-cases: • NAT44 Renderer (IPv4) • IPv6-route Renderer (IPv6) VPP-IPv6 VPP-SRv6 • SRv6 Renderer (IPv6 - future)

Node 1 Kernel App App Node 3

Kernel App App Kernel App App tap-v2 BVI BD BVI BVI BD VXLAN Mesh Single VNI BD Cloud (Overlay)Network

Data Plane Network

• Kubernetes does not provide a way to stitch micro-services together today • Ligato allows you to wire the data plane together into a service topology Define Define Define • Network functions can now become part of the service topology Placement Services Topology

K8s Master

High Performance Legacy Cloud- Apps Apps Native VNFs K8S State Ligato PodPod PodPod AgentPodPod Reflector Controller Pod Pod Pod Envoy Sidecar App App VNF Kubelet

VPP Kernel Host stack CNI TCP Contiv-VPP Etcd CRI Stack tapv2/veth memif

VPP Agent

Contiv-VPP vswitch

IPv4/IPv6/SRv6 Network

#CLUS BRKSDN-2262 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 45 Service Function Chaining with Ligato Logical Representation Ingress Network Egress Network Ingress NF NF NF Egress Ingress Classifier Router 1 2 3 Router Egress Classifier

Placement (K8s) Rendering Topology

Physical Representation

CNF1 CNF2 CNF CNF CNF3 CNF VPP VPP … VPP VPP VPP … VPP

VPP vSwitch VPP vSwitch 10.1.0.127

Overlay Tunnel Overlay Tunnel Overlay Tunnel

Ingress Classifier Egress Classifier

Direct East/West Memif Memif via vSwitch

CNF1 CNF2 CNF1 CNF2 memif memif policy vSwitch vSwitch

Node 1 Node 2

CNF1 CNF2

memif memif VXLAN Tunnel vSwitch policy policy vSwitch Dedicated VNI

Data Plane Network

Service Policy Service Topology Lifecycle Production-Grade Container Orchestration Production-Grade Container Orchestration

Kubernetes API Proxies

Performance-Centric Network Function and Network Topology Orchestration Contiv Container Networking Container Networking SFC Controller Calico K8s State Reflector

Cloud-native NF Orchestration Containerized Network Data Plane LIGATO Cloud-native NF Agent platform Networking Plugin Kubelet Calico Contiv-VPP Etcd

CNI

Containerized Fast Agent Agent Agent Agent

FD.io VPP Container Data Input/Output CNF Container Switch Network Function CNF

Contiv-VPP Ligato SFC Controller

Ligato VPP Agent GoVPP

VPP Data-Plane

DPDK NIC Driver

• SRv6 in Contiv-VPP and Ligato

• Network Service Mesh with VPP Data-Plane

• Native AES-NI Crypto in VPP • Current solution leverages DPDK drivers and is tied to IPsec

• Using VPP to create new CNFs for Ligato/NSM

• For IPv6 applications assign the cluster IP to the service pod • Just an additional “off-link” /128 address assigned to the interface • This is really just an anycast address

• Hide that cluster IP when crossing the fabric • SRv6 enables us to do this using a SID list (e.g. “Node” -> “Pod” -> “Service”) • Note that there’s no requirement for SRv6 in the fabric or pods – only the vSwitch

• Key requirement is flow-based forwarding to ensure stickiness • Scales nicely in VPP

• Potential to use SRLB (more intelligent than round-robin LB in K8s today)

• … and NAT44 is avoided

k8s-master k8s-worker1 k8s-worker2 k8s-worker3

> curl .. nginx 2001:0:0:2::5

2001:fd00::2 2001:fd00::10 2001:fd00::11 2001:fd00::12

kubectl get pods –o wide

kubectl expose deployment nginx --type=ClusterIP --port=80

kubectl get services –o wide

curl [2096::9695]

k8s-master k8s-worker1 k8s-worker2 k8s-worker3

> curl .. nginx nginx 2001:0:0:2::5 2001:0:0:4::6 2096::9695 2096::9695

2001:fd00::2 2001:fd00::10 2001:fd00::11 2001:fd00::12

nginx nginx 2001:0:0:2::5 2001:0:0:4::6 2096::9695 2096::9695

alpine 2001:0:0:4::7 > wget .. 2001:fd00::2 2001:fd00::10 2001:fd00::11 2001:fd00::12

https://github.com/contiv/vpp/blob/master/docs/setup/SRV6.md

k8s-master k8s-worker1 k8s-worker3

nginx nginx Host Host > curl [2096::9695] 2001:0:0:2::5 2001:0:0:4::6 2096::9695 2096::9695

VPP VPP VPP

Table LocalSID-DX6 Table LocalSID-DX6 Lookup 6666:/16 Lookup 6666:/16 Steering

PodVRF PodVRF

Table LocalSID-DX6 Table LocalSID-DX6 LocalSID-DX6 Lookup 6655:/16 Lookup 6655:/16 6655:/16

Policy Table LocalSID-End LocalSID-End 5555:/16 7766:/16 7766:/16 Lookup MainVRF MainVRF MainVRF

nginx nginx Host Host > curl [2096::9695] 2001:0:0:2::5 2001:0:0:4::6 2096::9695 2096::9695

VPP VPP VPP

Table LocalSID-DX6 Table LocalSID-DX6 Lookup 6666:/16 Lookup 6666:/16 Steering

PodVRF PodVRF

Table LocalSID-DX6 Table LocalSID-DX6 LocalSID-DX6 Lookup 6655:/16 Lookup 6655:/16 6655:/16

Policy Table LocalSID-End LocalSID-End 5555:/16 7766:/16 7766:/16 Lookup MainVRF MainVRF MainVRF

k8s-master k8s-worker1 k8s-worker3

nginx vppctl sh sr policiesnginx Host Host > curl [2096::9695] 2001:0:0:2::5 2001:0:0:4::6 2096::9695 2096::9695

VPP VPP VPP

Table LocalSID-DX6 Table LocalSID-DX6 Lookup 6666:/16 Lookup 6666:/16 Steering

PodVRF PodVRF

Table LocalSID-DX6 Table LocalSID-DX6 LocalSID-DX6 Lookup 6655:/16 Lookup 6655:/16 6655:/16

Policy Table LocalSID-End LocalSID-End 5555:/16 7766:/16 7766:/16 Lookup MainVRF MainVRF MainVRF

vppctl sh sr localsids

k8s-worker1

nginx Host 2001:0:0:2::5 2096::9695

VPP

Table LocalSID-DX6 Lookup 6666:/16

PodVRF

Table LocalSID-DX6 Lookup 6655:/16

LocalSID-End 7766:/16 MainVRF

• Use SRv6 instead of VXLAN for inter-server rendering • E.g. SID list is “Node -> Pod -> CNF”

• Or how about using SRv6 to specify the entire chain from ingress? • So SID list might be “Node-1 -> Pod-A -> CNF-X -> Node-2 -> Pod-B -> CNF-Y….”

• SRv6 also enables traffic engineering within the data-center • And fast protection in the data-plane

• SRv6 also enables integration of the SP WAN and the data-center • Can do this with MPLS-SR also • But often today would be integrating EVPN MPLS with EVPN VXLAN

Implementation Focused Developer Focused

Cloud 1.0 Cloud-native

#CLUS BRKSDN-2262 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 67 Two Ways to Abstract Networking Problems Implementation Focused Developer Focused - Previous implementation was: What does the Developer really want? - Interface - Subnets Add to the k8s API: - Pod protecting from THREATS - vInterfaces - vSubnets CONNECTIVITY to ISOLATED Resources Subnet

Pod vint2 int k8s 2 Allow POD to TALK to RADIO Guaranteed Network

LATENCY/BANDWIDTH

L2/L3 connection L2/L3 k8s int k8s

vint1 Something that functionally does the thing needed when Subnet sending packets

1 LOAD Balancing CONNECTIVITY TO corporate Intranet Connect to another CNF

Guaranteed Latency/Bandwidth Load Balancing Connectivity to isolated Resources

The Service a developer may want for their L2/L3 traffic

Protection from Proxying Threats

Interface/Subnet/Network to are implementation details What matters is the *Services* your L2/L3 payloads should receive

#CLUS BRKSDN-2262 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 69 Network Service Mesh kind: NetworkService apiVersion: V1 metadata: name: secure-intranet-connectivity spec: payload: IP matches: - match: sourceSelector: app:firewall route: - destination: Network Interconnect destinationSelector: App:vpn-gateway between PODs defined - match: route: using POD label selectors - destination: destinationSelector: app:firewall Rendered into: Network Service - additional interfaces in secure-intranet-connectivity PODs - forwarding rules between app=firewall the additional interfaces Client Firewall VPN Gateway L2/L3 connection L2/L3 connection Pod Pod Pod on VPP (xConnect + app=firewall app=vpn-gateway VXLANs / SRv6)

Node1 Node2

Sarah’s 4. Req Con 3. Req Con VPN Pod Gateway 1. Req Con NSMgr1 NSMgr2 5. Reply Pod (NSE) NSM 8. Reply InitContainer 11. Reply

9.Create & Inject Interface 6.Create & Inject Interface 10.Create tunnel 7.Create tunnel

Dataplane (kernel/vswitch) Dataplane (kernel/vswitch)

Kubecon NA 2019: Intro Kubecon NA 2019: Deep Dive

Slides Video Slides Video

Contiv-VPP Ligato SFC Controller NSM Manager

Ligato VPP Agent GoVPP

VPP Data-Plane

DPDK NIC Driver

• Firewall • Get the Code, Build the Code, Run the Code

• IDS • Try the vpp user demo

• Hardware Accelerators • Install vpp from binary packages • Control plane – support your (yum/apt) favorite SDN Protocol Agent • Install Honeycomb from binary packages • Spanning Tree • Read/Watch the Tutorials • DPI • Join the Mailing Lists • Test tools • Join the IRC Channels

• Explore the wiki

• Join FD.io as a member

#CLUS BRKSDN-2262 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 76 Complete your online session • Please complete your session survey after each session. Your feedback evaluation is very important.

• Complete a minimum of 4 session surveys and the Overall Conference survey (starting on Thursday) to receive your Cisco Live water bottle.

• All surveys can be taken in the Cisco Live Mobile App or by logging in to the Session Catalog on ciscolive.cisco.com/us.

Cisco Live sessions will be available for viewing on demand after the event at ciscolive.cisco.com.

Demos in the Walk-in labs Cisco campus

Meet the engineer Related sessions 1:1 meetings

#CLUS #CLUS