Network Function Virtualization Seminar

Matthias Falkner, Distinguished Engineer, Technical Marketing

Nikolai Pitaev, Technical Marketing Engineer,

TECSPG-2300 Cisco Webex Teams

Questions? Use Cisco Webex Teams to chat with the speaker after the session How 1 Find this session in the Cisco Events Mobile App 2 Click “Join the Discussion” 3 Install Webex Teams or go directly to the team space 4 Enter messages/questions in the team space

• Introduction • 08:45 – 09:05 Matt • NFV Primer • 09:05 – 10:00 Matt • Virtualizing Branch Infrastructure • 10:00 – 10:45 Nikolai Break • SP/Cloud Virtualization • 11:00 – 11:30 Nikolai • Connecting to Multiple Clouds • 11:30 – 12:15 Nikolai • Multi-Tenanted SMB Services • 12:15 – 12:50 Matt • Conclusion • 12:50 – 13:00 Matt

• Idea of de-coupling software from hardware is not new! • Linked to automation / orchestration

• Increased focus to simplify Enterprise

architectures https://www.dataports.eu/wp-content/uploads/google-datacenter- eemshaven-img-7.jpg • Particularly on L4-7 services

• SPs drive adoption, but Enterprises are following suit

https://bloximages.newyork1.vip.townnews.com/omaha.com/content • Both consumption models (MSP, self-managed) /tncms/assets/v3/editorial/1/ab/1ab55a42-195a-11e7-b177- considered 3f34b38ab18c/58e3d482ba843.image.jpg?resize=1200%2C673

https://cnet3.cbsistatic.com/img/8cRqI3rcyHCpNORJVjRkeTVUoLM=/724x407/2013/1 0/25/d451bda3-3f9b-11e3-a363-14feb5ca9861/Structure_from_Yerba_Buena.jpg

OPEX CAPEX • Deployment Flexibility • Deploy on standard x86 servers • Reduction of number of network elements • Economies of scale • Reduction of on-site visits • Service Elasticity – deploy as needed • Deployment of standard on-premise hardware • Simpler architectural paradigm • Simplification of physical network architecture • HA still needed? • Leveraging Virtualization benefits • Best-of-breed • Hardware oversubscription, vMotion, .. • Increased potential for automated network operations • Re-alignment of organizational boundaries

4 Automation / Orchestration (Cisco DNA Center, NSO)

Virtual WAN Virtual Router Virtual Firewall Virtual Wireless LAN Optimization 3rd Party VNFs (ISRv,CSR) (ASAv, NGFWv) Controller (vWLC) (vWAAS)

2 Network Functions Virtualization Infrastructure Software (NFVIS)

ISR 4000 + CSP-5444 / Enterprise Network Compute 1 UCS E-Series UCS C-Series System

It is all about Virtual Network Functions. We are not talking about generic Virtualization Techniques.

Topics, which will be covered next:

• IO: SR-IOV, Virtual Switches, Service Chaining

• CPU: Hyperthreading, vCPU pinning, NUMA Socket Allocation

• Putting all together: NFV Performance Insights

• VNF Virtualization vs. Containerization

VM1(4vCPU CSR 1000v) CSR VMFman2(1vCPU/ CSR 1000v)HQF

IOS PPE PPE Rx CSR 1 1 1 1 CMan Pkt Scheduler IRQ vNIC vNIC VM Linux Fman3 / 1 n VM (2vCPUPPE CSRHQF 1000v)

IOS Rx CSR CMan Pkt Scheduler IRQ2 vNIC 2 vNIC 2 VM Linux2 • Example: 3 CSR VMs Guest OSFman Scheduler/ 1 n PPE HQF IOS Pkt Scheduler Rx 3 3 3 3 scheduled on a 2-socket 8- Guest CManOS Scheduler IRQ vNIC1 vNICn VM Linux 1 vCPU 1 vCPU 1 1 vCPU0 1 2 vCPU3 core x86 Guest OS2 Scheduler vCPU0 – Different CSR footprints shown 3 3 vCPU0 vCPU1 • Type 1 Hypervisor vSwitch – No additional Host OS X86 Server vCPU 2 1 represented vCPU 3 0 Host Linux 2

vNICn Process Process Queue VM Kernel1 • HV Scheduler algorithm governs how HV Scheduler vCPU/IRQ/vNIC/VMKernel processes are allocated to Socket Socket 0 1 pCPUs pCPU0 pCPU1 pCPU2 pCPU3 pCPU0 pCPU1 pCPU2 pCPU3

pCPU4 pCPU5 pCPU6 pCPU7 pCPU4 pCPU5 pCPU6 pCPU7 • Note the various schedulers I/O I/O Memory Storage Memory I/O I/O – Running ships-in-the-night

Packet feature x86 Host (by example of FD.io VPP) processing

VM1 VMn User Guest Packet moved to CSR 1000v .. CSR 1000v VNF DPDK-VirtIO DPDK-VirtIO Ptr Ptr VNF interrupted, Ptr Ptr Ptr Ptr Why does this matter?

packet pointer

Kernel Guest passed to buffer • Illustrates contention of shared resources

Shared Pkt Mem User Host • Each packet move

vNIC vNIC / vHost-user notified Pkt Pkt Pkt Pkt (vHost_user) Qemu consumes resources (vHost_user) Pkt Pkt Pkt Pkt • Packet pointer buffers FD.io VPP

VPP kicked, have limited depth Kernel switching packet Host • can cause drops

Ptr pNIC Driver pNIC

Packet copied into Pkt Pkt Pkt Pkt Memory Pkt Pkt Pkt Pkt Pkt Pkt Pkt Pkt Packet Arrival

X86 Host (w/ OVS-DPDK, FD-IO/VPP)

VM Guest 1 Application VM2 Application Intra-VM Processing

(e.g. Features) IO Driver IO Driver User User Space

Hypervisor / (QEMU)

Virtualization Layer vNIC vNIC

Virtual Switch / HostKernel IO-Path

Virtual Switch pNIC Physical Interfaces pNIC pNIC Driver pNIC Driver Pkt Pkt Pkt Pkt Pkt Pkt Pkt Pkt

VM1(4vCPU CSR 1000v)

VMFm2(a1nvC/ PU CSR 100H0QvF) S

IOS PPE PPE Rx C 1 1 1 1 • CMan Pkt Scheduler IRQ vNIC1 vNICn VM Linux VNF can be associated with multiple S VMF3m(a2nv/C PU CSRH Q1F000v)

IOS PPE Rx C • VNF can be assoPkt Schecduleirated with m2 ultip2le 2 2 CMan IRQ vNIC1 vNICn VM Linux Guest OSF mSacnh/ eduler HQF S vCPUs R vCPUIOsS PPE Rx 3 3 3 3 Guest COMSan ScheduPlkteScrheduler IRQ vNIC1 vNICn VM Linux 1 vCPU 1 vCPU 1 1 vCPU0 1 2 vCPU3 • … and cGounsesut OmS2e Smcheemduoleryr • … and consume memory vCPU0 3 3 vCPU vCPU1 • VNF Softw0 are architecture can impact performance • vSwitch

VNF Software architecture can impact 2

X86 Server vCPU1

s s e 3

e vCPU0 u

c Host Linux e

performance o 2 u

r vNICn P CSR Resource Template Q VM Kernel1

*Available in 3.16.02 and later • Example: CSR1000V vCPU allocations HV Scheduler Default (Data Plane Heavy) Control Plane Heavy vCPUs 1 2 4 8 vCPUs 1 2 4 8 Socket0 Socket1 Control Control pCPU0 pCPU1 pCPU2 pCPU3 pCPU0 pCPU1 pCPU2 pCPU3 1 1 1 1 2 2 Service 1 Service 1 pCPU4 pCPU5 pCPU6 pCPU7 pCPU4 pCPU5 pCPU6 pCPU7 Data 1 3 7 Data 1 2 6 I/O I/O Memory Storage Memory I/O I/O

Service Plane Medium Service Plane Heavy vCPUs 1 2 4 8 vCPUs 1 2 4 8 Control Control 1 2 2 1 2 4 Service 1 Service 1 Data 1 2 6 Data 1 2 4

• NUMA is a memory sharing

technology to allows a processor VPP1 VM1 VPP2 VM2

core to use memory associated with Socket 0 Socket 1

other cores NUMA Node 0 NUMA Node 1

Core0 Core1 Core2 Core3 Core4 Core4 Core6 Core7 • Accessing remote memory happens

over the NUMA connection, which is Core0 L1 Core1 L1 Core2 L1 Data Core3 L1 Core4 L1 Data Core5 L1 Core6 L1 Data Core7 L1 typically slower than local memory Data Cache Data Cache Cache Data Cache Cache Data Cache Cache Data Cache Core0-1 Core2-3 Core4-5 Core6-7 access L1 Instruction Cache L1 Instruction Cache L1 Instruction Cache L1 Instruction Cache

Core0-1 Core2-3 Core3-4 Core6-7 L2 Cache L2 Cache L2 Cache L2 Cache • Benefits: each core can access its own Core1-3 Core4-7 memory -> allows for simultaneous L3 Cache (last-level cache) L3 Cache (last-level cache)

Core0-3 Memory NUMA Node 0 NUMA Node 1 Core4-7 Memory memory access controller (shared) Interconnect (shared) Interconnect (shared) controller (shared)

• Performance implications Mem-VPP1 NUMA Node 0 External Memory Mem-VPP2 Mem-VM1 NUMA Node 1 External Memory Mem-VM2 • Higher-latency for memory access

• Variable performance

• Application memory may not be local to the core

ESXi / SR-IOV/ Single Feature / IMIX

18000 16000 14000 12000 10000 8000 6000 4000 Throughput (Mbps) Throughput 2000 0 IPSec (Single CEF ACL NAT L4 FW Basic QoS AES) 1 vCPU 6546 4656 781 3448 3741 5342 2 vCPU 7093 5093 843 3516 3794 6250 4 vCPU 8606 7075 1218 3844 3787 7590 8 vCPU 15624 14494 2312 4546 4547 15396 Traffic Profile : IMIX {64 byes (58.33%), 594 bytes (33.33%), 1518 bytes (8.33%)} PDR(Packet Drop Rate): 0.01% *The max throughput license we offer today is 10Gbps and please contact us if you have use case requires more than 10G © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public CSR1000V Performance Polaris 16.10.01b KVM/SR-IOV

KVM-REHL / SR-IOV/ Single Feature / IMIX

18000 16000 14000 12000 10000 8000 6000 4000 Throughput (Mbps) Throughput 2000 0 IPSec (Single CEF ACL NAT L4 FW Basic QoS AES) 1 vCPU 3812 4312 750 3302 3575 4753 2 vCPU 7148 3624 843 3536 3813 6304 4 vCPU 8643 6911 1218 3781 3673 7786 8 vCPU 16718 15023 2405 7276 7120 14740 Traffic Profile : IMIX {64 byes (58.33%), 594 bytes (33.33%), 1518 bytes (8.33%)} PDR(Packet Drop Rate): 0.01% *The max throughput license we offer today is 10Gbps and please contact us if you have usecase requires more than 10G © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public I/O Architecture IO Technologies: OVS, VPP, PCIe Pass-Through, SR-IOV

2 copy operations: 1 copy operation: Direct Memory pNIC→OVS & OVS→VM pNIC→DPDK Access from VM

X86 Host (w/ SR-IOV)

VM Guest Many VMs to 1 pNIC Mapping 1 Application VM2 Application

Enhancement to PCIe: IO Driver: VirtIO IO Driver: VirtIO

• VM Hypervisor Bypass User User Space

• Direct I/O Resources (QEMU) • Reduced CPU Utilization

• Reduced System Latency HostKernel • Increased I/O Throughput Intel SR-IOV NIC Driver required in VM PF VM to VM bridged in NIC hardware (VEB) or Driver

pushed to switch (VEPA) pNIC PF VF VF Up to 63 VF Ports per pNIC Port (depends Pkt Pkt Pkt Pkt on HW) Pkt Pkt Pkt Pkt

• Set of data plane libraries and NIC drivers for fast packet processing.

• Initially developed by Intel, today - Open-source project.

• Caveats:

• NICs MUST be DPDK capable

• VNFs MUST have an DPDK driver

OVS-DPDK: fd.io VPP: - A DPDK implementation of OVS - An open source forwarder written - Works the same as OVS with flow exclusively for userspace packet based packet matching and forwarding forwarding on 5-tuple matches - Processes packets in batches (vectors) to use the CPU more Can be problematic with user-to- optimally internet flows - lots of users x lots of internet = Doesn’t care about flows; uses MAC lots of flows address table based forwarding

Assumption: 64 Byte Ethernet Layer-2 frames + 20B of preamble + inter-frame gap Line rate stream of packets on a 10GE interface: 14.88 million packets- per-second Simple math: each packet must be processed within 67.2 ns. This is your total budget per packet, to receive the packet, process, and transmit it. Main memory RAM is 70 nsec! You can’t use main memory access - when you need to access memory it’s too late. https://blogs.cisco.com/sp/a-bigger-helping-of-internet-please

OVS OVS-DPDK SR-IOV PCIe Pass- fd.io VPP through Flexibility Highest High Medium Low (Some feature (Feature limitations) (Limit on number of limitation) VNFs) Performance Lowest High Highest Highest

Physical interface to VNF 1:Many 1:Many 1:Many 1:1 Ratio I/O path Software Software Hardware Hardware (but vector packet processing) Security Open Open Hybrid Hardware

Hardware / Software Standard DPDK-enabled SR-IOV VNF support for requirements KVM drivers & pNIC enabled drivers pNIC drivers & pNIC

x86 Server with 2 NUMA sockets, 8 cores each = 16 cores total VPP or OVS-DPDK as vSwitch, NICs are mapped to worker / PMD threads 6 CSR 1000V VMs with 2vCPU each

Linux CSR CSR CSR Linux CSR4 CSR5 CSR 1 2 3 VPP 6 VPP VPP VPP worker1 worker2 vCPU vCPU vCPU vCPU Emul vCPU0 vCPU1 vCPU0 vCPU1 vCPU0 vCPU1 Emul 0 1 0 1 vCPU0 vCPU1

0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 CPU 0 CPU 1 CPU 2 CPU 3 CPU 4 CPU 5 CPU 6 CPU 7 CPU 0 CPU 1 CPU 2 CPU 3 CPU 4 CPU 5 CPU 6 CPU 7

Socket0 Socket1

Physical Physical Interface 1 Interface 2

Linux CSR CSR CSR CSR CSR CSR VPP 1 2 3 4 5 6 VPP VPP VPP worker1 worker2 main Emul vCPU0 vCPU1 vCPU0 vCPU1 vCPU0 vCPU1 vCPU0 vCPU1 vCPU0 vCPU1 vCPU0 vCPU1

0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 CPU 0 CPU 1 CPU 2 CPU 3 CPU 4 CPU 5 CPU 6 CPU 7 CPU 0 CPU 1 CPU 2 CPU 3 CPU 4 CPU 5 CPU 6 CPU 7

Socket0 Socket1

Physical Physical To improve: Interface 1 Interface 2 1. physical NIC – VPP mismatch 2. CSR3 – socket crossing “tax” 3. Emulator pin for VMs 4-6 on different socket

Number of DPDK worker threads can have positive impact on total system throughput if I/O path is the bottleneck. Placement of worker threads on sockets / NUMA nodes does matter! Balance the interface association to worker threads on the sockets System Throughput Effect of allocating different number of VPP Worker Threads (2vCPU, CEF, 0.01% PLR, IOS XE 16.3)

25 t

u 20

g u

o 15

h T

t s

y 5 S

0 1 2 3 4 5 1 Worker 6.31 5.286 4.395 4.047 3.979 2 Workers 6.13 8.365 8.199 8.919 9.077

Impact of Different server Core Speeds Two servers with: CSR 1000v, IMIX, SR-IOV, IOS XE 16.3 • 16 core @ 3.2 GHz • 24 core @ 2.6 GHz 7.367 SR-IOV with 2 x 10 GE Ports used 3.2 GHz, 16 core 20 CEF (IP forwarding) tested

6.001 2.6 GHz, 24 core For 1 VM, performance increase 18.101 proportional to the CPU Cycle 0 5 10 15 20 25 difference 3.2 7.4 ≈ linear! 1x2vCPU 3x2vCPU 2.6 6

For 3 VMs - not proportional

• IO-Limit – bottleneck switched from CPU to IO

Slope depends of feature set & packet Size

VM throughput Benchmark: ideal Benchmark 50 throughput that can be achieved with an 45

unconstrained I/O path ) s

p 35

(

u 30

h g

Derived from throughput of a single CSR u 25

h T

20 m

1000v with SR-IOV e

t s

y 15 S • SR-IOV used as best ‘ideal’ I/O path 10 5

• Hypervisor with a single VNF is not stressed 0 1 2 3 4 5 6 7 8 – only has to deal with one VNF Benchmark 5.819 11.638 17.457 23.276 29.095 34.914 40.733 46.552 Number of VNFs • Assuming perfect additivity for each VNF added • Under ideal conditions, each VNF adds the Measured Throughput from VM1 same throughput as the first VNF Multiplied by number Throughput for VM1 of VMs

Multi-PVP Throughput (Gbps) with various I/O IPv4 , IMIX, IOS XE 16.6, KVM, UCS C-Series, 2.3 GHz, 1vCPU CSR100V 25.00

) Physical Interface Limit s

p 20.00

G (

15.00

p h

g 10.00

o r

h 5.00 T

0.00 1 2 3 4 5 6 7 8 9 10 Best Case Additive 2.23 4.45 6.68 8.91 11.13 13.36 15.59 17.81 20.04 22.27 SR-IOV 2.23 4.47 6.72 8.93 11.19 13.44 15.64 17.93 20.00 20.00 FD.io VPP 2.23 4.47 6.72 7.97 6.52 6.33 5.61 5.70 5.53 5.27 OVS-DPDK 1.01 2.00 2.98 3.98 4.97 5.69 5.27 5.22 5.05 4.90 OVS 0.34 0.49 0.60 0.69 0.67 0.74 0.71 Number of Virtual Network Functions (PVP: in-VNF-out)

CEF results with SR-IOV Total System Throughput vs. Number of VNFs Variability due to features 2vCPU, multi-feature Set, IOS XE 16.3, 0.01% PLR

) s

IPv6 GRE p 35

b G

( 30 CEF

NAT, Firewall, u

p 25

h g

VRF-aware firewall u 20

o r

h 15 T Access Lists (for filtering NAT)

m 10 Multi-Feature Set

t s

y 5

100 VRFs S 0 1 2 3 4 5 6 7 8 9 10 11 VPP 0.67 1.309 1.921 2.491 3.05 3.67 4.265 4.836 5.44 6.03 SR-IOV 0.67 1.347 2.003 2.602 3.74 5.01 6.29 Benchmark 0.67 1.34 2.01 2.68 3.35 4.02 4.69 5.36 6.03 6.7 CEF Benchmark 7.367 14.734 22.101 29.468 36.835 44.202 51.569 58.936 66.303 73.67 81.037 CEF VPP 6.273 12.343 23.894

Multi-VM Throughput (Gbps) with various I/O architectures NAT+Firewall+QoS+DPI, 1500 B Packet Size, XE 16.3.1

Physical Interface Limit

)

(

t 15

o r

h 10

s y

S 5

0 1 2 3 4 5 6 7 8 9 10 Best case Additive 6.6 13.3 19.9 26.6 OVS-DPDK 4.3 9.4 15.2 15.5 12.6 12.1 10.0 10.0 9.1 8.1 SR-IOV 6.6 12.3 18.7 19.7 19.7 19.7 19.7 19.7 19.7 19.7 FD.io VPP 6.3 11.7 16.4 19.7 19.7 19.7 19.7 19.7 19.7 19.7 Number of Virtual Network Functions (VNF Virtual Machines)

• Use a Direct path I/O technology (SR-IOV w/ PCIe pass-through) with CPU tuning below! Otherwise the following configurations are recommended: Tuning Recommendation Details / Commands Tuning Hyperthreading Can be done in BIOS; Recommended to be OFF, but can use carefully. CPU Find I/O NUMA Node cat /sys/bus/pci/devices/0000:06:00.0/numa_node CPU/Memory Enable isolcpus run command “numactl -H” CPU Pin vCPUs ‘sudo virsh vcpupin test 0 6’ CPU Set CPU in performance Mode run /etc/init.d/ondemand stop. CPU Set Processor into pass-through virsh edit CPU add this line Enable / Disable IRQ Balance run “service irqbalance start” & “service irqbalance stop” NOTE: ONLY IF IRQ PINNING IS CPU DONE! NUMA-aware VM edit vm config by virsh edit . CPU 1 IRQ Pinning find specific nic interrupt number from /proc/interrupts. set affinity to other core than CPU pinned cpu than for CPU and vHost pinning CPU Speedstep Allows dynamic changes of pCPU clock speed; Turn Off (increases variability) CPU Turbo Boost Turn off in multi-VNF deployments CPU © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public For Your KVM Performance Tuning Recommendations (cont.) Reference Tuning Recommendation Details / Commands Tuning Pin vHost processes ‘sudo taskset -pc 4 ’, I/O Where is found using ‘ps -ef | grep vhost’ Change vnet txqueue length to 4000 Default tx queue length is 500 I/O ‘sudo ifconfig vnet1 txqueuelen 4000’ Turn off TSO, GSO, RSO, ‘ethtool -K vnet1 tso off gso off gro off’ I/O Physical NIC Configuration Change rx Interrupt coalescing to 100 for the 10G NICs I/O NOTE: these settings may impact the number of VMs that can be instantiated on a server / blade QEMU Queue Sizes Better absorption of packets on arrival; Ensure large enough (e.g. 1024x1024) I/O DisableNOTE: KSM Tuning steps are most impactfulecho 0 > /sys/kernel/mm/ for a small ksmnumber/run of VMs instantiated on a host. TuningLinux DisableimpactMemballoon diminishes with a large numberEdit “virsh ofedit VMs , find memballon in vm config file. Linux Please change as Disable ARP/IP Filtering sysctl -w net.bridge.bridge-nf-call-arptables=0 Linux sysctl -w net.bridge.bridge-nf-call-iptables=0 sysctl -w net.bridge.bridge-nf-call-ip6tables=0 Optional Linux Tuning sysctl -w net.core.netdev_max_backlog=20000 Linux sysctl -w net.core.netdev_budget=3000 sysctl -w net.core.wmem_max=12582912 sysctl -w net.core.rmem_max=12582912 service iptables stop ( if you don't want linux firewall) Hugepages Increase virtual memory / VNF; Recommended 2M Linux

TECSPG-2300 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 45 Additional Topics: - Service Chaining - Containers IO Technologies: Service Chaining VNFs Service Chaining in vSwitch Service Chaining in NIC Service Chaining in TOR (VEB) (VEPA)

• Flexible • Assign VNFs to the same • Assign VNFs to the same VLAN in the same NIC VLAN in TOR • Performance limited by virtual Switch • Feature Limitations? • Requires support in TOR (e.g. Broadcasts) Full statistics / traffic • Limited Statistics visibility in TOR

• High Performance TECSPG-2300 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 47 Different I/O Technologies mirror the PVP Behavior with Service Chaining (PVVP) Multi-PVVP Throughput (Gbps) with various I/O IPv4 , IMIX, IOS XE 16.6, KVM, UCS-C Series, 2.3 GHz, 1vCPU CSR 1000v

25.00

) s

p 20.00

(

t 15.00

u p

h Physical Interface Limit

g 10.00

o r

h 5.00 T

0.00 1 2 3 4 5 6 7 8 9 10 Best Case Additive 2.25 4.49 6.74 8.98 11.23 13.48 15.72 17.97 20.21 22.46 SR-IOV 2.25 4.49 6.72 8.98 10.66 10.98 10.58 10.98 10.73 10.82 FD.io VPP 2.25 3.91 3.46 2.77 2.79 2.93 2.89 2.83 2.81 2.81 OVS-DPDK 1.01 1.88 2.80 3.55 3.36 3.28 3.23 3.21 3.13 2.77 OVS 0.34 0.50 0.63 Number of Service Chainss (PVVP: in-VNF1-VNF2-out)

TECSPG-2300 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 48 With SR-IOV, VEB-based Service Chaining Performs Best Multi-SFC Throughput (Gbps) with SR-IOV VEB/VEPA/vSwitch, IPv4, IMIX, IOS XE 16.6, KVM, UCS-C Series, 2.3 GHz, 1vCPU CSR 1000V

) s

p 20

(

t 15

u p

h 10 Physical Interface Limit

u o

r 5

h T 0 1 2 3 4 5 6 7 8 9 10 SRIOV-PVP 2.23 4.47 6.72 8.93 11.19 13.44 15.64 17.93 20 20 SRIOV-VEB 2.23 4.47 6.72 8.93 10.66 10.98 10.58 10.98 10.73 10.82 SRIOV-KVM 2.06 3.26 4.3 5.46 6.54 7.5 9.16 10.4 10.02 10.49 SRIOV-VEPA 1.05 2.1 3.1 3.63 3.64 4.05 3.87 4.18 3.97 3.9 Number of Service Chains (PVVP: in-VNF1-VNF2-out)

VMs as we use them today: Containers as we use them today: KVM or ESXi-based Hypervisors All traffic via kernel, NIC owned by kernel X86 Host (w/ OVS-DPDK, FD-IO/VPP) Host VM1 Application VM2 G Application u

t IO Driver IO Driver

(

vNIC vNIC a

e Container Container

)

n Interface Interface

Virtual Switch e

l Kernel

pNIC Driver p pNIC Driver p

N N

I I

C C NIC Pkt Pkt Pkt Pkt Pkt Pkt Pkt Pkt

Host Multiple network options usable simultaneously memif • Shared memory packet paths (memif) DPDKContainer app DPDKContainer app • PCI pass through SRIOV devices to

user space forwarders SRIOV

Interface Interface NIC • Conventional networking for control traffic Kernel NIC

Almost all current VNF use cases are VM-based. Few exceptions: SD-WAN Security, IOS XE Containers

• Cost of Virtualization solution as a function of performance CAPEX / OPEX • Trading-off performance for virtualization flexibility • Tuning performance may impact virtualization elasticity

• Architectural Considerations • Capacity planning Service Function Chains? • Orchestration solution? • High-Availability requirements? Architecture Performance

TECSPG-2300 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 52 Virtualizing Branch Infrastructure Branch Virtualization Option 1: service-rich on-prem Branch Email Web WAAS UC FW Sec. Sec. LAN

FROM x86 host

Branch Email Web WAAS UC FW Sec. Sec. LAN Enterprise Fabric

Option 2: cloud-native / thin branch Branch

LAN Enterprise Fabric

x86 host

Enterprise Fabric

• Automation is mandatory • Think 1000s of branches! • Incl. PnP / ZTP

• Very different environment to DC! • Cost sensitive • often no on-site support = Remote / cloud-based Day 0 to Day N operations

• Single service chain anchored on physical LAN and WAN interfaces • Modest requirement for VNF hosting

VM1(4vCPU CSR 1000v)

VMFm2(a1nvC/ PU CSR 100H0QvF) S

R IOS PPE PPE Rx C 1 1 1 1 CMan Pkt Scheduler IRQ vNIC1 vNICn VM Linux F3man / S Vendor A, B, C – VNFs VM (2vCPU CSRHQ 1F000v) PPE R IOS Pkt Scheduler Rx C 2 2 2 2 CMan IRQ vNIC1 vNICn VM Linux Guest OS Scheduler S Fman / HQF PPE R IOS Pkt Scheduler Rx 3 3 3 3 Guest COMSan Scheduler IRQ vNIC1 vNICn VM Linux 1 vCPU 1 vCPU 1 1 vCPU0 1 2 vCPU3 Guest OS2 Scheduler vCPU0

3 3 Vendor D – Hypervisor / OS vCPU0 vCPU1

vSwitch X86 Server 2

s vCPU1 s e 3 e vCPU

u 0

c Host Linux e

o 2 u vNIC

r n Vendor E – x86 Hardware P Q VM Kernel1

HV Scheduler

Socket0 Socket1 Vendor F - Automation pCPU0 pCPU1 pCPU2 pCPU3 pCPU0 pCPU1 pCPU2 pCPU3

pCPU4 pCPU5 pCPU6 pCPU7 pCPU4 pCPU5 pCPU6 pCPU7

I/O I/O Memory Storage Memory I/O I/O

Solution from one hand: VNFs + NFVIS + ENCS + Cisco DNA Center

ISRv ASAv/FTD * vWAAS vWLC

High Performance Full DC-class Featured Application Optimization Built for small and medium Rich Features Functionality and Akamai Connect branches

Windows Server Linux 3rd Party

Active Directory, Custom Applications Network Services File Share, DNS/DHCP Management & Monitoring Server Applications

Traditional Enterprise NFV Physical Router Physical Router Virtual Router Virtual Router Physical Switch Virtual Services Virtual Services Virtual Services

4000 Series ISR + Enterprise Network UCS C-Series UCS® E-Series Compute System (ENCS) CSP 5444 Cisco® 4000 Series ISR Cisco ® Catalyst 9000 Series

Centralized services Upgradable hardware Elastic routing and services Elastic routing and services Fixed integrated services Deterministic routing Performance Router / Server Hybrid Conservative performance Early adopter

Cisco ONE™ Access to Ongoing License Investment Innovation Portability Protection

ENCS5412 ENCS5408 12-Core ENCS5406 8-Core ENCS5104 6-Core 4-Core

ENCS 5104 ENCS 5406 ENCS 5408 ENCS 5412 CPU 4-core, 3.4 6-core, 1.9GHz 8-core, 2.0GHz 12-core, 1.5GHz GHz PoE No No 200W 200W Capacity Guidance ISRv + 1 VNF ISRv + 2 VNFs ISRv + 3 VNFs ISRv + 5 VNFs

5400 ENCS Platform Data Path Control Path VNF 1 ISRv VNF 2 (NIC aware) (NIC aware)

HW offload for Software VM-VM traffic switched path NFVIS 6 SR-IOV LAN Networks ® Internal NIC Cisco Lights-out IMC management (10G) High-speed backplane Switch

Cisco X86 VLAN-aware NIM IMC HW switch mgmt mgmt PoE

Cellular, T1, Dual-PHY Dedicated management ports DSL, LAN, GE WAN GE or LAN uplink

ENCS5400

wan-net lan-net mgmt-net

i F

w wan-br mgmt-br

N lan-br

S v

VF VF VF VF VF VF VF VF VF VF WAN WAN Mgmt LAN Backplane NIM NIC NIC GE0/0 GE0/1 MGMT CPU Default - DHCP for NFVIS and Default - Integrated Switch 192.168.1.1/24 VNFs connected GE1/0 GE1/1 GE1/2 GE1/3 GE1/4 GE1/5 GE1/6 GE1/7 to WAN-NET

NFVIS can be accessed by default via the FPGE WAN ports or via the dedicated Management port. WAN network (wan-net) and a WAN bridge (wan-br) is set by default to enable DHCP. GE0-0 is by default associated to WAN bridge. All Switch ports – GE 1/0 to GE1/7 is associated to LAN bridge. An internal management network (int-mgmt-net) and a bridge (int-mgmt-br) is created and is internally used for system monitoring. TECSPG-2300 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 61 Virtualization OS: NFVIS is optimized for VNF deployments!

Network Hypervisor Zero-Touch Deployment Monitoring

• Supports segmentation of • Automatic connection to PnP • Netconf Notification virtual networks server • Host and VM Statistics • Abstract CPU, memory, and • Highly secure connection to the storage resources orchestration system • Packet Capture • Easy day-0 provisioning

Lifecycle Management Service Chaining Open API

• Provisioning and launch of VNFs • Elastic service insertion • Programmable API for service orchestration • Failure and recovery monitoring • Multiple independent service • Stop and restart services paths based on applications or • Rest and NETCONF API user profiles • Dynamically add and remove services

GUI CLI Cisco DNA REST Center API NETCONF NSO

Network Functions Virtualization Infrastructure Software (NFVIS)

X86 Host

VNFs used: • vEdgeCloud for SD-WAN • ISRv LAN-facing

Cisco Validate Profile includes: • Deployment steps • Orchestration (Cisco DNA Center, vManage) • Voice T1-PRI connectivity to PSTN • Full Router Configurations https://www.cisco.com/c/en/us/solutions/collateral/desig n-zone/cisco-validated-profiles/cvp-c17-741660.html

Option 1: Active-Standby Option 2: Active-Standby • Identical VNFs provisioned on both platforms • VNFs provisioned only on 1 hardware • At a given time only 1 set of VNFs active • 2nd Hardware booted but not provisioned with any VNF • During failure, traffic switches over to the • During failure, VNFs are spun up over to the second second hardware platform hardware platform • VNFs States maintained • VNFs States maintained

ISRv WAAS ASAv Windows Linux

Hypervisor (KVM)

wan-net inet-net service-net lan-net mgmt-net

i F

w wan-br inet-br mgmt-br

N service-br lan-br

S v

VF WAN PF WAN 8-Port GE Switch NIM NIC NIC GE6 GE7 GE8 GE9 GE0 GE1 GE2 GE3 GE4 GE5

Option 3: Active-Active • Non-Identical VNFs provisioned and running on both platforms • During failure, ALL or SELECT VNFs failover to the second hardware • VNF states may not be maintained • Enough hardware resources must be planned and provisioned

Branch On-prem: LAN • Secure access to Enterprise and Internet • “IPSec to the cloud and you are done” x86 host In the Cloud: • all service creation and advanced security Enterprise Fabric • Example: Firewall for Direct Internet Access:

▪ SIG includes DNS-layer security, Web Gateway (SWG), Layer 7 Firewall, CASB and correlated threat intelligence.

▪ Redirection for SIG services via IPSec tunnel.

▪ SD-WAN automated tunnel creation coming in 1HCY20. Today – simple manual provisioning.

Enable new application classes Deploy & manage with Ease • Deploy new cloud native, machine learning or • Seamless, cloud based remote analytics applications in addition to traditional management of all branch IT operations branch apps • Remote installation, upgrade and lifecycle • Create new revenue generating services for Zero management customers Touch • Enable local data mining and data analytics Branch of the Branch Health Monitoring/Remediation Future Increase Availability • Integrated branch dashboard with unified view of • Always on / high availability for all IT health of all IT infrastructure infrastructure • Connected TAC for automated troubleshooting • HA for compute / storage / LAN / WAN & diagnosis across all lifecycle operations

vMANAGE +

Generate HX Cluster Provision HX Server Automate remote Assign SD-WAN Generate and deploy HX cluster installation, Profile + SD-WAN Profile and bootstrap deployment of virtual device templates virtual routers bootstrap configuration, and cluster Solution Profile Hypervisor routers VMs onto HX to virtual routers in configuration formation servers vManage

Hyperflex Full Life Cycle Management SD WAN Orchestration | Analytics | API’s Cisco Intersight Cisco vManage

Remote Office | Branch Office

Cisco HyperFlex Edge Cisco SD-WAN Fabric -or-

HX220C-M5SX HXAF220C-M5SX

Catalyst 9300L Catalyst 9300 -or- C9300L-48P-4X-E - 4x10G fixed C9300-48P base switch + 8x10G FRU

ISP CPEs • Supports HX Edge 2, 3, and 4 node clusters

• Supports up 2, 3, or 4 WAN connections

• WAN connections can all be either single or dual terminated on vEdge Routers

• HX Edge 10G (Catalyst)

• vEdge Cloud deployed as SD-WAN Edge router • vEdge VM deployed with 5 vNICs by default • 4 vNICs for WAN facing access connections • 1 vNIC for LAN facing service side (trunk) connections

• vSwitch configuration automated by HX + SD- WAN Installer

• Physical switches are manually configured

• Requirement for legacy Interface support (T1/E1, DSL, 4G)

• Introduction of Automation operational procedures

• Encryption Throughput • Encrypting traffic without hardware assists hits throughput

• Direct-Internet-Access architectures • Order of VNFs: virtual firewall, virtual Router • Firewall/IPS within vRouter

• Branch in a box • What about the switching / wireless infrastructure in the branch?

TECSPG-2300 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 73 • Enterprise Network Compute System (ENCS) is purpose built for branch Use-case virtualization. • Virtualization OS: NFVIS Summary: offers functionality to Virtualizing simplify branch Enterprise virtualization. Branches • Cisco DNA Center provides a workflow-based Automation Environment

MANO

OSS/BSS NFV Orchestrator

EM 1 EM 2 EM 3 VNF-M (VNFVNF Managers)Manager VNF 1 VNF 2 VNF 3

NFVI • NFVI - Network Function Virtualization Infra- Virtual structure is the totality of all hardware and Virtual Storage Virtual Network Compute software components that build the platform in Virtualized which VNFs are deployed. Virtualization Layer Infrastructure Manager Hardware Resources • VIM - Virtualized Infrastructure Manager Controls and manages the NFVI compute, Compute Storage Network storage, and network resources.

North Bound APIs NFVO, Resource Orchestration & VNF Service Orchestration

NSO – Network Services Orchestrator enabled by Tail-f

Virtual Network Functions (Cisco and 3rd Party) VNF Manager

CSR ASAv Ultra VMS Video XRv vWSA 3rd Party Cisco ESC

Virtual Infrastructure VIM API Virtual Compute Virtual Storage Virtual Network Red Hat OSP (RHEL) (Ceph) (OVS, VTF, SR-IOV) Cisco VIM

Management Management Lifecycle Manager Infrastructure Abstraction with RHEL, KVM/Qemu, Host Packages, vSwitches GUI Cisco Physical Infrastructure

Optional Network VIM Unified Management Unified

Monitoring Assurance and Monitoring Compute (UCS) Network (Nexus) Storage (UCS) (Cisco VTS / Cisco ACI) Infrastructure Infrastructure

Cisco NFVI Scope

Mobility Infra VNFs Business Services 3rd party VNFs (e.g. Ultra) (e.g. vRR/vBNG/vPE) (e.g. Cisco vMS) (e.g. vIMS, vLB)

Open APIs for Platform Consumption

Virtual Infrastructure VIM

API Virtual Compute Virtual Storage Virtual Network Red Hat OSP (RHEL) (Ceph) (OVS, VTF, SR-IOV) LifecycleCisco VIMManager Management Management Lifecycle Manager Infrastructure Abstraction with RHEL, KVM/Qemu, Host Packages, vSwitches GUI Cisco Physical Infrastructure Optional Network VIM

Unified Management Unified Optional Network VIM

Monitoring Assurance and Monitoring Compute (UCS) Network (Nexus) Storage (UCS) (Cisco(SDN VTS Controller) / Cisco ACI) Infrastructure Infrastructure

Cisco NFVI Scope

Native HW Ultra Ultra CUPS Micro Services / Cloud Native Container

Ultra Services Platform VNF-EM

SCM SLAM LCM

Control Plane

CF CF

Centralized Management Ultra Services PlVaNtFf-oEMrm SCM SLAM LCM

Control Plane User Plane UPP UPP Control Plane / Forwarding Plane CF CF User Plane

Session Plane Session UPP UPP Plane SF SF Forwarding Remote User Plane Plane

Forwarding Plane

NF AF

User Plane

UPP UPP

• Hardware based • Virtualized EPC • Scale bandwidth Platform • 5G-Ready • Fully Programmable • VNF Automation • High session • Scalable and throughput • Multi Access • 5G Any Use Case reliable • Distributed IP anchor • Low latency

Branch Branch DC VMs Branch Internet/MPLS vRR

ASR1001 & ASR1001 & CSR1000v CSR1000v RP2 (8GB) RP2 (16GB) ASR1002-X ASR1002-X (8GB) (16GB) (8GB) (16GB) ipv4 routes 7M 13M 8.5M 24.8M 8M 24M

vpnv4 routes 6M 12M 8.1M 23.9M 7M 18M

ipv6 routes 6M 11M 7.4M 21.9M 6M 17M

vpnv6 routes 6M 11M 7.3M 21.3M 6M 15M

BGP sessions 4000 4000 4000 4000 8000 8000

RT OSS

Network Control Orchestrator

L2 VLAN vBRAS vLNS – Retail ISP LNS –Retail ISP attachments x86 servers HGWs vBNG vLNS IPv6 IPv6 Core tunnels vRouter LNS VPEF CSR vLNS LNS VMs LNS Internet CSR LNS VMs

HGWs DHCPv4 RADIUS IPv6 tunnel L2TPv2oIPv4 Retailer Physical LNS end-points tunnels RADIUS WAN Network Data Center

Miercom CSR1k Test Report also includes vBNG use case Key Finding: • Optimize on 4 levels: BIOS, IO, CPU, VM • Horizontal scaling – Performance of 3 x 2 vCPU VMs > 1 x 8 vCPU VM

Complex interactions between services, databases, messaging queues, etc.,

Health and performance of a cloud is difficult to quantified, verify and monitor

Updates/upgrades require extensive human effort and are prone to issues

Unified Management System (Multi-Pod & Multi-Site, Single Pane of Glass, GUI, REST API)

Lifecycle Manager (Day N operations – Pod Mgmt, Update/Upgrades, Reconfig, REST)

Integrated Tools (Benchmarking: Networking, Storage, Compute)

Logging & Assurance (ELK stack, Zenoss, …)

Health Checks & Failure Recovery (CloudPulse, Cloud Recovery, REST) Day N

Control and Data Plane HA (Compute, Network & Storage) Cisco VIM

Turn Key Ubiquitous Security (TLS, SELinux, non-root, RBAC, etc. ) Packaged Software Performance Enhancement (Fast Data Stacks like VPP, tuning – CPU pinning, NUMA and many more) Integrated SDN Controller (VTS, ACI)

Containerized Deployment (OpenStack Services, CI/CD Capable Platform) Day 0 Fully Automated Installer (1-click, Modular, Robust)

Red Hat Enterprise Linux OpenStack Platform (RHEL OSP) Red Hat Ceph Storage Solution OpenStack, Linux & Storage Operating Systems – Red Hat Enterprise Linux (RHEL) and Cisco NX-OS / IOS-XR Distribution

Cisco UCS Cisco Cisco Cisco H/W Hardware VIC NIC FPGA* GPU* Compute Nexus 9000 UCS FI NCS5000* Accelerator* * Future

An integrated network performance benchmarking toolkit, pre-installed on every POD along with a set of best known practices

Build node Cisco VIM Pod 2 Traffic TOR-SW A TOR-SW B generator Build node 1 Controller 1 Controller 2 Controller 3 5 4 Storage 1 Storage 2 Storage 3 Compute 1 Compute 2 Compute 3 NFVbench 3 Compute i container Compute Compute Compute n

1 Stage VNF chain (OpenStack API) 3 Clear counters in vswitch(es) 5 Traffic flows to the VNF 2 Stitch traffic generator interfaces to VNF chain 4 Start traffic

+------+------+------+------+------+------Traffic+------generator+------(TRex------) + | Interface | Device | Packets (fwd) | Drops (fwd) | Drop% (fwd) | Packets (rev) | Drops (rev) | Drop% (rev) | +======+======+======+======+======+======+======+======+ | traffic-generator | trex | 3,561,150,633 | | | 3,561,152,091 | 0 | 0.0000% | +------+------+------+------+------+------Physical+------switch+------+ | vni-4096 | n9k | 3,561,150,633 | 0 | 0.0000% | 3,561,152,091 | 0 | 0.0000% | +------+------+------+------+------+------+------+------+ | vxlan_tunnel1 | vpp | 3,561,150,433 | 200 | 0.0000% | 3,561,152,091 | 0 | 0.0000% | +------+------+------+------+------+------+------+------+ | VirtualEthernet0/0/0 | vpp | 3,561,150,433 | 0 | 0.0000% | 3,561,152,091 | 0 | 0.0000% | +------+------+------+------+------+------+------+------+ | VirtualEthernet0/0/8 | vpp | 3,561,150,433 | 0 | 0.0000% | 3,561,152,091 VPP | and 0VNF | 0.0000% | +------+------+------+------+------+------+------+------+ | vxlan_tunnel0 | vpp | 3,561,150,433 | 0 | 0.0000% | 3,561,152,091incoming | and 199 outgoing| 0.0000% | +------+------+------+------+------+------+------+------+ | vni-4097 | n9k | 3,561,150,433 | 0 | 0.0000% | 3,561,152,290 | 0 | 0.0000% | +------+------+------+------+------+------+------+------+ | traffic-generator | trex | 3,561,150,431 | 2 | 0.0000% | 3,561,152,290 | | | +------+------+------+------+------+ Physical switch

Traffic generator (TRex)

TECSPG-2300 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 85 • Control plane virtualization • vRR with CSR1k • vBNG with CSR1k/XRv • NFV Infrastructure (NFVI) Use-case Summary: SP Infrastructure Virtualization

Pros: Cons:

• Potential cost savings • Complexity (management, know-how)

• No vendor lock-in • Greater attack surface

• Reliability and/or redundancy • Increased latency

58% of businesses using combination of AWS, Azure, or Google Cloud! * * Source: Kentik 2019 Report

VPCs

CoLo AWS

LB WAAS

WAAS IPS Apps IPS

Enterprise Fabric Apps NFVIS

WAAS

IPS • Apps increasingly live in multiple clouds Apps • Cloud provider Independence • SaaS

• CoLo Virtualization facilitates multi-CP connectivity

DMZ (physical)

Enterprise pFW SLB pFW Web pFW ALG pFW Network Internet

APP

DMZ (virtualized)

Enterprise pFW SLB vFW Web vFW vALG pFW Network SLB vFW Web vFW vALG Internet SLB vFW Web vFW vALG

APP

Per-App network functions and operation

Dynamic & automated service insertion with focus on Security

Usage-based consumption model

• Applications are moving into the cloud!

Customers

DMZCarrier (virtualized)-Neutral Facility / CoLo Partners Guests Enterprise pFW SLB vFW Web vFW vALG pFW Network SLB vFW Web vFW vALG Internet SaaS SLB vFW Web vFW vALG Employees Enterprise DC

APP Public Cloud DB

Applications

Security Simplicity Fidelity Economics • Automation • Regional CoLo • Consistent • Carrier / CP • Orchestration Breakouts Security Independence • Hardware • Reduction of • Central • Tight SLA reduction Circuit Costs interconnection point

• Expressing the relationships between Employees, customers, partners and applications by policy

Employee Employee Partner B2B Customer Private DC Partner UnmanagedCustomers WAN VPN Extranet B2C Clouds Internet Ecomm S/P/I Carrier-Neutral Facility / CoLo Partners Guests Employee WAN Enterprise pFW SLB vFW Web vFW vALG pFW Employee VPN Network SLB vFW Web vFW vALG Internet SaaS SLB vFW Web vFW vALG Partner B2B Extranet Employees EnterpriseCustomer DC B2C Ecomm Private DC Partner Clouds S/P/I Public Cloud

ApplicationsUnmanaged Internet

Cisco & 3rd Party Flexible Switching CSP 5444 Open Orchestration VNFs Fabric

VNF Hypervisor Virtual First Focus VNF Ready Fabric Automate and Orchestrate Cisco and Performance Focused Consistent Software Scales from Small to rd Hardware between Virtual and Large Deployments 3 Party VNFs CLI, GUI, and API Hardware Create repeatable service chain models Driven

Multi-Cloud Exchange Scope

Zones Zones Services catalogue/ Employees Orchestration DC WAN/ WAN Campus

PEP Mobile VPC: Workers Switching AWS

Partners VPC: Azure Internet

Customers Internet NFV Appliance SAAS

Virtual Network Services

Hardware Overview • Two Intel Xeon Gold 6152 @ 2.1Ghz (44 Cores) • 192GB DDR4 RAM (2666Mhz) • Eight 1.2TB disks (RAID10, 4.4TB usable) • Two onboard Intel (Niantic) 10Gb/ps ports (Management) • Two, two-port PCIe Intel (Niantic) 10Gb/ps cards (SR-IOV) • One, four-port PCIe Intel Fortville 10Gb/ps card (OVS)

Software Overview • NFVIS v3.11.1 • CCM • vDaemon

Metric CSP-5444

Physical form factor Cisco UCS C240M5 (2RU)

PCIe NIC slots 6x

Disk slots 24, 11.5TB SSD or 14.4TB HDD

RAID 12-Gbps SAS HW controller; 4-GB flash-backed write cache (FBWC); RAID 10 Memory Up to 756 GB, 24 DIMM slots Base Networking 2x10GbE LOM on-board NIC 1 VIC slot: 4x10GbE VIC1457, 1GbE i350, 2x10GbE i520, 4x10GbE i710 PCIe Network Interface Cards (NIC) Intel X520 2-port 10G and/or Intel XL710 4-port 10G SFP+

Processors (2) Two Intel Xeon Gold 6152 @ 2.1Ghz (44 Cores)

Power Supplies (1 or 2) AC or DC power

• RHEL 7.2 based system Device Console/ YANG Web SSH Portal • Manage via GUI, CLI (IOX XR synax), REST API, Netconf/Yang NFVIS-DC CLI NETCONF REST • GUI & REST connections are over HTTPS Health HTTPS • YANG models – used by NSO or other Orchestration APIs Monitor MANO

• Service Image Types: ISO, OVA, Virtualization Layer – Hypervisor & vSwitch QCOW/QCOW2, VMDK, RAW

Interface Platform Drivers Linux Drivers • Day.0 config file support for services like CSR1000V, ASAv

• CIMC Support

• VM Life Cycle Management CLI NETCONF REST

• VM Service chaining Web Server/ Cluster Confd esc-lite Portal • VM image management (in the local storage) Manager

Image Statistics Service Host • Platform management AAA Manager Collector Chaining Manager • Local WebUI portal

• Netconf and REST APIs libvirt Open vSwitch Qemu Snmpd Syslogd

• Command line interface (CLI) commands Centos Linux 7.4 + KVM + Kernel Drivers • AAA features (Tacacs+, Radius)

• Syslog, SNMP

• Multi-Cloud Exchange image • Contains CCM as a bundle • Enables SR-IOV on physical ports

Cisco VNFs Supported • CSR 1000v • ASAv • NGFWv

3rd Party VNF Support

• EN BU sponsored certification program to validate 3rd Party VNFs on Cisco NFVI Platforms

• Vendors can certify their VNFs on one or more Cisco NFVI Platforms:

• Includes: NFVIS-DC, CSP5444, NFVI (OpenStack)

• Based on EN BU Pipeline, SVS certifies or re-certifies 3rd Party VNFs before granting access to the Open VNF Ecosystem

• Multiple Multi-Cloud Exchange CoLo Deployments under a single Management / Orchestration

Multi-Cloud Exchange Orchestration

CoLo 1 CoLo 2 CoLo N

CSP5444 CSP5444 CSP5444 CSP5444 CSP5444 CSP5444 CSP5444 CSP5444 CSP5444

Cat9K Cat9K Cat9K Cat9K Cat9K Cat9K

• Greenfield solution deployments only • vManage based Orchestration from the cloud of • global config, • service chains • VNF management / monitoring / troubleshooting • Capabilities supported: • zero touch deployments for Multi-Cloud Exchange CSP platforms, Cat9K switches, CCM, and VNF • PnP Cloud Connect used for CSP and Cat9K • vManage bring up for VNF and CCM • Global config on CSP and Cat9K • VNF bringup with and without HA mode (Add/Edit/Delete) • Colo level service chaining (Add/Edit/Delete) • Colo health monitoring and troubleshooting of all platforms in a colo • Image management of CSP and Cat9K

Multi-Cloud Exchange Orchestration • Multiple CSP 5444 • Host VNFs vManage vBond PnP Cloud Connect

• Multi-Cloud Exchange Connection Manager (CCM) • Docker container CoLo N • Hosted on one of CSP5444 CoLo 2 CoLo 1 • Switching Fabric CSP5444 CSP5444 • Minimum 2 Catalyst 9000 CSP5444 PAN PAN ASAv CCM ASAvCSP5444 FW FWPAN ASAv • Orchestrator CSR AVI CSR FWAVIPAN ASAv 1k LB 1kCSR LBFWAVI PNFPNF Components Platform Software Version PNF 1k LB NFVIS-DC NFVIS-DCCSR AVI 1k LB Compute Platform CSP-5444-x2(M5) NFVIS-DC 3.9.1 PnP LCM ConfD vDaemon PnPNFVIS-LCMDC ConfD vDaemon PnPNFVISLCM-DC ConfD vDaemon Switching Cat9500-40X IOS-XE 16.9.1 PnP LCM ConfD vDaemon

Orchestrator vManage 19.2 ConfD Cat9K PnP ConfD Cat9K PnP

• All physical network elements connect to OOB Management Network • Connection via Management Port

• Requirement to host DNS and DHCP server per CoLo

CoLo 1 CSP5444 CSP5444 CSP5444 PAN PAN ASAv CCM ASAvCSP5444 FW FWPAN ASAv CSR AVI CSR FWAVIPAN DNS DHCP ASAv 1k LB 1kCSR LBFWAVI PNFPNF PNF 1k LB NFVIS-DC NFVIS-DCCSR AVI PnP LCM ConfD vDaemon PnPNFVIS-LCMDC1k ConfDLB vDaemon WAN/ PnPNFVISLCM-DC ConfD vDaemon Internet Cat9K PnP LCM ConfD vDaemon

ConfD Cat9K PnP ConfD Cat9K PnP

• Redundant connectivity between CSP 5444 and Catalyst 9000

• CSP 5444 physical ports CSP5444 PAN CSR AVI ASAv • 2 x Intel X520 2-port 10G (Niantic) FW 1k LB

• SR-IOV mode for high-performance VNFs configured for VEPA NFVIS-DC SR- OVS HA-br PnP LCM ConfD vDaemon IOV • No link redundancy

• Intel XL710 4-port 10G SFP+ (Fortville) Mgmt Intel XL710 (fortville) Intel X520 (Niantic) Intel X520 (Niantic) • 2 ports for VNF HA state synchronization

• Port channel configuration

• Connected to OVS OOB Cat9K • 2 ports for production traffic for virtIO-based VNFs OOB Cat9K

• Port channel configuration

• Onboard GE for NFVIS-DC and VNF Management ConfD Cat9K PnP ConfD Cat9K PnP

• Port-channel configuration

• vManage can be used to upload VMs in .tar.gz format • User can package VM in root disk format (qcow2/raw) Tar.gz information • need to enter meta data to save VM into catalog • root disk image/s to boot the VM • package manifest for checksum validation • No distinction made by vManage between Cisco of the file listing in the package rd and 3 party VNFs • image properties file in xml format listing the VM meta data • Meta data • (optional) day0 config/s, other files • VNF type required to bootstrap the VM • (optional) HA day0 config if VM supports • HA, SR-IOV, stateful HA • Resource requirement • HA Nics, Mgmgt NICs • NIC 0 is default for Mgmt, NIC1 is default for HA

• Chain of VNFs (SFC) connect ingress to cloud

• First and last is router or firewall • Can be shared

• For each SFC, need firewall • Throughput • Input/output VLAN • HA: yes,no router loadbalancer firewall router • VNF list with descriptors (name, image, resource requirement, IP, bootstrap config..)

• Placement logic produces per CSP service chain list firewall • Support for FULL chains connecting ingress with egress

• Service chains orchestrated by vManage • Assigns VLANs to individual VM vNICs • Configures switch • vManage does NOT configure individual VNFs

• Supporting full chains

• CSP5444 VNFs in a chain can run on one or multiple CSP CSR ASAv VNF 1k 3 • In same cluster VLAN VLAN VLAN VLAN • L2 based (VLAN, MAC address) -> needs to be supported by 100 110 120 130 switch NFVIS-DC OVS • SR-IOV • Uses VEPA-mode (service chaining on Cat9K) Mgmt Intel XL710 (fortville)

• VirtIO • Also uses VEPA (Not x86 internal OVS-based service chaining) • Each VM NIC configured on same access VLAN on the CSP • Switch pop/push VLAN tags • VNFS are unaware of VLANs ConfD Cat9K PnP ConfD Cat9K PnP

Service Chain Intent • Based on available NW, redundancy, compute Device Service Chain Catalog Configs

• Finds CSP for active VMS and different CSP for VM standby VMs Throughput • CPU HA • Memory • Disk • NICs • Not connected to same Cat9K

• If a chain contains VMs with stateful failover, then create an active/sby chain

• No overprovisioning, no hyperthreading Placement Logic • No attempt to find a CSP to fit all VNFs of a chain (but may happen)

CSP5444 CSP5444 CSP5444 CSP5444 Config Config Config Config

Multi-Cloud Exchange Orchestration • Instantiates, Monitors, managed components vManage vBond PnP Cloud Connect • Cloud hosted

• vBond: provides vManage info to devices running behind a NAT CoLo N • Initial AAA CoLo 2 • STUN server CoLo 1 CSP5444 CSP5444 • vManage CSP5444 PAN PAN ASAv CCM ASAvCSP5444 FW FWPAN • Config management, monitoring, ASAv CSR AVI CSR FWAVIPAN Troubleshooting ASAv 1k LB 1kCSR LBFWAVI PNFPNF • Shows CoLo status for cluster down to device PNF 1k LB level NFVIS-DC NFVIS-DCCSR AVI PnP LCM ConfD vDaemon PnPNFVIS-LCMDC1k ConfDLB vDaemon PnPNFVISLCM-DC ConfD vDaemon • vOrchestrator PnP LCM ConfD vDaemon • Automates provisioning of vBond, vManage ConfD Cat9K PnP ConfD Cat9K PnP

• Bootstraps switches • Configuration of Cat9K and CSP5444 Multi-Cloud Exchange Orchestration • Leverages a PnP server

• Image management for CSP5444 vManage vBond PnP Cloud Connect • Local image cache

• CoLo VNF Placement Logic CoLo N • Configures Service Chains CoLo 2 CoLo 1 • Configures and manages the service chain config on switches and PNFs. CSP5444 CSP5444 CSP5444 PAN PAN ASAv CCM ASAvCSP5444 • Catalyst 9000 config management FW FWPAN ASAv CSR AVI CSR FWAVIPAN ASAv 1k LB 1kCSR LBFWAVI • Health monitoring of physical devices PNFPNF PNF 1k LB NFVIS-DC NFVIS-DCCSR AVI 1k LB • Event Notification for CCM and PnP LCM ConfD vDaemon PnPNFVIS-LCMDC ConfD vDaemon PnPNFVISLCM-DC ConfD vDaemon switches PnP LCM ConfD vDaemon

ConfD Cat9K PnP ConfD Cat9K PnP

• Runs in a container on ONE of the CSP • Functional Components

5444 nodes in the cluster • Service chain manager

• PnP Server • Integrated NSO / vDaemon • Notification Handler • NSO CDB used for config repository • Switch config manager

• Image Manager

CSP5444 CloudCock Connection Manager NSO Image Mgr Logging / Operation Mgr SSH AAA (CCM, Physical Devices) Syslog

Notification Mgr LB Policy. Mgr Cat9K Security Mgr PnP Server Config DB Service Chain Mgr Config Mgr

Linux

• Colocation is the key offering Use-case significant cost savings Summary: • Cloud onRamp for Colo is an Connecting to automated solution, which provides virtualization and Multiple Clouds multicloud support

Thin Branch SP PoP - vCPE

WAAS

IPS Thin CPE NFVI/NFVIS

• Many MSPs are already offering virtualized CPE services • Verizon: http://www.verizonenterprise.com/products/networking/managed-network-services/ • NTT-E: https://newsroom.cisco.com/press-release-content?type=webcontent&articleId=1896371&dtid=osscdc000284 • Remove complexity (Functions) from the branch • Leverage virtualization in the SP Data Center / PoP to add services

• Requirements • Cloud / DC hosted with thin CPE • Feature Functionality: DHCP, PPP, NAT, Overlay Tunnels (GREv6), DNS Proxy • Accommodate Voice Services • Accommodate future services: WAN optimization, Web Security, UTM • Low Bandwidth (1 Mbps – 10 Mbps) per SoHo Cisco MANO Thin Branch NSO VTS ESC OpenStack 3 PODs

SP Core Internet Thin CPE DC GW Internet GW

x thousands • 10-20 x CSR 1000v • For each CSR 1000v … • 50 – 100 SoHo CSR 1000v x 10-20 CSR 1000v

Multi- Cost/ MSP tenancy Performance Orchestration

• Multiple single-tenant • Minimize hardware / • OSS/BSS Integration VNFs vs. single multi- licensing costs costs tenant VNFs • Maximize VNF density • Service automation for • Single-Service VNFs vs. SMB service multi-Service VNFS

CSR 1000V Infrastructure agnostic software • Familiar IOS XE software App • No dependency on specific server or vSwitch OS Throughput Elasticity • Licensable throughput from 10 Mbps to 10 Gbps Virtual Switch • Footprint options from 1 to 8 virtual CPUs Hypervisor Multiple Licensing Models • Term (1 or 3 Year), perpetual, hourly (AWS) Server usage Programmability • NetConf/Yang, RESTConf and SSH/Telnet for automated provisioning and management

• Most features in IOS XE are already VRF-aware • vrf-aware IPSEC

• Routing, IP Services, NAT, Firewall, IPSEc, SBC • vrf-aware snmp

• Can deploy CSR 1000v as in a multi-tenant / VRF configuration in the MSP • vrf-aware netflow infrastructure • vrf-aware dhcp • Examples: • vrf-aware firewall • vrf-aware • Routing (incl. BGP) telnet/ssh/tftp/SCP/HTTP/FTP • vrf-aware eigrp • MPLS (L3VPN, L2VPN, multicast, VRF-lite) • vrf-aware vpdn • vrf-aware ospf • HSRP support for MPLS VPNs • vrf-aware EVN • vrf-ware bgp • DHCP Class Capability • vrf-aware IPSLA • • DHCPv6 Relay and Server - MPLS VPN Support vrf-aware mpls-vpn • vrf-aware bcd • VRF-aware DNS • vrf-aware gre • vrf-aware ACL • Match-in-VRF support for NAT • vrf-aware NAT • vrf-aware pbr

• IPSLA for TCP connect, FTP, HTTP, DNS client • vrf-aware DMVPN • VRF-aware software infrastructure (VASI)

• VRF-aware IPSec

• VRF-aware ZBFW

• VRF-aware SSLVPN

• Create, Manage, Operate Managed Services Offers

• Key Capabilities: • Services Catalog • Zero touch Provisioning • Lifecycle management of physical and virtual assets • Self Service Operator & Tenant Portal • Identity Management • Mapping Functions • Service Analytics and Management Displays

OSS/BSS

UI UI / API Silo Integrations

UI UI / API UI UI / API UI / API

UI UI / API

UI UI / API UI / API UI UI / API MSX Common framework for service integration, catalog, delivery and operation.

SD-WAN Branch mCPE DNA-C SP DC Cloud Meraki CDO NFV NFV

Increase in Service Velocity Reduced Operations and Integration Costs

Lower Solution Complexity Increased Differentiation Potential

Virtual Branch (vBranch x86 based) Cisco SD-Access / Managed LAN • Universal CPE deployment of VNFs • Deployment of Managed LAN/ SD-Access • Service Chaining, 3rd party support • ZTP, Template push, Configuration • Day 0 to Day N • Multi-tenant DNA Center Operations Cisco SD-WAN Cisco Cloud Connect • Deployment of SD-WAN Solution • Simplify connections to AWS VPC • ZTP, Bulk Template Provisioning for pNF/VNFs • Simply and securely connect customers sites • Complements vBranch to AWS applications Managed Device • Plug&Play any Device (NSO NED supported) • Configuration template support • 3rd Party support

• Simple operations based on Cloud Management • aaS or on-prem • Micro-services, K8s, Docker, ELK

• Integration into SP Operations using REST APIs

• Extensible via SDK

• Customizable using Workflow Engine

Cloud Managed Cloud Connector SDWAN SDBranch Meraki CUTD SDA Connect Device Interface (CCI)

MSX (µs) Infrastructure Services MSX Services nginx/waf consul vault ipnp workflow manage usermanager admin consume ESC

sdabeat redis snmpbeat kibana billing notification orchestration alerting router LiveSP kafka service zookeeper elastic cassandra msx-ui monitor extensions

encsbeat nso heartbeat sshbeat

Kubernetes

VIM

Use-case • VNF density on an x86 Summary: Multi- host can impact tenant SMB performance • You need to understand Services the cost/performance trade-offs

• ENFV Solution Overview -http://www.cisco.com/c/en/us/solutions/collateral/enterprise-networks/enterprise-network- function-virtualization-nfv/solution-overview-c22-736582.pdf

• ENFV AAG - https://www.cisco.com/c/dam/en/us/solutions/collateral/enterprise-networks/enterprise-network-functions- virtualization-nfv/at-a-glance-c45-736581.pdf

• ESA Datasheet - http://www.cisco.com/c/en/us/products/collateral/cloud-systems-management/application-policy- infrastructure-controller-enterprise-module/datasheet-c78-736830.html

• ENFV FAQ - http://www.cisco.com/c/en/us/solutions/collateral/enterprise-networks/enterprise-network-functions- virtualization-nfv/q-and-a-c67-736831.html

• ISRv Datasheet - http://www.cisco.com/c/en/us/solutions/collateral/enterprise-networks/enterprise-network-functions- virtualization-nfv/datasheet-c78-736768.html

• ENCS Datasheet – http://www.cisco.com/c/en/us/products/collateral/routers/5400-enterprise-network-compute- system/datasheet-c78-738512.html

• ENCS FAQ - https://www.cisco.com/c/dam/en/us/products/collateral/routers/5400-enterprise-network-compute- system/q-and-a-c67-738424.pdf

• ENFV Ordering Guide - http://www.cisco.com/c/en/us/solutions/collateral/enterprise-networks/enterprise-network- functions-virtualization-nfv/guide-c07-738514.html

• Deployment Guide for Amazon Web Service\http://www.cisco.com/c/en/us/td/docs/routers/csr1000/software/aws/csraws/awsinstall.html

• CSR 1000v in Microsoft Azure Deployment Guide • https://supportforums.cisco.com/document/12744996/cisco-csr-1000v-deployment-guide-microsoft-azure

• CSR 1000v Unified Performance Testing Results • http://up-tools.cisco.com/trending/index.html

• EANTC OVS and VPP Performance report • http://www.lightreading.com/nfv/nfv-tests-and-trials/validating-ciscos-nfv-infrastructure-pt-1/d/d-id/718684? • With significant input from the CTAO team under D. Ward! • Youtube Video: https://www.youtube.com/watch?v=Z5M0Zl0uvj0

• Vnet SLA Performance Testing (Cisco Intern only) • http://wikicentral.cisco.com/display/PACKETCOM/Virtual+Network+SLA

• OVS Performance • OVS Performance Characterization (Madhu Challa) • OVS Performance Blog • OVS Performance Characterization Paper, TU Munich, IEEE Conference on Cloud

• Paper written to describe the details of Virtualized system architectures

• Extension of Cisco Live Presentation TECSPG- 2300

• Lots of additional references therein

• Reference:

• M. Falkner, A. Leivadeas, I. Lambadaris, G. Kesidis, “Performance Analysis of Virtualized Network Functions on Virtualized Systems Architectures”, IEEE CAMAD, Oct. 2016.

TECSPG-2300 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 135 Complete your online session • Please complete your session survey survey after each session. Your feedback is very important.

• Complete a minimum of 4 session surveys and the Overall Conference survey (starting on Thursday) to receive your Cisco Live t-shirt.

• All surveys can be taken in the Cisco Events Mobile App or by logging in to the Content Catalog on ciscolive.com/emea.

Cisco Live sessions will be available for viewing on demand after the event at ciscolive.com.

Demos in the Walk-In Labs Cisco Showcase

Meet the Engineer Related sessions 1:1 meetings