EUF-NET-T1745

ARM® V8 FOR LAYERSCAPE MULTICORE COMMUNICATIONS PROCESSORS

MARCH 2016 [email protected] - SR. FAE EMEA 22 - טל אביב NXP TECH DAY

EXTERNAL USE Agenda

• Virtualization Introduction

• Layerscape ARM ®v8 Virtualization Status & Roadmap

• I/O in KVM Environments

− Device Virtualization - virtio

− Device Direct-Assignment - VFIO

• Q&A

1 EXTERNAL USE VIRTUALIZATION INTRODUCTION

2 EXTERNAL USE Virtualization Technologies for QorIQ Layerscape architecture KVM Containers Embedded

• Linux ® Hypervisor • Low Overhead • Lightweight Hypervisor • Resource Virtualization • Isolation and Resource • Resource Partitioning Control in Linux • Resource • Para-Virtualization Oversubscription • Decreased Isolation • Failover support (Kernel sharing) • 3rd Party OSs • 3rd Party OSs

VM VM

App App App VM VM VM Cont Cont Cont

App App App App App App OS OS

LXC LXC LXC KVM OS OS OS

Linux Linux ® Embedded Hypervisor

Multicore Hardware Multicore Hardware Multicore Hardware

3 EXTERNAL USE KVM/QEMU – Overview

• KVM/QEMU– open source Virtual Machine 1 Virtual Machine 2 virtualization technology based on the QEMU QEMU

App App • KVM is a Linux kernel module • QEMU is a emulator that App OS OS uses KVM for acceleration • Run virtual machines alongside Linux KVM Linux applications

Multicore • No or minimal OS changes required Hardware • Virtual I/O capabilities • Direct/pass thru I/O – assign I/O devices to VMs

4 EXTERNAL USE KVM/QEMU

• QEMU is a user space emulator that uses KVM for acceleration − Uses dedicated threads for vcpus and I/O − KVM leverages to run guest with higher privileges − Virtual chip emulation in kernel − I/O  Provides dedicated virtio I/O devices and standard drivers in Linux kernel  Uses VFIO Linux framework to direct assign physical PCI devices  Direct notifications between I/O threads and KVM using eventfds  vhost provides virtio emulation and I/O thread and in kernel  Multi-queue virtio devices connected to multi-queue tap devices − Provides services for console, debug, reset, watchdog, etc

5 EXTERNAL USE Linux Containers • Linu XContainers is based on a Container Container collection of technologies including

App App kernel components (, App namespaces) and user-space tools (LXC). Linux ® • OS level virtualization • Guest kernel is the same as the host kernel, but OS appears isolated 1 • Low overhead, lightweight, secure 1 7 12 partitioning of Linux applications 4 9 15 1 into different domains 15 17 21 4 7

Container 1 1 • Can control resource utilization of 1 3 Container 2 Container 3 domains– CPU, Memory, I/O • close to 0% performance overhead • process-level virtualization bandwidth

6 EXTERNAL USE Libvirt

• A toolkit to interact with the Domain Domain virtualization capabilities of Linux (and other OSes / ) LXC KVM • Goal: to provide a common and LXC Qemu libvirtd stable layer sufficient to securely driver driver manage domains on a node, Libvirt API possibly remote

Linux • Has drivers for KVM/QEMU and Linux containers • Many management applications supported Multicore Hardware • http://libvirt.org/

7 EXTERNAL USE Linux Containers

• Platforms supported: all … not platform dependent • Features − Technologies: LXC, , Libvirt − Setups: Busybox system containers, application containers − Networking  Shared with host  Host interface assignment  Virtual Ethernet device pair  VLAN / MACVLAN  USDPAA − Security: capabilities, , user namespace • Upstream status: upstream

8 EXTERNAL USE Container Technologies

Other Container Flockport DockerHub Technologies Distribution

Jails FreeBSD

Client LXC virsh docker Zones Solaris

OpenVZ Container LXD libvirtd Docker Linux Engine VServer () Google Containers

Low-level liblxc libvirt_lxc libcontainer API

Migration cgroups secomp Kernel CRIU

9 EXTERNAL USE Container Comparison

LXC/LXD Docker Libvirt

• Full system and application containers • Single application virtualization engine • Virtualization high-level API with • Focus on performance and stability based on containers support for containers • Lightweight Linux containers • Focus on ease of use . Easy delivery of • Focus on unification with different apps in a Docker container virtualization technologies • Containers are like VMs with a fully functional OS • Each application has its own container. • Own version of container API – libvirt_lxc “Container as an app ” – tradeoff in order to fit the overall • Comprehensive set of tools for container architecture of libvirt lifecycle management • Docker restricts the container to a single process only • Developed by • Data can be saved in a container or outside • Instances are ephemeral. Persistent data is stored in host • LXD allows you to use LXC to create containers on other machines • Trade off in complexity and constraints. Suitable for read only app that is ‘frozen in • LXD aiming to used hardware that state’ “guaranteed isolation of containers ” on the chip level • Container Distribution Platform – Docker Hub • Container distribution platform – Flockport.com • Developed by PaaS providers (dotCloud) • Developed by Ubuntu/Canonical “Blindingly fast virtualization” “Great application delivery mechanism”

10 EXTERNAL USE Containers vs Hypervisors

Linux Embedded KVM Containers Hypervisor HW Support Needed? No Yes Yes Overhead Low Yes Yes Isolation Good High High Partitioning Yes Yes Yes Virtualization Yes Yes Yes Yes Yes Linux No Linux Multi OS RTOS (Linux Only) Bare-board Bare-board 3rd Party OSs 3rd Party OSs Features Failover Oversubscription Mainstream Mainstream Private Licensing Open Source Open Source Open Source Mature

11 EXTERNAL USE QORIQ LAYERSCAPE SDK VIRTUALIZATION STATUS & ROADMAP

12 EXTERNAL USE SDK Virtualization Status

Technology e500v2 e500mc e5500 e6500 ARMv7 ARMv8 KVM-PPC Up Up Up Up KVM-ARM Leveraged Leveraging FSL Hypervisor Public Sources (bare metal) LXC Leveraged & Fixes Leveraging Leveraged Leveraging Libvirt Leveraged & Fixes

• 3 complementary solutions supported on multiple platforms • Focus on enabling core virtualization support, upstreaming and good OOB experience in NXP SDK • I/O performance optimization in progress

13 EXTERNAL USE Layerscape ARM ®v8 Virtualization Roadmap

LS1043 BSP 0.5 LS2 EAR 0.6 SDK 2.0

[KVM] Basic KVM [LXC] LXC with seccomp [KVM] [KVM] virtio [Docker] Docker engine • DPAA 2.x Direct Assignment v1 (VFIO) [LXC] LXC Support • PCI Device Assignment (VFIO-PCI)

LS2080 SDK 1.0 Benchmarking Upstreaming Upstreaming [KVM] Kernel 4.1 & QEMU 2.4 [KVM] Benchmarking • MSI support [KVM] DPAA 2.x Direct [KVM] DPAA 2.x Direct • vhost-net, vhost-blk dataplane Assignment v2 (VFIO) Assignment v2 (VFIO) [LXC] LXC [Libvirt] Libvirt POC POC

LS2 POC POC [KVM] [KVM] [KVM] [KVM] PCI device assignment to • LS2 User Space (e.g ODP) • Performance improvements • DPAA 2.x Direct Assignment KVM VMs direct assignment in guest • Direct assignment of with interrupts [LXC] CRIU Demo • Performance improvements platform devices • KVM-RT on LS2 • LE Guest on BE Host • virtio-crypto demo [Docker] Docker engine on LS2 • BE Guest on LE Host • USB pass-through • Direct assignment of SEC • Shared-memory (ivshmem) • DPAA 2.x vhost-user optimizations • Live Migration Demo 2015 2016 1Q 2Q 3Q

Color Legend Italic features depend on upstream support Current Release Released

14 EXTERNAL USE Roadmap Date Major Release KVM - Out-Of- RFS Enablement

• Required components : − KVM support enabled in kernel − Guest image − Guest root file system − QEMU

Kernel config Guest rootfs Guest image QEMU fsl-image-core NO NO NO NO fsl-image-full NO NO NO YES fsl-image-virt NO YES YES YES

15 EXTERNAL USE I/O IN KVM ENVIRONMENTS

16 EXTERNAL USE I/O Virtualization - Performance vs Flexibility

Non-Interposable Interposable

Bare Metal Trend Direct Assignment (VFIO) vhost- vhost user optim vhost

virtio (para-virtualized) No Guest Modifications

Emulated I/O Scalability and Performance and Scalability

Flexibility

17 EXTERNAL USE Device Usage in Virtual Environments

Direct Access Partitionable Emulated Para-Virtualized HW • Fast native performance • Hardware partitioned • Driver in Hypervisor • Driver in Hypervisor • Direct access to hardware • One hardware block • Emulation in Hypervisor • Modified Drivers in • Unmodified Drivers in Guest Guest OS OS OS OS OS OS OS OS Custom Custom Driver Driver Driver Driver Driver Driver Driver

Emulation Driver Driver

I/O I/O I/O I/O I/O

Hardware/software access Hypercalls Traps

18 EXTERNAL USE DEVICE VIRTUALIZATION VIRTIO

19 EXTERNAL USE virtio virtio- virtio- virtio- virtio- virtio- virtio- console blk pci net scsi balloon

Guest • Device abstraction layer of para- Linux virtio virtualized hypervisor transport −Standard for VMs/VNFs Host Linux / QEMU −Appearance as physical devices virtio back-end drivers −Uses standard virtual drivers and discovery mechanisms  virtio-net : Ethernet virtual driver virtio frontend virtio frontend  vhost-net : optimizes Ethernet virtual driver by eliminating QEMU context switch  virtio-pci • Backend drivers are vendor specific in host Linux; transparent to VM/VNFs

Sources: https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Virtualization_Tuning_and_Optimization_Guide 20 EXTERNAL USE virtio

• Device facilities − Device status field / Feature bits / Device Configuration space / virtqueues • Transport protocols: PCI, MMIO • virtio specification, defined by OASIS technical committee − Straightforward - use normal bus mechanisms for interrupts and DMA − Efficient - rings of input/output descriptors − Standard - no assumptions about guest environment beyond supporting MMIO, Channel I/O or PCI bus transports. − Extensible - devices contain feature bits acknowledged by the guest OS

21 EXTERNAL USE KVM/Qemu virtio Back-end Drivers

virtio front-end Qemu/KVM back-ends virtio-net (legacy) Qemu

virtio-net (data-plane) Qemu, I/O thread virtio-net vhost Kernel

vhost-user User space

virtio-blk Qemu virtio-blk virtio-blk data-plane Qemu, I/O thread

virtio-scsi Qemu virtio-scsi vhost-tcm Kernel

22 EXTERNAL USE Back-end - virtio-net (legacy)

Qemu 2 Guest PCI emulation virtio backend

Interrupt virtio driver 1 CP 1 DP Controller 4 DP driver 3 /dev/net/tun 4 4 3 1 1 2 2 Kernel 3 KVM 1 MMIO 3 4 Emulated 4 Interrupt Tap Switch MacVTap Controller 2 Interrupt Controller driver Ethernet driver

2 irq DPNI DPIO HW

Descriptor Table Available Used Memory DMA AIOP Interrupt Switch, I/O IOMMU Controller Switch, I/O

Control Path Data Path 23 EXTERNAL USE IRQ Path Kick Path Back-end - virtio-net (data-plane)

Qemu I/O Thread . 2 Guest PCI emulation virtio backend I/O Thread

Interrupt virtio driver DP Controller 4 CP 1 DP driver /dev/net/tun

3 1

1 1 1 Kernel KVM ioeventfd 3 2 2 4 MMIO

Emulated FDIRQ irqfd Interrupt 4 Tap Controller Switch MacVTap 2 Interrupt Controller driver Ethernet driver

2 irq DPNI DPIO HW

Descriptor Table Available Used Memory DMA AIOP Interrupt Switch, I/O IOMMU Controller Switch, I/O

Control Path Data Path 24 EXTERNAL USE IRQ Path Kick Path Back-end - vhost

Qemu 2 Guest PCI emulation vhost CP logfd

Interrupt virtio driver vhost API Controller 4 CP 1 DP driver

3 1

1 1 1 Kernel KVM vhost ioeventfd MMIO 3 4 DP

Emulated FDIRQ irqfd Tap Interrupt 4 MacVTap Controller Switch Interrupt Controller driver 2 Ethernet driver 2

2 irq DPNI DPNI HW DPIO DPIO

Descriptor Table Available Used Memory DMA AIOP Interrupt Switch, I/O IOMMU Controller Switch, I/O

Control Path Data Path 25 EXTERNAL USE IRQ Path Kick Path Back-end - vhost-user

Qemu vhost-user 2 logfd Guest PCI emulation vhost-user CP DP Interrupt vhost-user virtio driver 1 Controller 4 CP socket API DP driver ODP

3 1

1 1 1 Kernel

KVM DPNI/DPIO ioeventfd 3 2 4 MMIO

Emulated FDIRQ irqfd Interrupt 4 Controller Interrupt Controller driver 2 VFIO

2 irq HW

Descriptor Table Available Used Memory DMA AIOP Interrupt Switch, I/O IOMMU Controller Switch, I/O

Control Path Data Path 26 EXTERNAL USE IRQ Path Kick Path Back-end - virtio-blk

Qemu Guest PCI emulation virtio backend

Interrupt virtio driver CP DP Controller DP driver Block layer raw, qcow2,…

Kernel KVM return MMIO

Emulated ioctl Interrupt File system Controller Block device Eth LUN

SATA PCI … HW

Descriptor Table Available Used Memory Interrupt IOMMU Controller Storage PCI HBA

Control Path Data Path 27 EXTERNAL USE IRQ Path Kick Path Back-end - virtio-blk data-plane

I/O Thread . Qemu

Guest PCI emulation virtio backend I/O Thread

Interrupt virtio driver DP Controller CP DP driver AIO

Kernel KVM ioeventfd MMIO

Emulated FDIRQ irqfd Interrupt File system Controller Block device Eth LUN

SATA PCI … HW

Descriptor Table Available Used Memory Interrupt IOMMU Controller Storage PCI HBA

Control Path Data Path 28 EXTERNAL USE IRQ Path Kick Path Back-end virtio-scsi - vhost-tcm

Qemu

Guest PCI emulation vhost CP logfd

Interrupt virtio driver vhost API Controller CP DP IOCTLs driver

Kernel KVM vhost-tcm ioeventfd MMIO DP

Emulated FDIRQ irqfd Interrupt File system Controller Interrupt Controller driver Block device Eth LUN

irq SATA PCI … HW Memory Descriptor Table Available Used Interrupt IOMMU Controller Storage PCI HBA

Control Path Data Path 29 EXTERNAL USE IRQ Path Kick Path Back-end Acceleration - vhost-acc (preliminary)

Qemu vhost-acc logfd Guest PCI emulation vhost-user CP Interrupt vhost-user virtio driver 1 vhost-user API Controller 4 CP socket API DP vhost-AIOP CP driver NADK

3 1

1 1 1 Kernel

KVM DPCI/DPIO ioeventfd 3 1 4 MMIO 4 3 4 Emulated IRQ FD irqfd Interrupt 4 Controller Interrupt Controller 4 driver VFIO

irq HW

Descriptor Table Available Used Memory DMA AIOP Interrupt DP IOMMU Controller 4 Switch, I/O

Control Path Data Path 30 EXTERNAL USE IRQ Path Kick Path DEVICE DIRECT-ASSIGNMENT VFIO

31 EXTERNAL USE Device Direct-Assignment

• Device drivers access from user-space − Device pass-through ( libusb , libscsi ) − Map /mem (not recommended) − UIO (User-space I/O)  Device access ( mmap device MMIO regions)  Interrupt support  No isolation or translation − VFIO (Virtual Function IO)  Linux user space driver infrastructure for DMA devices  Device access ( mmap device MMIO regions)  Enforces IOMMU translation and isolation ( iova to real address)  High performance interrupt support (INTx, MSIs & MSI-X)

32 EXTERNAL USE VFIO VM App App App

• VFIO (Virtual Function IO) : Linux user space driver infrastructure Guest OS

− Enforces IOMMU protection Host OS DMA MMIO − Device access : mmap() device MMIO regions IRQs IOMMU MMU − IOMMU programming interface

− High performance interrupt support Device Multicore Hardware − Bus support : PCI, platform devices, LS2 MC bus

• VFIO PCI - abstracts devices as :

− Regions :

 PCI configuration space

 MMIO and I/O port BAR spaces

 MMIO PCI ROM access

− IRQs include

 INTx (legacy interrupts)

 Message Signaled Interrupts (MSI & MSI-X) 33 EXTERNAL USE Source: www.linux-kvm.org/wiki/images/e/ed/Kvm-forum-2013-VFIO-VGA.pdf VFIO for PCI Bus

Qemu

Guest emulation

Interrupt PCI PCI Controller 3 DP 1 driver 2

3 Kernel KVM 2 MMIO

Emulated FDIRQ irqfd Interrupt VFIO Interrupt Controller Controller driver

irq HW Memory IOMMU Interrupt PCI-SRIOV Controller 4 CFG BAR CFG BAR ICID ICID VF PF ICID 5 ICID ICID

Control Path Data Path 34 EXTERNAL USE IRQ Path Kick Path DPAA 2 Secure Direct Assignment

ICID ICID ICID

• DPAA 2 architecture GPP DPAA 2 − Optimized for resource assignment to various MMU ICID ICID software contexts through Management Complex ICID

 Linux MC bus Authorize Memory Q/BMan ICID

 Resource management tool ICID ICID − IOMMU translation and protection for user-space (ODP and QEMU)

ICID  ICID ICID (StreamID) ICID  MC bus integration with VFIO IOMMU  Device reset ACC

ICID − DPAA secured with Authorization Tables ICID ICID

35 EXTERNAL USE VFIO for MC Bus

Qemu DPRC DPNI Guest Device Tree … Interrupt Eth Device driver Controller 4 DP 1 driver

4 3 2

Kernel KVM MMIO 4 3 Emulated FDIRQ irqfd Interrupt VFIO Interrupt Controller Controller driver

irq HW Memory IOMMU Interrupt 4 I/O MC Controller

ICID ICID ICID 4 ICID

Control Path Data Path 36 EXTERNAL USE IRQ Path Kick Path virtio vs Direct Assignment (VFIO) virtio Direct Assignment Flexibility High Med Guest Driver Generic HW dependent Device Sharing Yes No Live Migration Yes PoC prototyping Performance Medium High Backend is SW emulated in Host or in Reduced processing Processing in Host HW support for isolation No Required (SMMU) Licensing Open Source* Open Source Upstreamable ? Firmware accelerations - NO YES Started as software implementation in Framework implemented in Linux for PCI Linux and now API is standardized (OASIS) History devices that is extended for Platform Standard add-ons may not be accepted in devices. Linux upstream.

37 EXTERNAL USE Q & A

38 EXTERNAL USE

VM LIVE MIGRATION

40 EXTERNAL USE VM Live Migration with virtio devices

Qemu

Guest PCI emulation logfd MigrationMigration Guest emulation vhost CP PCI Interrupt virtio driver vhost API Controller CP DP IOCTLs driver

Kernel KVM vhost vCPU ioeventfd registersvCPU MMIO DP registersvCPU Emulated FDIRQ irqfd registers Tap Interrupt MacVTap Controller Interrupt Controller Switch driver Ethernet driver

irq HW DPNI DPIO DPNI DPIO

Descriptor Table Available Used Memory DMA AIOPAIOP Interrupt Switch, I/O IOMMU Controller Switch, I/O

Control Path Data Path 41 EXTERNAL USE IRQ Path Kick Path VM Live Migration with Direct Assignment

VM VMVM Application Application Application ApplicationApplication

Kernel KernelKernel

Bonding (active-passive) BondingBonding (active-passive) (active-passive)

NIC driver VIRTIO driver NIC driver VIRTIOVIRTIO driver

Host Linux Host Linux VIRTIO Source VIRTIO Destination back-end back-end

Tap Tap

NIC driver NIC driver

Hardware Hardware NIC NIC NIC NIC

42 EXTERNAL USE DATACENTER – SERVERS INIC SOLUTIONS

43 EXTERNAL USE Increasing Server Intelligence and Performance

x86 x86 x86

(VMM) LS3

(VMM) Std Smart Smart NIC NIC++ NIC (VMMxl) (VMMxl) T2080 / LS2 T4240

Server NIC More Unified manages manages capacity or Architecture packets packets services (simplification)

44 EXTERNAL USE Server non-iNIC case - iNIC demo Traffic Flow

X86 Xeon Linux Platform NIC Traffic Flow VM0 VM1 Openflow 1• VM-to-VM Intel DPDK Intel DPDK Controller* 2• VM-to-ethernet

Veth-port Veth-port 3• ethernet-to-ethernet 4 4• Openflow Control Plane path Open vSwitch 1 Openflow Agent S/W Packet Forwarding Engine 2

• DPDK NIC driver Hypervisor • Support for up to 128 VF PF VF1 VF2 VF3 VFn 10G Eth • Kernel Space vSwitch 10G Eth • Openflow Agent for traffic 3 management 10G Eth

10G Eth

* Can be external 45 EXTERNAL USE Server non-iNIC case - iNIC demo Traffic Flow

x86 Xeon Linux Platform

VM0 VM1 Openflow • NFV/SDN/Firewall/ACL • IPSEC Intel DPDK Intel DPDK Controller* • TCP offload • Data Compression Veth-port Veth-port • Deep Packet Inspection • Load Balancing • OpenSSL + record offload • Vendor defined applications 4 1 2

Hypervisor • Offloading of x86 CPU to increase aggregate with PF VF1 VF2 VF3 VFn application performance cost effectively. PCIe (SR_IOV) 128VF • Increase top end server 10G Eth performance • Scalable iNIC platform L4-7 performance T2080 to T4240. Openflow 10G Eth Apps Agent Reusable software. H/W Accel.Packet • Hardware acceleration for Forwarding Engine PME 10G Eth Data Path, Pattern Matching, SEC 3 Security and Decompression DCE User Space Open vSwitch 10G Eth /Compression, PKC/Record T2/T4/LS2 +c290 iNIC offload. * Can be external 46 EXTERNAL USE VCPE / EDGE NETWORK

VNFS - VIRTUALIZED NETWORKS FUNCTIONS

47 EXTERNAL USE vCPE Solution (OPNFV Platform + vCPE functions )

vVPN vRouter vFW vNF

Route, dhcp, etc iptables IKEvRouter (strongSwan) nd rd fpm fpm Fpm (2 /3 party) ODP Open fastpath Open fastpath Open fastpath Veth-port DPDK ODP DPDK ODP DPDK ODP socket socket socket socket User Space User Space virtio-net User Space virtio-net User Space virtio-net

Kernel Space (Ubuntu) Kernel Space (OpenWRT) Kernel Space (Ubuntu/OpenWRT) Kernel Space (Ubuntu)

virtio-net Net (ipsec) virtio-net Net virtio-net Net virtio-net Net front-end front-end front-end front-end

vhost-user backend vSwitch (dpdk/odp) Qemu (virtio) User Space

Kernel Space

virtio-net backend (vhost)

vSwitch

KVM Hypervisor

1G Eth 1G Eth

48 EXTERNAL USE NXP CONFIDENTIAL INFORMATION vCPE Solution (LS1088/LS2088 ARM v8 ISA + Advanced Packet Processing)

 Up to 8 GPP cores available for vNFs

 Full OPNFV platform compliance.

LS1088 OPNFV Platform  vNFs 100% source compatible with x86  vNF1 vNF2 1 to 8x performance scalability on a vNF0 vNF1 vNF2 single software platform DPDK/ODP DPDK/ODP DPDK/ODP ODP ODP ODP  AIOP packet forwarding engine frees up GPP cores AND significantly improves Veth-port Veth-port Veth-port Veth-port Veth-port Veth-port network throughput

AIOP packet forwarding engine Openflow Agent

vSwitch User Space

Kernel Space

KVM Hypervisor

1G Eth 1G Eth

Openflow Controller

49 EXTERNAL USE NXP CONFIDENTIAL INFORMATION CONCLUSIONS

50 EXTERNAL USE Conclusions

• I/O in virtualized environments is driven by new trends NFV, vCPE, vAccess • Goal is to provide standard I/O support efficient and flexible • KVM provides virtio, direct-assigned and pass-through devices • Optimizing virtio, direct-assignment, pass-through is instrumental • Supporting efficiently the Virtual Machine migration is a priority

51 EXTERNAL USE NXP SOLUTIONS FOR VIRTUALIZATION

52 EXTERNAL USE NFV/SDN Service Delivery - NXP Reference Designs

DC Solutions SDN Data  NFV Compute / Storage Center  iNIC / SSL Accel  -RAN L1 Accel NFV Services  ADC / WoC ToR Router NFV iNIC / Compute (T4/T2+C29x) Metro Metro Edge Solutions iNIC OVS offload with DPDK support OpenStack / Open Daylight Framework Edge  Metro Routers VortiQa v1.3 OF-Controller+L4-7 QorIQ  L4-7 Appliances Processors  Content Delivery  WAN Optimization Provider Edge Metro Access Solutions  Aggregation Routers  Broadband Gateway SDN L2-7 Router + NFV (T2/LS2+C29x)  Mobile BTS / C-RAN VortiQa OF-Agent+L4-7 Processing Premise Virtualized data plane (QorIQ & Switch) Service-Provider Networks Service-Provider CPE Access Solutions NFV edge-based services Router  Campus Router  Broadband Access  Wireless / Mobile AP IoT IoT IoT Gateway IoT Gateway + NFV (LS1/T1) Gateway Gateway  Building / Factory WLAN AP Mobile AP  Smart Energy IoT AP (e.g. BSC913x) OpenWRT  Transportation Sensor/ Sensor/ mgmt Sensor/ Sensor/  Digital Signage ActuatorSensor/ ActuatorSensor/ VortiQa OF-Switch+L4-7 ActuatorActuator ActuatorActuator  Medical / Fitness NFV edge-based Services End User Networks User End  Remote Monitoring

53 EXTERNAL USE