Overview of Qoriq Processor Solutions for Virtualization Features on T2080-Based Inic EUF-NET-T0967

Overview of QorIQ Processor Solutions for Virtualization Features on T2080-based iNIC EUF-NET-T0967

Peter Van Ackeren

J A N . 2 0 1 5

External Use Session Introduction

• This session explores the virtualization solutions available for on QorIQ communication processors

• After completing this session you will have become familiar with : − General virtualization principles and use cases − The features, benefits and some details of KVM/QEMU, Linux® containers and libvirt implementations as supplied in the QorIQ Linux SDK − Architecture specific details of Power Architecture and ARM® based platforms as applicable to virtualization use cases

TM External Use 2 Introduction

TM External Use 3 Traditional System Model

• Single OS runs on a single or multi-core hardware system

App

TM External Use 4 Multi-processing Usage Models SMP

• SMP = Symmetrical Multi-processing CPU

− multi-processor homogeneous computer architecture CacheCache

Cache

Cache − where two or more identical CPU MemoryMemory CPU processors are connected to globally shared main CacheCache memory and I/O CPU I/OI/O − processors could be separate devices, multiple cores in a single device or a mix Single OS − a single OS instance runs on and manages the cores, with shared memory, I/O and interrupt CPU CPU resources

− Each processor can run independent processes and Mem ctrl threads I/O − Any idle processor can be assigned any task

TM External Use 5 Multi-processing Usage Models AMP

• AMP = Asymmetrical Multi-processing − Individual CPUs are dedicated to particular tasks, like running an OS − different software instances run on separate CPUs . e.g. 2 copies of Linux image are loaded at different physical address locations

• Both CPUs must cooperate to share the resources − Neither OS can own the whole system − I/O and interrupts are separated − Static configuration of resources

TM External Use 6 What is Virtualization ?

• Virtualization – an abstraction layer that enables running multiple operating systems on a single system, implemented using hardware and software technologies

App App App

Linux® RTOS Linux®

Hypervisor Hardware • A hypervisor is a software component that creates and manages virtual machines which can run operating systems.

TM External Use 7 Types of Virtualization

• Full Virtualization: − Virtual machine behaves identically to physical hardware − OS code unchanged − Advantage: no changes to the GOS − Disadvantage: can be lower in over-all performance − Common examples: KVM, VirtualBox

• Para-virtualization: − Guest is hypervisor-aware and uses defined API for some services − Guest OS is modified − Advantage: potentially better performance − Disadvantage: modifications to Guest OS (e.g. drivers) − Examples: Freescale Embedded HV

TM External Use 8 Virtualization Use Cases

Cost Reduction Flexibility and Scalability Reliability and Protection

Split Workload Sandboxing Consolidation Data CTRL Plane Plane App App App App OS OS OS OS OS OS Multicore Multicore HW HW HW HW

Dynamic Resource Failover App App Management

OS OS App App App App

OS OS OS OS Multicore HW Multicore Multicore HW HW

App App App App

OS OS OS OS

Multicore Multicore HW HW

TM External Use 9 Virtualization Use Cases – Cost Reduction

• Consolidation : − redeploy multiple discrete systems/domains onto a single multi-core processor App App − Benefits: . Cost effective : bill-of-material, power OS OS . Preserve investment : software re-use

. Improved hardware utilization Multicore . Flexibility HW

TM External Use 10 Virtualization Use Cases – Reliability & Protection

• Sandboxing : App App − Add untrusted software to a system, e.g. operator applications OS OS − Run GPL based software in isolation − Run test software safely Multicore HW − Isolate security-sensitive tasks : access rights control, rule definitions, key management, …

• High availability App App App App − active/standby configuration OS OS OS OS without additional hardware Multicore Multicore HW HW

TM External Use 11 Virtualization Use Cases – Flexibility & Scalability

• Run legacy software / OS on Linux • Add functionality to existing system by dropping in a VM • Use different versions of the Linux kernel • Better resource management − Allocation of physical CPUs / control CPU load − Create/destroy VMs as needed • Maps well to split workload Data CTRL − e.g. control plane, data plane Plane Plane • In-service upgrade OS OS

Multicore HW

TM External Use 12 Virtualization & Hypervisors

• Virtualization – Hardware and software technologies that provide an abstraction layer that enables running multiple operating systems on a single computer system • Hypervisor - Software component that creates and manages virtual machines which can run guest operating systems

App App App App App

Guest Guest Guest Guest Guest OS OS OS App OS OS Hypervisor Host OS

HW HW

• Hypervisor runs “bare metal” • Hypervisor is integrated in Host OS • Reuses OS infrastructure • Host OS runs other applications

TM External Use 13 Virtualization & Partitioning

Partitioning Virtualization

OS OS OS OS

CPU CPU CPU I/O

I/O I/O

Paravirtualization Full Virtualization

Modified Modified OS OS OS OS hypercalls trap

emulation emulation

CPU I/O CPU I/O

TM External Use 14 Device Usage in Virtual Environments

Direct Access Partitionable HW Emulated Para-Virtualized

•Best native performance •Hardware partitioned •Driver in Hypervisor •Driver in Hypervisor •Direct access to hardware •One hardware block •Emulation in Hypervisor •Modified Drivers in •Unmodified Drivers in Guest Guest

OS OS OS OS OS OS OS OS Custom Custom Driver Driver Driver Driver Driver Driver Driver

Emulation Driver Driver

I/O I/O I/O I/O I/O

Hardware/software access Hypercalls Traps

TM External Use 15 Multi-OS Systems

Desktop

Enterprise/Datacenter servers, cloud Mobile computing

Embedded

Aerospace, military (separation kernels)

TM External Use 16 Virtualization Technologies for QorIQ

TM External Use 17 Virtualization Technologies for QorIQ Freescale Embedded KVM OS Virtualization (LXC) Hypervisor ® • Lightweight Hypervisor • Linux Hypervisor • Low Overhead

• Resource Partitioning • Resource Virtualization • Isolation and Resource Control in Linux ® • Para-Virtualization • Resource Oversubscription rd • Decreased Isolation (Kernel • Failover support • 3 Party OSs sharing) • 3rd Party OSs

VM VM

VM VM VM App App App Cont Cont Cont

App App App App App App OS OS

LXC LXC LXC OS OS OS KVM

Embedded Hypervisor Linux Linux ®

Multicore Hardware Multicore Hardware Multicore Hardware

TM External Use 18 Virtualization Technologies for QorIQ

• Linux-based • Delivered with the QorIQ Linux SDK for Power and ARMv7 architecture based products (ARMv8 : planned 2015)

KVM/QEMU LXC libvirt

Linux

TM External Use 19 Freescale Embedded Hypervisor – Overview

• Lightweight hypervisor offering partitioning

VM VM and isolation

• Only one OS per core App App

OS OS • Combination of full virtualization and paravirtualization Embedded Hypervisor

CPU CPU CPU • OS has direct control on high speed IOMMU

Memory Memory peripherals Interrupt Controller I/O I/O Multicore Hardware • Provides good performance and minimal changes to Guest OS

TM External Use 20 KVM / QEMU – Overview

• Open source virtualization technology VM VM based on the Linux kernel QEMU QEMU • Boot operating systems in virtual App App machines alongside Linux applications App OS OS • KVM is a Linux kernel module

Linux KVM • QEMU is a user space emulator that uses KVM for acceleration Multicore Hardware • Run virtual machines alongside Linux applications • No or minimal OS changes required • Virtual I/O capabilities : virtual disk, network interfaces, serial, etc. • Direct/pass thru I/O – assign I/O devices to VMs

TM External Use 21 KVM / QEMU – Details

• QEMU is a user space emulator that uses KVM for acceleration − Uses dedicated threads for vcpus and I/O − KVM leverages hardware virtualization to run guest with higher privileges − MPIC virtual chip emulation in kernel − I/O . Provides dedicated virtio I/O devices and standard drivers in Linux kernel . Uses vfio Linux framework to direct assign physical PCI devices . Direct notifications between I/O threads and KVM using eventfds . Vhost provides virtio emulation and I/O thread and in kernel . Multi-queue virtio devices connected to multi-queue tap devices − Provides services for console, debug, reset, watchdog, etc.

TM External Use 22 KVM / QEMU

• QEMU provides … − Virtual machine setup and initialization − Memory allocation − Virtual I/O services − Debug stub

• KVM provides … − Isolation– using hardware mechanisms and virt extensions − Virtual CPU services − API used by QEMU

• Linux Kernel provides … − Scheduling − Memory management

TM External Use 23 KVM / QEMU Virtual CPUs VM • Each vcpu is a Linux thread QEMU App App − created by QEMU VM init, Virtual boot I/O • Full capabilities of the Linux OS scheduler can be used to manage VCPUs/threads debug vcpu vcpu − CPU affinity − priority I/O Threads Thread Thread − isolcpus − LXC

KVM ® Linux CPU services Kernel

cpu cpu cpu cpu

TM External Use 24 KVM / QEMU Debugging VM • Debug stub in QEMU QEMU allows guest debugging App App VM init, Virtual using GDB boot I/O OS • QEMU monitor shell allows examining VM state debug

Monitor

gdb KVM Linux® Kernel

TM External Use 25 KVM Status

• KVM on QorIQ P-series and T-series − available since QorIQ SDK 1.2 (2012) − upstreamed • KVM on QorIQ Layerscape 1 − 1st release in QorIQ SDK 1.7 (dec-2014) • KVM on ARM is available now in upstream Linux since : − 32-bit : kernel 3.9 − 64-bit : kernel 3.11 • KVM and QEMU have an active open source community developing the technology– led by ARM and Linaro

TM External Use 26 I/O and KVM

TM External Use 27 KVM – I/O Pass Through of PCI Devices

• Assign a physical PCI device to a KVM virtual machine • Device becomes a private resource of the VM • OS gets direct access to device registers • DMA is direct to OS buffers • QEMU mediates all interrupts from the PCI device • QEMU presents PCI device on a virtual PCI bus

VM VM

QEMU QEMU App App OS OS App

vfio Linux

e1000

TM External Use 28 KVM – I/O Pass Through of Platform Devices

• Assign a physical SoC device (e.g. UART) to a VM • Device becomes a private resource of the VM • OS gets direct access to device registers • DMA is direct to OS buffers • Guest sees standard device node in its device tree • QEMU mediates all interrupts from the device

VM VM

QEMU QEMU App App

App OS OS

vfio Linux

UART

TM External Use 29 Pass Through of USB Devices

• Assign a physical USB device or port to a KVM virtual machine • QEMU mediates device access • Guest sees a virtual USB controller on a virtual PCI bus

VM VM

QEMU QEMU libusb App App

App OS OS

Linux

USB Drive

TM External Use 30 Virtio Networking

• Enables sharing of host network interfaces • Host : - Bridge (virtual switch) is connected to physical host interface - QEMU uses tun/tap device connected to the bridge • Guest : - Sees a private virtio network device on PCI bus - Virtio network driver is needed in guest

VM VM

QEMU QEMU App App

App OS OS

tap0 tap1

br0 eth0 Linux

TM External Use 31 Virtio Block

• Give each virtual machine a private storage device • Virtual disk could be single binary image on host file system or logical volume on the host’s disk • Guest sees a private “virtio” network device on PCI bus • Virtio block driver is needed in guest

VM VM

QEMU QEMU App App

App OS OS

Linux

TM External Use 32 Linux Containers

TM External Use 33 Linux Containers (LXC) – Overview

Container Container • OS level virtualization • Guest kernel is the same as the Host App App App kernel, but OS appears isolated • Provides low overhead, lightweight,

Linux ® secure partitioning of Linux applications into different domains • Can control resource utilization of domains– CPU, memory, I/O bandwidth 1 • LinuX Containers is based on kernel 1 7 12 components (cgroups, namespaces) 4 9 15 1 and user-space tools (LXC) 15 17 21 4 7 • KVM virtual machines can be Container 1 1 1 3 run in containers • close to 0% performance overhead • process-level virtualization Container 2 Container 3

TM External Use 34 Libvirt Overview http://libvirt.org/

• A toolkit to interact with the virtualization capabilities of OS-es and hypervisors • Goal : provide common and stable layer sufficient to securely manage domains on a node, possibly remote • Has drivers for KVM/QEMU and Linux containers • Many management applications supported

TM External Use 35 Libvirt Overview http://libvirt.org/

VM VM

App QEMU QEMU

libvirt App App Management protocol libvirtd OS OS Application

KVM Linux®

Hardware

TM External Use 36 Virtualization Hardware Comparison

TM External Use 37 Comparison of Processor Virtualization Capabilities

• ARM, Power, x68 architectures all support similar mechanisms to support virtualization.

Capabilities ARM Power x86

3rd privilege level Yes Yes Yes Extended Address space Yes Yes Yes Hardware guest physical address Yes Yes Yes translation (2-stage) (LRAT) (EPT/NPT) Direct guest interrupt management Yes Yes Yes (x2 APIC) IOMMU Yes Yes Yes (SMMU) (PAMU) (VT-d)

TM External Use 38 Virtualization Features in QorIQ Silicon

• CPU − e500mc / e5500 − 3rd privilege level − Partition ID / extended virtual address space − Key registers duplicated for guests − Direct system calls − Direct external hardware interrupts to guest − LRAT– gphys -> phys translation in hardware • SoC − IOMMU (PAMU) provides isolation from I/O device memory accesses

TM External Use 39 ARM Hardware Virtualization Extensions

• A7/A15/A53/A57 Core − New Hypervisor mode . Privileged operations trap to hypervisor . Banked registers − Two stage address translation . Virtual address (VA) -> Intermediate physical (IPA) . Intermediate (IPA) -> Physical (PA) − Direct system calls − Guest timer in core − Guest GIC (interrupt controller) interface for ACK, EOI

• SoC − IOMMU (SMMU) provides isolation from I/O device memory accesses

TM External Use 40 Networking Virtualization T-Series 10 GbE iNIC

TM External Use 41 QorIQ T2080 Power Optimized Multicore Solution

T1 T2 T1 T2 T1 T2 T1 T2 Processor Power™ Power™ Power™ Power™ • 4x e6500, 64b, 1.2 - 1.8 GHz e6500 e6500 e6500 e6500 • Dual threaded, with 128 b AltiVec 32 KB 32 KB 32 KB 32 KB 32 KB 32 KB 32 KB 32 KB • 2MB shared L2; 256 KB per thread D-Cache I-Cache D-Cache I-Cache D-Cache I-Cache D-Cache I-Cache 64-bit Memory Subsystem

512KB DDR3/3LDDR2/3 fetch

- • 512 KB Platform Cache w/ECC Platform Memory

2MB Banked L2 Pre Cache Controller • 1x DDR3/3L Controller up to 2.1 GHz Security Fuse Processor Coherency Fabric Security Monitor • Up to 1 TB addressability

IFC PAMU PAMU Peripheral Access Mgmt Unit PAMU − 40-bit physical addressing Power Management Real Time Debug • HW Data Prefetching Security Parse, Classify, SDXC/eMMC DCE 5.2 Queue Distribute 8ch 8ch 8ch Watchpoint Switch Fabric

1.0 (XoR, Mgr. DMA DMA DMA Cross

2x DUART Trigger CRC) HiGig/+ DCB High Speed Serial IO 4x I2C

Perf CoreNet

Frame Manager

SPI, GPIO Monitor Trace • 4x PCIe Controllers: Gen1.1/2.0/3.0 SATA2.0 PME Buffer SATA2.0

2x USB2.0 + PHY RMan 4x 4x − 1 with SR-IOV support

PCIe PCIe PCIe PCIe sRIO 2.1 Mgr. 1 / 2.5 / 10G 1 / 2.5G sRIO Aurora − x8 Gen2

8-Lane 10GHz SERDES 8-Lane 8GHz SERDES • 2x sRIO Controller − Type 9 and 11 messaging − Interworking to DPAA via RMan Network IO • 2 SATA 2.0 3Gb/s • Up to 25Gbps Simple PCD each direction • 2 USB 2.0 with PHY • 4x1/10 GE, 4x1 GE or 2.5 Gb/s SGMII • SEC- crypto acceleration • XFI, 10 GBase-KR, XAUI, HiGig, HiGig+, SGMII, RGMII, • DCE - Data Compression 17.5 Gbps 1000Base-KX • PME – Pattern Matching to 10 Gbps

TM External Use 42 T2080 RDB System

TM External Use 43 Target Application: 20 Gb/s iNIC

• Well-balanced device for 20 Gb/s bi-directional application: − FMan moves about 25 Gb/s

− 3x DMA engines move about 20 Gb/s 10G Data − x4 Gen3 or x8 Gen2 PCIe moves ports 32 Gb/s FMan

10G • SR-IOV allows virtual machines on

host to see a private iNIC USB T2080 Mgmt 1G • 15.5 W power fits in 30 W slot-

ports

PCIe EP provided power budget 1G with SR-IOV • Improved PCIe Endpoint capabilities support customization of Device ID, Debug, 2133MT/s code USB DDR3/3L Class Code, and Vendor ID. Driver upload can be stored in Expansion ROM

USB • Offload accelerators for services x8 Gen2 or cards: 10 Gb/s IPSEC or Kasumi, 10 x4 Gen3 Gb/s pattern matching, 17.5 Gb/s data compression • PCIe card reference board available

TM External Use 44 Server with Freescale iNIC T4 iNIC demo Traffic Flow x86 Xeon Linux Platform Enhanced L4-7 Functionality • NFV/SDN/Firewall/ACL VM0 VM1 Openflow • IPSEC • TCP offload Intel DPDK Intel DPDK Controller* • Data Compression Veth-port Veth-port • Deep Packet Inspection • Load Balancing • OpenSSL + record offload • Vendor defined applications 4 1 2 Benefits • Offloading of x86 CPU to Hypervisor increase aggregate with application performance cost PF VF1 VF2 VF3 VFn effectively. • Increase top end server PCIe (SR_IOV) 128VF performance • Scalable iNIC platform 10G Eth performance T2080 to T4240.

L4-7 Reusable software. Openflow 10G Eth • Hardware acceleration for Apps Agent H/W Accel.Packet Data Path, Pattern Matching, Security and Decompression PME Forwarding Engine 10G Eth /Compression, PKC/Record SEC 3 offload. DCE User Space Open vSwitch TM 10G Eth T2/T4 +c290 iNIC External Use 45 * Can be external Freescale iNIC Performance Advantage

• Intel Xeon platforms with a standard NIC require 4 cores of the Xeon CPU to run OVS (3 cores) and VMM (1 core). • With a Freescale iNIC, 1 Xeon core continues running the VMM; the 3 cores running OVS are offloaded to the Freescale CPU. • Additionally Freescale processors contain network application oriented hardware accelerators (security, compress/decompress, pattern matching ) which accelerate key iNIC use cases.

X86 Xeon Linux Platform X86 Xeon Linux Openflo Openflo Platform VM0 VM1 w VM0 VM1 w Intel Intel Controlle Intel Intel Controlle 3 Xeon cores DPDK DPDK DPDK DPDK r* r* Veth-port Veth-port Veth-port Veth-port freed up

Open Openflow 4 vSwitch 1 Agent 2 Packet Forwarding Hypervisor Engine P VF1 VF2 VF3 VFn F PCIe (SR_IOV) 128VF Hypervisor P 10G Eth VF1 VF2 VF3 VFn 10G Eth F

H/W Openflow 10G Eth 10G Eth Accel.Packet Agent Forwarding 10G Eth 10G Eth Engine 3 User Space Open vSwitch 10G Eth 10G Eth T2/T4 +c290 iNIC

TM External Use 46 Q&A

TM External Use 47 TM