EUF-NET-T1745 ARM® V8 VIRTUALIZATION FOR LAYERSCAPE MULTICORE COMMUNICATIONS PROCESSORS MARCH 2016 [email protected] - SR. FAE EMEA 22 - טל אביב NXP TECH DAY EXTERNAL USE Agenda • Virtualization Introduction • Layerscape ARM ®v8 Virtualization Status & Roadmap • I/O in KVM Environments − Device Virtualization - virtio − Device Direct-Assignment - VFIO • Q&A 1 EXTERNAL USE VIRTUALIZATION INTRODUCTION 2 EXTERNAL USE Virtualization Technologies for QorIQ Layerscape architecture KVM Linux Containers Embedded Hypervisor • Linux ® Hypervisor • Low Overhead • Lightweight Hypervisor • Resource Virtualization • Isolation and Resource • Resource Partitioning Control in Linux • Resource • Para-Virtualization Oversubscription • Decreased Isolation • Failover support (Kernel sharing) • 3rd Party OSs • 3rd Party OSs VM VM App App App VM VM VM Cont Cont Cont App App App App App App OS OS LXC LXC LXC KVM OS OS OS Linux Linux ® Embedded Hypervisor Multicore Hardware Multicore Hardware Multicore Hardware 3 EXTERNAL USE KVM/QEMU – Overview • KVM/QEMU– open source Virtual Machine 1 Virtual Machine 2 virtualization technology based on the Linux kernel QEMU QEMU App App • KVM is a Linux kernel module • QEMU is a user space emulator that App OS OS uses KVM for acceleration • Run virtual machines alongside Linux KVM Linux applications Multicore • No or minimal OS changes required Hardware • Virtual I/O capabilities • Direct/pass thru I/O – assign I/O devices to VMs 4 EXTERNAL USE KVM/QEMU • QEMU is a user space emulator that uses KVM for acceleration − Uses dedicated threads for vcpus and I/O − KVM leverages hardware virtualization to run guest with higher privileges − Virtual chip emulation in kernel − I/O Provides dedicated virtio I/O devices and standard drivers in Linux kernel Uses VFIO Linux framework to direct assign physical PCI devices Direct notifications between I/O threads and KVM using eventfds vhost provides virtio emulation and I/O thread and in kernel Multi-queue virtio devices connected to multi-queue tap devices − Provides services for console, debug, reset, watchdog, etc 5 EXTERNAL USE Linux Containers • Linu XContainers is based on a Container Container collection of technologies including App App kernel components (cgroups, App namespaces) and user-space tools (LXC). Linux ® • OS level virtualization • Guest kernel is the same as the host kernel, but OS appears isolated 1 • Low overhead, lightweight, secure 1 7 12 partitioning of Linux applications 4 9 15 1 into different domains 15 17 21 4 7 Container 1 1 • Can control resource utilization of 1 3 Container 2 Container 3 domains– CPU, Memory, I/O • close to 0% performance overhead • process-level virtualization bandwidth 6 EXTERNAL USE Libvirt • A toolkit to interact with the Domain Domain virtualization capabilities of Linux (and other OSes / hypervisors) LXC KVM • Goal: to provide a common and LXC Qemu libvirtd stable layer sufficient to securely driver driver manage domains on a node, Libvirt API possibly remote Linux • Has drivers for KVM/QEMU and Linux containers • Many management applications supported Multicore Hardware • http://libvirt.org/ 7 EXTERNAL USE Linux Containers • Platforms supported: all … not platform dependent • Features − Technologies: LXC, Docker, Libvirt − Setups: Busybox system containers, application containers − Networking Shared with host Host interface assignment Virtual Ethernet device pair VLAN / MACVLAN USDPAA − Security: capabilities, seccomp, user namespace • Upstream status: upstream 8 EXTERNAL USE Container Technologies Other Container Flockport DockerHub Technologies Distribution Jails FreeBSD Client LXC virsh docker Zones Solaris OpenVZ Container LXD libvirtd Docker Linux Engine VServer (Daemon) Google Containers Low-level liblxc libvirt_lxc libcontainer API Migration Linux Namespaces cgroups secomp Kernel CRIU 9 EXTERNAL USE Container Comparison LXC/LXD Docker Libvirt • Full system and application containers • Single application virtualization engine • Virtualization high-level API with • Focus on performance and stability based on containers support for containers • Lightweight Linux containers • Focus on ease of use . Easy delivery of • Focus on unification with different apps in a Docker container virtualization technologies • Containers are like VMs with a fully functional OS • Each application has its own container. • Own version of container API – libvirt_lxc “Container as an app ” – tradeoff in order to fit the overall • Comprehensive set of tools for container architecture of libvirt lifecycle management • Docker restricts the container to a single process only • Developed by Red Hat • Data can be saved in a container or outside • Instances are ephemeral. Persistent data is stored in host • LXD allows you to use LXC to create containers on other machines • Trade off in complexity and constraints. Suitable for read only app that is ‘frozen in • LXD aiming to used hardware that state’ “guaranteed isolation of containers ” on the chip level • Container Distribution Platform – Docker Hub • Container distribution platform – Flockport.com • Developed by PaaS providers (dotCloud) • Developed by Ubuntu/Canonical “Blindingly fast virtualization” “Great application delivery mechanism” 10 EXTERNAL USE Containers vs Hypervisors Linux Embedded KVM Containers Hypervisor HW Support Needed? No Yes Yes Overhead Low Yes Yes Isolation Good High High Partitioning Yes Yes Yes Virtualization Yes Yes Yes Yes Yes Linux No Linux Multi OS RTOS (Linux Only) Bare-board Bare-board 3rd Party OSs 3rd Party OSs Features Failover Oversubscription Mainstream Mainstream Private Licensing Open Source Open Source Open Source Mature 11 EXTERNAL USE QORIQ LAYERSCAPE SDK VIRTUALIZATION STATUS & ROADMAP 12 EXTERNAL USE SDK Virtualization Status Technology e500v2 e500mc e5500 e6500 ARMv7 ARMv8 KVM-PPC Up Up Up Up KVM-ARM Leveraged Leveraging FSL Hypervisor Public Sources (bare metal) LXC Leveraged & Fixes Leveraging Leveraged Leveraging Libvirt Leveraged & Fixes • 3 complementary solutions supported on multiple platforms • Focus on enabling core virtualization support, upstreaming and good OOB experience in NXP SDK • I/O performance optimization in progress 13 EXTERNAL USE Layerscape ARM ®v8 Virtualization Roadmap LS1043 BSP 0.5 LS2 EAR 0.6 SDK 2.0 [KVM] Basic KVM [LXC] LXC with seccomp [KVM] [KVM] virtio [Docker] Docker engine • DPAA 2.x Direct Assignment v1 (VFIO) [LXC] LXC Support • PCI Device Assignment (VFIO-PCI) LS2080 SDK 1.0 Benchmarking Upstreaming Upstreaming [KVM] Kernel 4.1 & QEMU 2.4 [KVM] Benchmarking • MSI support [KVM] DPAA 2.x Direct [KVM] DPAA 2.x Direct • vhost-net, vhost-blk dataplane Assignment v2 (VFIO) Assignment v2 (VFIO) [LXC] LXC [Libvirt] Libvirt POC POC LS2 POC POC [KVM] [KVM] [KVM] [KVM] PCI device assignment to • LS2 User Space (e.g ODP) • Performance improvements • DPAA 2.x Direct Assignment KVM VMs direct assignment in guest • Direct assignment of with interrupts [LXC] CRIU Demo • Performance improvements platform devices • KVM-RT on LS2 • LE Guest on BE Host • virtio-crypto demo [Docker] Docker engine on LS2 • BE Guest on LE Host • USB pass-through • Direct assignment of SEC • Shared-memory (ivshmem) • DPAA 2.x vhost-user optimizations • Live Migration Demo 2015 2016 1Q 2Q 3Q Color Legend Italic features depend on upstream support Current Release Released 14 EXTERNAL USE Roadmap Date Major Release KVM - Out-Of-Box RFS Enablement • Required components : − KVM support enabled in kernel − Guest image − Guest root file system − QEMU Kernel config Guest rootfs Guest image QEMU fsl-image-core NO NO NO NO fsl-image-full NO NO NO YES fsl-image-virt NO YES YES YES 15 EXTERNAL USE I/O IN KVM ENVIRONMENTS 16 EXTERNAL USE I/O Virtualization - Performance vs Flexibility Non-Interposable Interposable Bare Metal Trend Direct Assignment (VFIO) vhost- vhost user optim vhost virtio (para-virtualized) No Guest Modifications Emulated I/O Scalability and Performance and Scalability Flexibility 17 EXTERNAL USE Device Usage in Virtual Environments Direct Access Partitionable Emulated Para-Virtualized HW • Fast native performance • Hardware partitioned • Driver in Hypervisor • Driver in Hypervisor • Direct access to hardware • One hardware block • Emulation in Hypervisor • Modified Drivers in • Unmodified Drivers in Guest Guest OS OS OS OS OS OS OS OS Custom Custom Driver Driver Driver Driver Driver Driver Driver Emulation Driver Driver I/O I/O I/O I/O I/O Hardware/software access Hypercalls Traps 18 EXTERNAL USE DEVICE VIRTUALIZATION VIRTIO 19 EXTERNAL USE virtio virtio- virtio- virtio- virtio- virtio- virtio- console blk pci net scsi balloon Guest • Device abstraction layer of para- Linux virtio virtualized hypervisor transport −Standard for VMs/VNFs Host Linux / QEMU −Appearance as physical devices virtio back-end drivers −Uses standard virtual drivers and discovery mechanisms virtio-net : Ethernet virtual driver virtio frontend virtio frontend vhost-net : optimizes Ethernet virtual driver by eliminating QEMU context switch virtio-pci • Backend drivers are vendor specific in host Linux; transparent to VM/VNFs Sources: https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Virtualization_Tuning_and_Optimization_Guide 20 EXTERNAL USE virtio • Device facilities − Device status field / Feature bits / Device Configuration space / virtqueues • Transport protocols: PCI, MMIO • virtio specification, defined by OASIS technical committee − Straightforward - use normal bus mechanisms for interrupts and DMA − Efficient - rings