Overview of QorIQ Processor Solutions for Virtualization Features on T2080-based iNIC EUF-NET-T0967 Peter Van Ackeren J A N . 2 0 1 5 TM External Use Session Introduction • This session explores the virtualization solutions available for on QorIQ communication processors • After completing this session you will have become familiar with : − General virtualization principles and use cases − The features, benefits and some details of KVM/QEMU, Linux® containers and libvirt implementations as supplied in the QorIQ Linux SDK − Architecture specific details of Power Architecture and ARM® based platforms as applicable to virtualization use cases TM External Use 2 Introduction TM External Use 3 Traditional System Model • Single OS runs on a single or multi-core hardware system App OS HW TM External Use 4 Multi-processing Usage Models SMP • SMP = Symmetrical Multi-processing CPU − multi-processor homogeneous computer architecture CacheCache Cache Cache Cache Cache − where two or more identical CPU MemoryMemory CPU processors are connected to globally shared main CacheCache memory and I/O CPU I/OI/O − processors could be separate devices, multiple cores in a single device or a mix Single OS − a single OS instance runs on and manages the cores, with shared memory, I/O and interrupt CPU CPU resources − Each processor can run independent processes and Mem ctrl threads I/O − Any idle processor can be assigned any task TM External Use 5 Multi-processing Usage Models AMP • AMP = Asymmetrical Multi-processing − Individual CPUs are dedicated to particular tasks, like running an OS − different software instances run on separate CPUs . e.g. 2 copies of Linux image are loaded at different physical address locations • Both CPUs must cooperate to share the resources − Neither OS can own the whole system − I/O and interrupts are separated − Static configuration of resources TM External Use 6 What is Virtualization ? • Virtualization – an abstraction layer that enables running multiple operating systems on a single system, implemented using hardware and software technologies App App App Linux® RTOS Linux® Hypervisor Hardware • A hypervisor is a software component that creates and manages virtual machines which can run operating systems. TM External Use 7 Types of Virtualization • Full Virtualization: − Virtual machine behaves identically to physical hardware − OS code unchanged − Advantage: no changes to the GOS − Disadvantage: can be lower in over-all performance − Common examples: KVM, VirtualBox • Para-virtualization: − Guest is hypervisor-aware and uses defined API for some services − Guest OS is modified − Advantage: potentially better performance − Disadvantage: modifications to Guest OS (e.g. drivers) − Examples: Freescale Embedded HV TM External Use 8 Virtualization Use Cases Cost Reduction Flexibility and Scalability Reliability and Protection Split Workload Sandboxing Consolidation Data CTRL Plane Plane App App App App OS OS OS OS OS OS Multicore Multicore HW HW HW HW Dynamic Resource Failover App App Management OS OS App App App App OS OS OS OS Multicore HW Multicore Multicore HW HW App App App App OS OS OS OS Multicore Multicore HW HW TM External Use 9 Virtualization Use Cases – Cost Reduction • Consolidation : − redeploy multiple discrete systems/domains onto a single multi-core processor App App − Benefits: . Cost effective : bill-of-material, power OS OS . Preserve investment : software re-use . Improved hardware utilization Multicore . Flexibility HW TM External Use 10 Virtualization Use Cases – Reliability & Protection • Sandboxing : App App − Add untrusted software to a system, e.g. operator applications OS OS − Run GPL based software in isolation − Run test software safely Multicore HW − Isolate security-sensitive tasks : access rights control, rule definitions, key management, … • High availability App App App App − active/standby configuration OS OS OS OS without additional hardware Multicore Multicore HW HW TM External Use 11 Virtualization Use Cases – Flexibility & Scalability • Run legacy software / OS on Linux • Add functionality to existing system by dropping in a VM • Use different versions of the Linux kernel • Better resource management − Allocation of physical CPUs / control CPU load − Create/destroy VMs as needed • Maps well to split workload Data CTRL − e.g. control plane, data plane Plane Plane • In-service upgrade OS OS Multicore HW TM External Use 12 Virtualization & Hypervisors • Virtualization – Hardware and software technologies that provide an abstraction layer that enables running multiple operating systems on a single computer system • Hypervisor - Software component that creates and manages virtual machines which can run guest operating systems App App App App App Guest Guest Guest Guest Guest OS OS OS App OS OS Hypervisor Host OS HW HW • Hypervisor runs “bare metal” • Hypervisor is integrated in Host OS • Reuses OS infrastructure • Host OS runs other applications TM External Use 13 Virtualization & Partitioning Partitioning Virtualization OS OS OS OS CPU CPU CPU I/O I/O I/O Paravirtualization Full Virtualization Modified Modified OS OS OS OS hypercalls trap emulation emulation CPU I/O CPU I/O TM External Use 14 Device Usage in Virtual Environments Direct Access Partitionable HW Emulated Para-Virtualized •Best native performance •Hardware partitioned •Driver in Hypervisor •Driver in Hypervisor •Direct access to hardware •One hardware block •Emulation in Hypervisor •Modified Drivers in •Unmodified Drivers in Guest Guest OS OS OS OS OS OS OS OS Custom Custom Driver Driver Driver Driver Driver Driver Driver Emulation Driver Driver I/O I/O I/O I/O I/O Hardware/software access Hypercalls Traps TM External Use 15 Multi-OS Systems Desktop Enterprise/Datacenter servers, cloud Mobile computing Embedded Aerospace, military (separation kernels) TM External Use 16 Virtualization Technologies for QorIQ TM External Use 17 Virtualization Technologies for QorIQ Freescale Embedded KVM OS Virtualization (LXC) Hypervisor ® • Lightweight Hypervisor • Linux Hypervisor • Low Overhead • Resource Partitioning • Resource Virtualization • Isolation and Resource Control in Linux ® • Para-Virtualization • Resource Oversubscription rd • Decreased Isolation (Kernel • Failover support • 3 Party OSs sharing) rd • 3 Party OSs VM VM VM VM VM App App App Cont Cont Cont App App App App App App OS OS LXC LXC LXC OS OS OS KVM Embedded Hypervisor Linux Linux ® Multicore Hardware Multicore Hardware Multicore Hardware TM External Use 18 Virtualization Technologies for QorIQ • Linux-based • Delivered with the QorIQ Linux SDK for Power and ARMv7 architecture based products (ARMv8 : planned 2015) KVM/QEMU LXC libvirt Linux TM External Use 19 Freescale Embedded Hypervisor – Overview • Lightweight hypervisor offering partitioning VM VM and isolation • Only one OS per core App App OS OS • Combination of full virtualization and para- virtualization Embedded Hypervisor CPU CPU CPU • OS has direct control on high speed IOMMU Memory Memory peripherals Interrupt Controller I/O I/O Multicore Hardware • Provides good performance and minimal changes to Guest OS TM External Use 20 KVM / QEMU – Overview • Open source virtualization technology VM VM based on the Linux kernel QEMU QEMU • Boot operating systems in virtual App App machines alongside Linux applications App OS OS • KVM is a Linux kernel module Linux KVM • QEMU is a user space emulator that uses KVM for acceleration Multicore Hardware • Run virtual machines alongside Linux applications • No or minimal OS changes required • Virtual I/O capabilities : virtual disk, network interfaces, serial, etc. • Direct/pass thru I/O – assign I/O devices to VMs TM External Use 21 KVM / QEMU – Details • QEMU is a user space emulator that uses KVM for acceleration − Uses dedicated threads for vcpus and I/O − KVM leverages hardware virtualization to run guest with higher privileges − MPIC virtual chip emulation in kernel − I/O . Provides dedicated virtio I/O devices and standard drivers in Linux kernel . Uses vfio Linux framework to direct assign physical PCI devices . Direct notifications between I/O threads and KVM using eventfds . Vhost provides virtio emulation and I/O thread and in kernel . Multi-queue virtio devices connected to multi-queue tap devices − Provides services for console, debug, reset, watchdog, etc. TM External Use 22 KVM / QEMU • QEMU provides … − Virtual machine setup and initialization − Memory allocation − Virtual I/O services − Debug stub • KVM provides … − Isolation– using hardware mechanisms and virt extensions − Virtual CPU services − API used by QEMU • Linux Kernel provides … − Scheduling − Memory management TM External Use 23 KVM / QEMU Virtual CPUs VM • Each vcpu is a Linux thread QEMU App App − created by QEMU VM init, Virtual boot I/O • Full capabilities of the Linux OS scheduler can be used to manage VCPUs/threads debug vcpu vcpu − CPU affinity − priority I/O Threads Thread Thread − isolcpus − LXC KVM ® Linux CPU services Kernel cpu cpu cpu cpu TM External Use 24 KVM / QEMU Debugging VM • Debug stub in QEMU QEMU allows guest debugging App App VM init, Virtual using GDB boot I/O OS • QEMU monitor shell allows examining VM state debug Monitor gdb KVM Linux® Kernel TM External Use 25 KVM Status • KVM on QorIQ P-series and T-series − available since QorIQ SDK 1.2 (2012) − upstreamed • KVM on QorIQ Layerscape 1 − 1st release in QorIQ SDK 1.7 (dec-2014) • KVM on ARM is available now in upstream Linux since : − 32-bit : kernel 3.9 − 64-bit : kernel 3.11 • KVM and QEMU have an active open