VIRTUALIZATION ON MULTICORE MPUS

[email protected] - SR. FAE EMEA

EXTERNAL USE Agenda

• Introduction

− KVM/QEMU

− Containers

−

• I/O in KVM Environments

− Device Virtualization - virtio

− Device Direct-Assignment - VFIO

. including a whirl wind introduction of VFIO integration of the QorIQ LS-series Data Path Acceleration Architecture (DPAA2)

• Q&A

1 EXTERNAL USE VIRTUALIZATION INTRODUCTION

2 EXTERNAL USE What is … ? Virtualization … Hardware and software technologies that provide an abstraction layer that enables running multiple operating systems on a single computer system

App App App App App App App App App App App App

Linux RTOS Linux RTOS Linux

Hypervisor

A … is a software component that creates and manages virtual machines which can run operating systems

3 EXTERNAL USE Basic Use Cases

• Consolidation / Migration − Consolidate separate (legacy) systems onto one hardware platform − Multiple operating systems/partitions on a single multi-core chip − Multiple homogeneous operating systems on multiple cores − Preserve investment in software − Run legacy software alongside new software − Add Linux services to a non-Linux platform

• Divided workload (e.g. control plane, data plane) − Multiple operating systems, possibly heterogeneous, need to work securely and seamlessly together − Isolation mechanisms are needed for safety, robustness − Efficient inter-partition communication mechanisms are needed for cooperation

4 EXTERNAL USE Basic Use Cases

• Isolate or sandbox untrusted software − Isolate untrusted operating systems: Proprietary OS + open OS (eg Linux) − Isolate end-user installed software − Software under test

• Security − Secure partition for sensitive security tasks (e.g. access rights control, rule definitions, key storage/management)

• High availability − active/standby configuration without additional hardware

5 EXTERNAL USE SDN/NFV Use Cases

• Specialized processing functions (firewall, …) are now commonly implemented in virtual OS instances call Virtual Network Functions (VNFs) • Full processing sequence (e.g. data plane) is implemented through "service chaining" multiple VMs • This requires efficient, high-performance I/O between VMs (network) or between VMs and (storage, PCIe) • Originated in cloud and data center, now strongly expanding in networking

6 EXTERNAL USE Virtualization Technologies Approaches

KVM Linux Containers Embedded Hypervisor (OS Virtualization)

• Linux ® Hypervisor • Low Overhead • Lightweight Hypervisor

• Resource Virtualization • Isolation and Resource • Resource Partitioning Control in Linux • Resource • Para-Virtualization Oversubscription • Decreased Isolation (Kernel sharing) • Failover support • 3rd Party OSs • 3rd Party OSs

VM VM

App App App VM VM VM Cont Cont Cont

App App App App App App OS OS

LXC LXC LXC KVM OS OS OS

Linux Linux ® Embedded Hypervisor

Multicore Hardware Multicore Hardware Multicore Hardware

7 EXTERNAL USE KVM/QEMU – Overview

• KVM/QEMU– open source virtualization 1 Virtual Machine 2 technology based on the

QEMU QEMU • KVM is a Linux kernel module App App • QEMU is a that

App OS OS uses KVM for acceleration • Run virtual machines alongside Linux KVM Linux applications

Multicore • No or minimal OS changes required Hardware • Virtual I/O capabilities • Direct/pass thru I/O – assign I/O devices to VMs

8 EXTERNAL USE KVM/QEMU

• QEMU is a user space emulator that uses KVM for acceleration − Uses dedicated threads for vcpus and I/O − KVM leverages to run guest with higher privileges − Virtual chip emulation in kernel − I/O . Provides dedicated virtio I/O devices and standard drivers in Linux kernel . Uses VFIO Linux framework to direct assign physical PCI devices . Direct notifications between I/O threads and KVM using eventfds . vhost provides in-kernel virtio emulation . Multi-queue virtio devices connected to multi-queue tap devices − Provides services for console, debug, reset, watchdog, etc

9 EXTERNAL USE Linux Containers • LinuX Containers : Low overhead, Containe Containe r r lightweight, secure partitioning of Linux App App App applications into different domains • Guest kernel == Host kernel Linux ® … but OS appears isolated  OS level virtualization • Based on a collection of technologies

1 including kernel components (, 1 7 12 namespaces) and user-space tools (LXC).

4 9 15 1 • Can control resource utilization of domains – 15 17 21 4 7

Container 1 1 CPU, Memory, I/O bandwidth 1 3

Container 2 Container 3 • close to 0% performance overhead • process-level virtualization • Not platform dependent

10 EXTERNAL USE Container Technologies

Other Container Flockport DockerHub Technologies Distribution

Jails FreeBSD

Client LXC virsh Zones Solaris

OpenVZ Container LXD libvirtd Docker Linux Engine VServer () Google Containers

Low-level liblxc libvirt_lxc libcontainer API

Migration cgroups Kernel CRIU

11 EXTERNAL USE Libvirt

• A toolkit to interact with the Domain Domain virtualization capabilities of Linux (and other OSes / ) LXC KVM • Goal: to provide a common and LXC Qemu libvirtd stable layer sufficient to securely driver driver manage domains on a node, Libvirt API possibly remote

Linux • Has drivers for KVM/QEMU and Linux containers • Many management applications supported Multicore Hardware • http://libvirt.org/

12 EXTERNAL USE DEVICE AND I/O VIRTUALIZATION

13 EXTERNAL USE Device Usage in Virtual Environments

Direct Access Partitionable Emulated Para-Virtualized HW HW • Fast native performance • Hardware partitioned • Driver in Hypervisor • Direct access to hardware • One hardware block • Driver in Hypervisor • Modified Drivers in • Emulation in Hypervisor Guest • Unmodified Drivers in OS OS OS OS OS OS GuestOS OS Custom Custom Driver Driver Driver Driver Driver Driver Driver

Emulation Driver Driver

I/O I/O I/O I/O I/O

14 EXTERNAL USE I/O Virtualization - Performance vs Flexibility

Bare Metal Trend Direct Assignment (VFIO) vhost- vhost user optim vhost

virtio (para-virtualized) No Guest Modifications

Emulated

I/O Scalability Performance and Scalability

Flexibility

15 EXTERNAL USE DEVICE VIRTUALIZATION VIRTIO

16 EXTERNAL USE virtio virtio- virtio- virtio- virtio- virtio- virtio- console blk pci net scsi balloon

Guest • Device abstraction layer of para- Linux virtio virtualized hypervisor transport −Standard for VMs/VNFs Host Linux / QEMU −Appearance as physical devices virtio back-end drivers −Uses standard virtual drivers and discovery mechanisms . : virtual driver virtio-net virtio frontend virtio frontend . vhost-net : optimizes Ethernet virtual driver by eliminating QEMU context switch . virtio-pci • Backend drivers are vendor specific in host