KVM PV DEVICES [email protected] Agenda

KVM PV DEVICES Dor.Laor@Qumranet.Com Agenda

Kernel Virtual Machine overview

Paravirtualized drivers intro & brief history

VirtIO

Enhanced VirtIO with KVM support

VM is a regular Linux process. KVM handles cpu & memory virtualization Qemu virtualizes IO devices

Fully virtualized devices performs bad 55 Mbps for RTL Lots of io-exits per packet

Decided to implement a modern e1000 Advantage: Only Qemu coding no guest tools involved Irq coalescing Only 2-3 io-exits per packet Can be the base of user-space PV

But then came Ingo…

VM VM VM HVM VMguest HVM guestR PV I R HVMfrontend guest PV N I driver frontend VM G HVMN guestR driver PV G I R frontend PV PV N I driverfrontend backend PV Qemu G N driver driver backend Qemu G driver PV backend PV Qemu driverbackend Qemu driver

Linux kernel & KVM

PV backend KVM driver

V0 - leveraging Ingo Molnar’s PV code Make loadable module Add HVM support Add NAPI Add memory barriers and improved ring Keep running after performance & stability Merge to the kernel?

Alternatives Xen Polished drivers Xen specific VirtIO was just published.

An API for virtual I/O Implements network & block driver logic Written by Rusty Russell

Motivation Many hypervisors of all types Hard to tune and maintain each one Code reuse – The KVM way ;)

Implementations Lguest KVM Possible (Xen, UML, Qemu, VMware?..)

GuestVM kernel VIRTIO VIRTIO BlK NET

Add_buf Get_buf Sync Detach VM Restart Hypervisor Specifics PV code

Qemu Kick Linux kernel & KVM

PV backend KVM driver

The front end logic is implemented by VirtIO

The backend needs GuestVM kernel VIRTIO VIRTIO BlK NET

Probing & Bus services VM Hypervisor Specifics Enumeration PV code Irq Qemu Parameters (mac,..) Linux kernel & KVM

KVM PV backend Shared memory with remote side driver Hypercalls Host driver/userspace backend

Motivation Increase re-use

GuestVM Allow operation with various bus types kernel VIRTIO VIRTIO BlK NET

Make new devices code smallest VM Hypervisor Specifics PV code

Qemu Components Linux kernel & KVM

Shared memory code PV backend KVM driver With per hypervisor I/O hypercalls Bus (pci, virtual bus) Host backend

VM Blk Net VIRTIO Net VIRTIO BlK Backend Backend

Guest Enhanced VIRTIO VIRTIO kernel

Shared Virtual Bus Hypercalls mem backend VM

Qemu

Linux kernel & KVM

PV KVM backend driver

Status: 162 LOC 195 LOC Interface was enhanced VM Blk Net VIRTIO Net VIRTIO BlK KVM support Backend Backend Guest Enhanced VIRTIO VIRTIO PCI like configuration space kernel Shared added Virtual Bus Hypercalls mem Actually we got re-used… backend VM

Result: Qemu

Makes backend driver tiny Linux kernel & KVM 620 Mbps throughput for network PV HVM Linux guest KVM backend driver Before optimization Userspace backend driver

VirtIO backend Implements VirtIO interface Callbacks to hypervisor and

Code consists of add_buf, get_buf, restart, detach_buf VM Blk Net VIRTIO Net VIRTIO BlK Backend Backend

Only shared memory logic needed Guest Enhanced VIRTIO VIRTIO kernel sync Shared Virtual Bus Hypercalls mem VM Ring logic backend

IO pending hypercall Qemu

be_virtqueue_interrupt handler Linux kernel & KVM

PV KVM backend driver

2.Kernel 1.User VM 3.start_xmit Tcp/ip write Blk Net VIRTIO Net VIRTIO BlK Backend Backend 4.add_buf 12. skb_recv_done Guest Enhanced VIRTIO VIRTIO kernel 6.sync 13. get_buf… Shared Virtual Bus Hypercalls mem 5.add_buf backend VM 11.irq 7.sync 8.Notify hcall Qemu

Linux kernel &10.Trigger KVM irq

Based on lguest

1-1 shared memory

Data structure Page of descriptors for rx, tx. Available pointers page controlled by guest Used pointers page controlled by host

SG list is currently internal to descriptors Descriptors are chained by next pointer

Implements kvm_virtnet_probe for pci bus Creates tx,rx be_new_virtqueue 195 LOC VM Blk Net VIRTIO Net VIRTIO BlK Backend Backend

Guest Enhanced VIRTIO VIRTIO Probes virtnet kernel Shared Virtual Bus Hypercalls mem backend VM

Request_irq Qemu Irq# taken from bus Linux kernel & KVM

PV KVM backend Register hypercall - shared memory pfn driver Device key for enumeration taken from bus

Basic Add readv/writev handlers to Qemu Complete the user-space block device VM Complete migration support Blk Net VIRTIO Net VIRTIO BlK Backend Backend

Guest Enhanced VIRTIO VIRTIO Advanced (also simple) kernel Shared Optimize and stabilize Virtual Bus Hypercalls mem backend VM Add host back end drivers Add virtual bus HVM improvements Qemu Test with PV kernel Linux kernel & KVM

PV KVM backend driver

Thank you ;)