KVM PV DEVICES

[email protected]

1 Agenda

 Introduction & brief history

 VirtIO

 Enhanced VirtIO with KVM support

 Further implementation

2 General & history  Fully virtualized devices performs bad  55 Mbps for RTL  Lots of io-exits per packet

 Decided to implement a modern e1000  Advantage:  Only Qemu coding  no guest tools involved  Irq coalescing  Only 2-3 io-exits per packet  Can be the base of user-space PV

 But then came Ingo…

3 PV driver architecture

VM VM VM HVM VMguest HVM guestR PV I R HVMfrontend guest PV N I driver frontend VM G HVMN guestR driver PV G I R frontend PV PV N I driver frontend backend PV Qemu G N driver driver backend Qemu G driver PV backend PV Qemu driver backend Qemu driver

Linux kernel & KVM

PV backend KVM driver

4 General & history  V0 - leveraging Ingo Molnar’s PV code  Make loadable module  Add HVM support  Add NAPI  Add memory barriers and improved ring  Keep running after performance & stability  Merge to the kernel?

 Alternatives   Polished drivers  Xen specific  VirtIO was just published.

5 PV driver requirements  Close to native performance

 Merge with the kernel

 Leverage existing code

 Have usermode implementation  Block device for qcow, vmdk formats  Ability to function without KVM (-no-kvm)

 Ability to run HVM guests

 Robust code  Easily add drivers

6 VirtIO  An API for virtual I/O  Implements network & block driver logic  Written by

 Motivation  Many of all types  Hard to tune and maintain each one  Code reuse – The KVM way ;)

 Implementations  Lguest  KVM  Possible (Xen, UML, Qemu, VMware?..)

7 VirtIO

GuestVM kernel VIRTIO VIRTIO BlK NET

Add_buf Get_buf Sync VM Restart Detach Specifics PV code

Kick Qemu

Linux kernel & KVM

PV backend KVM driver

 scatterlist sg[] for skb/blk_req data 8 VirtIO – Hypervisor specifics  The front end logic is implemented by VirtIO

GuestVM kernel  The backend needs VIRTIO VIRTIO BlK NET

VM  Probing & Bus services Hypervisor Specifics  Enumeration PV code Qemu

 Irq Linux kernel & KVM

 Parameters (mac,..) KVM PV backend driver  Shared memory with remote side  Hypercalls  Host driver/userspace backend

9 Enhanced VirtIO  Motivation  Increase re-use  Allow operation with various bus types GuestVM kernel VIRTIO VIRTIO BlK  Make new devices code smallest NET VM Hypervisor Specifics PV code

Qemu

 Components Linux kernel & KVM

KVM PV backend  Shared memory code driver  With per hypervisor I/O hypercalls  Bus (pci, virtual bus)  Host backend

10 Enhanced VirtIO

VM Blk Net VIRTIO Net VIRTIO BlK Backend Backend

Guest Enhanced VIRTIO VIRTIO kernel

Shared Virtual Bus Hypercalls mem backend VM

Qemu

Linux kernel & KVM

PV KVM backend driver

11 Enhanced VirtIO  Status: Interface needs polishing 162 LOC 195 LOC  VM Blk Net VIRTIO Net VIRTIO BlK  KVM support Backend Backend Guest Enhanced VIRTIO VIRTIO  PCI bus support kernel Shared Virtual Bus Hypercalls mem backend VM  Result: Qemu  Makes backend driver tiny  620 Mbps throughput for Linux kernel & KVM PV network KVM backend driver  HVM Linux guest  Before optimization  Userspace backend driver 12 Enhanced VirtIO – shared memory backend

 VirtIO backend  Implements VirtIO interface  Callbacks to hypervisor and

 Code consists of VM Blk Net VIRTIO Net VIRTIO BlK  add_buf, get_buf, restart, detach_buf Backend Backend Guest Enhanced VIRTIO VIRTIO  Only shared memory logic needed kernel Shared Virtual Bus Hypercalls mem  sync backend VM  Ring logic Qemu  IO pending hypercall Linux kernel & KVM  be_virtqueue_interrupt handler PV KVM backend driver

13 Enhanced VirtIO – Day in a life of packet

2.Kernel 1.User VM 3.start_xmit Tcp/ip write Blk Net VIRTIO Net VIRTIO BlK Backend Backend 4.add_buf 12. skb_recv_done Guest Enhanced VIRTIO VIRTIO kernel 6.sync 13. get_buf… Shared Virtual Bus Hypercalls mem 5.add_buf backend VM 11.irq 7.sync 8.Notify hcall Qemu

Linux kernel &10.Trigger KVM irq

PV KVM backend driver 9.The pkts travels to tap/backend host device 14 Enhanced VirtIO – shared memory details

 Based on lguest

 1-1 shared memory

 Data structure  Page of descriptors for rx, tx.  Available pointers page controlled by guest  Used pointers page controlled by host

 SG list is currently internal to descriptors  Descriptors are chained by next pointer

15 Enhanced VirtIO – network be driver  Implements kvm_virtnet_probe for pci bus  Creates tx,rx be_new_virtqueue VM Blk Net VIRTIO Net VIRTIO BlK Backend Backend

Guest Enhanced VIRTIO VIRTIO 195 LOC kernel

Shared Probes virtnet Virtual Bus Hypercalls mem  VM backend

Qemu  Request_irq Linux kernel & KVM

 Irq# taken from bus PV KVM backend driver

 Register hypercall - shared memory pfn  Device key for enumeration taken from bus

16 Further work  Basic  Add readv/writev handlers to Qemu  Complete the user-space block device

 Update with VirtIO gso. VM Blk Net VIRTIO Net VIRTIO BlK  Complete migration support Backend Backend

Guest Enhanced VIRTIO VIRTIO kernel

Shared  Advanced (also simple) Virtual Bus Hypercalls mem VM  Publish the enhanced interface backend  Optimize and stabilize Qemu  Add host back end drivers

 Add virtual bus Linux kernel & KVM

 HVM improvements PV KVM backend  Test with PV kernel driver

17 Q&A

Thank you ;)

18