KVM PV DEVICES [email protected] 1 Agenda Introduction & brief history VirtIO Enhanced VirtIO with KVM support Further implementation 2 General & history Fully virtualized devices performs bad 55 Mbps for RTL Lots of io-exits per packet Decided to implement a modern e1000 Advantage: Only Qemu coding no guest tools involved Irq coalescing Only 2-3 io-exits per packet Can be the base of user-space PV But then came Ingo… 3 PV driver architecture VM VM VM HVM VMguest HVM guestR PV I R HVMfrontend guest PV N I driver frontend VM G HVMN guestR driver PV G I R frontend PV PV N I driver frontend backend PV Qemu G N driver driver backend Qemu G driver PV backend PV Qemu driver backend Qemu driver Linux kernel & KVM PV backend KVM driver 4 General & history V0 - leveraging Ingo Molnar’s PV code Make loadable module Add HVM support Add NAPI Add memory barriers and improved ring Keep running after performance & stability Merge to the kernel? Alternatives Xen Polished drivers Xen specific VirtIO was just published. 5 PV driver requirements Close to native performance Merge with the kernel Leverage existing code Have usermode implementation Block device for qcow, vmdk formats Ability to function without KVM (-no-kvm) Ability to run HVM guests Robust code Easily add drivers 6 VirtIO An API for virtual I/O Implements network & block driver logic Written by Rusty Russell Motivation Many hypervisors of all types Hard to tune and maintain each one Code reuse – The KVM way ;) Implementations Lguest KVM Possible (Xen, UML, Qemu, VMware?..) 7 VirtIO GuestVM kernel VIRTIO VIRTIO BlK NET Add_buf Get_buf Sync VM Restart Detach Hypervisor Specifics PV code Kick Qemu Linux kernel & KVM PV backend KVM driver scatterlist sg[] for skb/blk_req data 8 VirtIO – Hypervisor specifics The front end logic is implemented by VirtIO GuestVM kernel The backend needs VIRTIO VIRTIO BlK NET VM Probing & Bus services Hypervisor Specifics Enumeration PV code Qemu Irq Linux kernel & KVM Parameters (mac,..) KVM PV backend driver Shared memory with remote side Hypercalls Host driver/userspace backend 9 Enhanced VirtIO Motivation Increase re-use Allow operation with various bus types GuestVM kernel VIRTIO VIRTIO BlK Make new devices code smallest NET VM Hypervisor Specifics PV code Qemu Components Linux kernel & KVM KVM PV backend Shared memory code driver With per hypervisor I/O hypercalls Bus (pci, virtual bus) Host backend 10 Enhanced VirtIO VM Blk Net VIRTIO Net VIRTIO BlK Backend Backend Guest Enhanced VIRTIO VIRTIO kernel Shared Virtual Bus Hypercalls mem backend VM Qemu Linux kernel & KVM PV KVM backend driver 11 Enhanced VirtIO Status: Interface needs polishing 162 LOC 195 LOC VM Blk Net VIRTIO Net VIRTIO BlK KVM support Backend Backend Guest Enhanced VIRTIO VIRTIO PCI bus support kernel Shared Virtual Bus Hypercalls mem backend VM Result: Qemu Makes backend driver tiny 620 Mbps throughput for Linux kernel & KVM PV network KVM backend driver HVM Linux guest Before optimization Userspace backend driver 12 Enhanced VirtIO – shared memory backend VirtIO backend Implements VirtIO interface Callbacks to hypervisor and Code consists of VM Blk Net VIRTIO Net VIRTIO BlK add_buf, get_buf, restart, detach_buf Backend Backend Guest Enhanced VIRTIO VIRTIO Only shared memory logic needed kernel Shared Virtual Bus Hypercalls mem sync backend VM Ring logic Qemu IO pending hypercall Linux kernel & KVM be_virtqueue_interrupt handler PV KVM backend driver 13 Enhanced VirtIO – Day in a life of packet 2.Kernel 1.User VM 3.start_xmit Tcp/ip write Blk Net VIRTIO Net VIRTIO BlK Backend Backend 4.add_buf 12. skb_recv_done Guest Enhanced VIRTIO VIRTIO kernel 6.sync 13. get_buf… Shared Virtual Bus Hypercalls mem 5.add_buf backend VM 11.irq 7.sync 8.Notify hcall Qemu Linux kernel &10.Trigger KVM irq PV KVM backend driver 9.The pkts travels to tap/backend host device 14 Enhanced VirtIO – shared memory details Based on lguest 1-1 shared memory Data structure Page of descriptors for rx, tx. Available pointers page controlled by guest Used pointers page controlled by host SG list is currently internal to descriptors Descriptors are chained by next pointer 15 Enhanced VirtIO – network be driver Implements kvm_virtnet_probe for pci bus Creates tx,rx be_new_virtqueue VM Blk Net VIRTIO Net VIRTIO BlK Backend Backend Guest Enhanced VIRTIO VIRTIO 195 LOC kernel Shared Probes virtnet Virtual Bus Hypercalls mem VM backend Qemu Request_irq Linux kernel & KVM Irq# taken from bus PV KVM backend driver Register hypercall - shared memory pfn Device key for enumeration taken from bus 16 Further work Basic Add readv/writev handlers to Qemu Complete the user-space block device Update with VirtIO gso. VM Blk Net VIRTIO Net VIRTIO BlK Complete migration support Backend Backend Guest Enhanced VIRTIO VIRTIO kernel Shared Advanced (also simple) Virtual Bus Hypercalls mem VM Publish the enhanced interface backend Optimize and stabilize Qemu Add host back end drivers Add virtual bus Linux kernel & KVM HVM improvements PV KVM backend Test with PV kernel driver 17 Q&A Thank you ;) 18.