VMs, Unikernels and Containers: Experiences on the Performance of Virtualizaon Technologies Felipe Huici, Filipe Manco, Jose Mendes, Simon Kuenzer NEC Europe Ltd. (Heidelberg) In the Beginning… VM In the Beginning… “Tinyfied VMs” VM In the Beginning… “Tinyfied VMs” unikernels VM In the Beginning… “Tinyfied VMs” containers unikernels VM In the Beginning… “Tinyfied VMs” containers unikernels VM Virt. Technology Benchmarking • Metrics: – VM Image and memory consump=on – VM creaon =me – Delay – Throughput Virt. Technology Benchmarking • Metrics: – VM Image and memory consump=on – VM creaon =me – Delay – Throughput higher lower overhead overhead Virt. Technology Benchmarking • Metrics: – VM Image and memory consump=on – VM creaon =me – Delay – Throughput higher lower overhead overhead Virt. Technology Benchmarking • Metrics: – VM Image and memory consump=on – VM creaon =me – Delay – Throughput higher lower overhead overhead Virt. Technology Benchmarking • Metrics: – VM Image and memory consump=on – VM creaon =me – Delay – Throughput higher lower overhead overhead Virt. Technology Benchmarking • Metrics: – VM Image and memory consump=on – VM creaon =me – Delay – Throughput higher lower overhead overhead Virt. Technology Benchmarking • Metrics: – VM Image and memory consump=on – VM creaon =me – Delay – Throughput ? higher lower overhead overhead Virt. Technology Benchmarking • Metrics: – VM Image and memory consump=on – VM creaon =me – Delay – Throughput higher lower overhead overhead Virt. Technology Benchmarking • Metrics: – VM Image and memory consump=on – VM creaon =me – Delay – Throughput ? higher lower overhead overhead Virtualizaon Technology Benchmarking • Metrics: – VM image and memory consump=on: ls, top, xl – VM creaon =me: SYN flood + RST detec=on – Throughput: iperf, guest to host (TCP traffic) – RTT: ping flood • VM-based tests run on both Xen and KVM • Hardware: x86_64 server with an Intel Xeon E5-1630 v3 3.7GHz CPU (4 cores), 32GB RAM. Virtualizaon Technologies • “Standard” VM – Standard Debian-based Linux VM • “Tinyfied” VM – Tinyx, based on Linux kernel/busybox • Unikernel – On Xen: MiniOS + miniperf – On KVM: OSv + iperf • Containers – Docker Virtualizaon Technologies • “Standard” VM – Standard Debian-based Linux VM • “Tinyfied” VM – Tinyx, based on Linux kernel/busybox • Unikernel – On Xen: MiniOS + miniperf – On KVM: OSv + iperf • Containers – Docker Standard VM: Applicaon on Top of Distro User Applica5on 3rd Party Applica5ons Libraries Services Kernel Most of the VM not Used… Nginx User Applica5on memcached bash 3rd Party Applica5ons libssl Libraries libc ssh init Services ext4 netfront blkfront Kernel Tinyx: Keep Only What’s Needed Nginx User Applica5on memcached bash 3rd Party Applica5ons libssl Libraries libc ssh init Services ext4 netfront blkfront Kernel Tinyx: Taylor-made Distro Nginx User Applica5on memcached bash 3rd Party Applica5ons libssl Libraries libc ssh init Services netfront blkfront Kernel ext4 Tinyx: Taylor-made Distro Nginx User Applica5on memcached bash 3rd Party Applica5ons libssl Libraries libc ssh init Services netfront blkfront Kernel ext4 Tinyx: Taylor-made Distro Nginx User Applica5on memcached bash 3rd Party Applica5ons libssl Libraries libc ssh init Services netfront blkfront Kernel ext4 Tinyx: Taylor-made Distro Nginx User Applica5on memcached bash 3rd Party Applica5ons libssl Libraries libc ssh init Services netfront blkfront Kernel ext4 Tinyx: Taylor-made Distro ▌ Keep only the Nginx User Applica5on necessary bits memcached and pieces bash 3rd Party Applica5ons l Specialized kernel build containing only the necessary modules libssl Libraries l Root filesystem libc populated with only necessary ssh services, libraries and 3rd party init Services applications netfront blkfront Kernel ext4 Virtualizaon Technologies • “Standard” VM – Standard Debian-based Linux VM • “Tinyfied” VM – Tinyx, based on Linux kernel/busybox • Unikernel – On Xen: MiniOS + miniperf – On KVM: OSv + iperf • Containers – Docker Virtualizaon Technologies • “Standard” VM – Standard Debian-based Linux VM • “Tinyfied” VM – Tinyx, based on Linux kernel/busybox • Unikernel – On Xen: MiniOS + miniperf – On KVM: OSv + iperf • Containers – Docker What’s a Unikernel? • Specialized VM: single applicaon + minimalis=c OS • Single address space, co-operave scheduler so low overheads What’s a Unikernel? • Specialized VM: single applicaon + minimalis=c OS 1 pp a app N app 2 • Single address space, USER SPACE co-operave scheduler so low overheads KERNEL SPACE driverN driver1 driver2 GENERAL-PURPOSE OPERATING SYSTEM (e.G., Linux, FreeBSD) What’s a Unikernel? • Specialized VM: single applicaon + minimalis=c OS 1 pp a app N app 2 • Single address space, USER SPACE co-operave scheduler so low overheads app SINGLE ADDRESS SPACE KERNEL SPACE driverN driver1 driver2 vdriver2 Vdriver1 GENERAL-PURPOSE MINIMALISTIC OPERATING SYSTEM OPERATING SYSTEM (e.G., Linux, FreeBSD) (e.G., MiniOS, OSv) Unikernels for Benchmarking apps guest OS On Xen Xen Unikernels for Benchmarking apps iperf guest mini OS OS On Xen Xen Xen Unikernels for Benchmarking apps iperf guest mini OS OS On Xen Xen Xen apps guest OS On KVM KVM Unikernels for Benchmarking apps iperf guest mini OS OS On Xen Xen Xen apps iperf guest OSv OS On KVM KVM KVM Nota Bene… • Our unikernel numbers include op=mizaons to the underlying virtualizaon plaorms (Xen, KVM) – Toolstacks – Back-end stores – Hotplug scripts – Network drivers (on Xen Tx) • No =me to go over these… RESULTS Image Size, Memory Usage (log scale) 1000 913 913 img size mem usage 100 112 82 61 52 MB 31 30 10 12 8 3.8 3.7 3.5 2 1 Boot Times (log scale) 10000 6500 2988 ) 1000 1711 1081 ms 431 330 100 Boot Time ( 31 10 1 RTT 40 35 30 34 ) 25 ms 20 19 18 RTT ( 15 15 10 5 9 5 4 0 Throughput 60 Tx 50 Rx 40 30 Throughput (Gb/s) 20 10 0 Conclusions • Common lore: VMs provide good isolaon but are heavyweight – Results with standard VMs confirm this • Containers provide lighter-weight virtualizaon – But nyfied VMs and especially unikernels yield comparable performance Conclusions • Common lore: VMs provide good isolaon but are heavyweight – Results with standard VMs confirm this • Containers provide lighter-weight virtualizaon – But nyfied VMs and especially unikernels yield comparable performance Potenal Contribuons to dra-natarajan-nfvrG-containers-for-nfv-01 2.1.1 Challenges - VNF provisioning me - Run=me performance (throughput, scaling up/down) 3. Benefits of Containers - Service agility vs VMs - Containers have beoer run=me performance - Auto-scaling of VNFs - Cross-VNF compability: container unikernel/minimalis=c distro - Overall performance: VMs -25% throughput vs containers 5. Conclusion - Containers have significant advantages vs hypervisor-based solu=ons .