VMs, Unikernels and Containers: Experiences on the Performance of Virtualizaon Technologies

Felipe Huici, Filipe Manco, Jose Mendes, Simon Kuenzer NEC Europe Ltd. (Heidelberg) In the Beginning…

VM In the Beginning…

“Tinyfied VMs” VM In the Beginning…

“Tinyfied VMs”

unikernels VM In the Beginning…

“Tinyfied VMs” containers unikernels VM In the Beginning…

“Tinyfied VMs” containers unikernels VM Virt. Technology Benchmarking

• Metrics: – VM Image and memory consumpon – VM creaon me – Delay – Throughput Virt. Technology Benchmarking

• Metrics: – VM Image and memory consumpon – VM creaon me – Delay – Throughput

higher lower overhead overhead Virt. Technology Benchmarking

• Metrics: – VM Image and memory consumpon – VM creaon me – Delay – Throughput

higher lower overhead overhead Virt. Technology Benchmarking

• Metrics: – VM Image and memory consumpon – VM creaon me – Delay – Throughput

higher lower overhead overhead Virt. Technology Benchmarking

• Metrics: – VM Image and memory consumpon – VM creaon me – Delay – Throughput

higher lower overhead overhead Virt. Technology Benchmarking

• Metrics: – VM Image and memory consumpon – VM creaon me – Delay – Throughput

higher lower overhead overhead Virt. Technology Benchmarking

• Metrics: – VM Image and memory consumpon – VM creaon me – Delay – Throughput ? higher lower overhead overhead Virt. Technology Benchmarking

• Metrics: – VM Image and memory consumpon – VM creaon me – Delay – Throughput

higher lower overhead overhead Virt. Technology Benchmarking

• Metrics: – VM Image and memory consumpon – VM creaon me – Delay – Throughput ? higher lower overhead overhead Virtualizaon Technology Benchmarking • Metrics: – VM image and memory consumpon: ls, top, xl – VM creaon me: SYN flood + RST detecon – Throughput: iperf, guest to host (TCP traffic) – RTT: ping flood

• VM-based tests run on both and KVM

• Hardware: x86_64 server with an Intel Xeon E5-1630 v3 3.7GHz CPU (4 cores), 32GB RAM.

Virtualizaon Technologies

• “Standard” VM – Standard Debian-based VM • “Tinyfied” VM – Tinyx, based on Linux kernel/busybox • – On Xen: MiniOS + miniperf – On KVM: OSv + iperf • Containers – Docker

Virtualizaon Technologies

• “Standard” VM – Standard Debian-based Linux VM • “Tinyfied” VM – Tinyx, based on Linux kernel/busybox • Unikernel – On Xen: MiniOS + miniperf – On KVM: OSv + iperf • Containers – Docker

Standard VM: Applicaon on Top of Distro

User Applicaon

3rd Party Applicaons

Libraries

Services

Kernel Most of the VM not Used…

Nginx User Applicaon

memcached bash 3rd Party Applicaons

libssl Libraries libc

ssh init Services

ext4 netfront blkfront Kernel Tinyx: Keep Only What’s Needed

Nginx User Applicaon

memcached bash 3rd Party Applicaons

libssl Libraries libc

ssh init Services

ext4 netfront blkfront Kernel Tinyx: Taylor-made Distro

Nginx User Applicaon memcached bash 3rd Party Applicaons

libssl Libraries libc

ssh init Services

netfront blkfront Kernel ext4 Tinyx: Taylor-made Distro

Nginx User Applicaon memcached bash 3rd Party Applicaons

libssl Libraries libc

ssh init Services

netfront blkfront Kernel ext4 Tinyx: Taylor-made Distro

Nginx User Applicaon memcached bash 3rd Party Applicaons

libssl Libraries libc

ssh init Services

netfront blkfront Kernel ext4 Tinyx: Taylor-made Distro

Nginx User Applicaon memcached bash 3rd Party Applicaons

libssl Libraries libc

ssh init Services

netfront blkfront Kernel ext4 Tinyx: Taylor-made Distro

▌ Keep only the Nginx User Applicaon necessary bits memcached and pieces bash 3rd Party Applicaons l Specialized kernel build containing only the necessary modules libssl Libraries l Root filesystem libc populated with only necessary ssh services, libraries and 3rd party init Services applications

netfront blkfront Kernel ext4 Virtualizaon Technologies

• “Standard” VM – Standard Debian-based Linux VM • “Tinyfied” VM – Tinyx, based on Linux kernel/busybox • Unikernel – On Xen: MiniOS + miniperf – On KVM: OSv + iperf • Containers – Docker

Virtualizaon Technologies

• “Standard” VM – Standard Debian-based Linux VM • “Tinyfied” VM – Tinyx, based on Linux kernel/busybox • Unikernel – On Xen: MiniOS + miniperf – On KVM: OSv + iperf • Containers – Docker

What’s a Unikernel? • Specialized VM: single applicaon + minimalisc OS • Single address space, co-operave scheduler so low overheads What’s a Unikernel? • Specialized VM: single

applicaon +

minimalisc OS 1 pp a app N app 2 • Single address space, co-operave scheduler so low overheads KERNEL SPACE driverN driver1 driver2

GENERAL-PURPOSE (e.g., Linux, FreeBSD) What’s a Unikernel? • Specialized VM: single

applicaon +

minimalisc OS 1 pp a app N app 2 • Single address space, USER SPACE co-operave scheduler so low overheads

app SINGLE ADDRESS SPACE KERNEL SPACE driverN driver1 driver2 vdriver2 Vdriver1

GENERAL-PURPOSE MINIMALISTIC OPERATING SYSTEM OPERATING SYSTEM (e.g., Linux, FreeBSD) (e.g., MiniOS, OSv) Unikernels for Benchmarking

apps

guest OS On Xen

Xen Unikernels for Benchmarking

apps iperf

guest mini OS OS On Xen

Xen Xen Unikernels for Benchmarking

apps iperf

guest mini OS OS On Xen

Xen Xen

apps

guest OS On KVM

KVM Unikernels for Benchmarking

apps iperf

guest mini OS OS On Xen

Xen Xen

apps iperf

guest OSv OS On KVM

KVM KVM Nota Bene…

• Our unikernel numbers include opmizaons to the underlying virtualizaon plaorms (Xen, KVM) – Toolstacks – Back-end stores – Hotplug scripts – Network drivers (on Xen Tx) • No me to go over these… RESULTS Image Size, Memory Usage (log scale)

1000 913 913 img size mem usage 100 112 82 61 52 MB 31 30 10 12 8 3.8 3.7 3.5 2 1 Boot Times (log scale)

10000 6500 2988 ) 1000 1711 1081 ms 431 330 100

Boot Time ( 31 10

1 RTT 40 35 30 34

) 25 ms 20 19 18

RTT ( 15 15 10 5 9 5 4 0 Throughput

60 Tx 50 Rx 40

30 Throughput (Gb/s) 20

10

0 Conclusions

• Common lore: VMs provide good isolaon but are heavyweight – Results with standard VMs confirm this • Containers provide lighter-weight virtualizaon – But nyfied VMs and especially unikernels yield comparable performance Conclusions

• Common lore: VMs provide good isolaon but are heavyweight – Results with standard VMs confirm this • Containers provide lighter-weight virtualizaon – But nyfied VMs and especially unikernels yield comparable performance Potenal Contribuons to dra-natarajan-nfvrg-containers-for-nfv-01

2.1.1 Challenges - VNF provisioning me - Runme performance (throughput, scaling up/down)

3. Benefits of Containers - Service agility vs VMs - Containers have beer runme performance - Auto-scaling of VNFs - Cross-VNF compability: container unikernel/minimalisc distro - Overall performance: VMs -25% throughput vs containers

5. Conclusion - Containers have significant advantages vs hypervisor-based soluons