VMs, Unikernels and Containers: Experiences on the Performance of Virtualiza on Technologies
Felipe Huici, Filipe Manco, Jose Mendes, Simon Kuenzer NEC Europe Ltd. (Heidelberg) In the Beginning…
VM In the Beginning…
“Tinyfied VMs” VM In the Beginning…
“Tinyfied VMs”
unikernels VM In the Beginning…
“Tinyfied VMs” containers unikernels VM In the Beginning…
“Tinyfied VMs” containers unikernels VM Virt. Technology Benchmarking
• Metrics: – VM Image and memory consump on – VM crea on me – Delay – Throughput Virt. Technology Benchmarking
• Metrics: – VM Image and memory consump on – VM crea on me – Delay – Throughput
higher lower overhead overhead Virt. Technology Benchmarking
• Metrics: – VM Image and memory consump on – VM crea on me – Delay – Throughput
higher lower overhead overhead Virt. Technology Benchmarking
• Metrics: – VM Image and memory consump on – VM crea on me – Delay – Throughput
higher lower overhead overhead Virt. Technology Benchmarking
• Metrics: – VM Image and memory consump on – VM crea on me – Delay – Throughput
higher lower overhead overhead Virt. Technology Benchmarking
• Metrics: – VM Image and memory consump on – VM crea on me – Delay – Throughput
higher lower overhead overhead Virt. Technology Benchmarking
• Metrics: – VM Image and memory consump on – VM crea on me – Delay – Throughput ? higher lower overhead overhead Virt. Technology Benchmarking
• Metrics: – VM Image and memory consump on – VM crea on me – Delay – Throughput
higher lower overhead overhead Virt. Technology Benchmarking
• Metrics: – VM Image and memory consump on – VM crea on me – Delay – Throughput ? higher lower overhead overhead Virtualiza on Technology Benchmarking • Metrics: – VM image and memory consump on: ls, top, xl – VM crea on me: SYN flood + RST detec on – Throughput: iperf, guest to host (TCP traffic) – RTT: ping flood
• VM-based tests run on both Xen and KVM
• Hardware: x86_64 server with an Intel Xeon E5-1630 v3 3.7GHz CPU (4 cores), 32GB RAM.
Virtualiza on Technologies
• “Standard” VM – Standard Debian-based Linux VM • “Tinyfied” VM – Tinyx, based on Linux kernel/busybox • Unikernel – On Xen: MiniOS + miniperf – On KVM: OSv + iperf • Containers – Docker
Virtualiza on Technologies
• “Standard” VM – Standard Debian-based Linux VM • “Tinyfied” VM – Tinyx, based on Linux kernel/busybox • Unikernel – On Xen: MiniOS + miniperf – On KVM: OSv + iperf • Containers – Docker
Standard VM: Applica on on Top of Distro
User Applica on
3rd Party Applica ons
Libraries
Services
Kernel Most of the VM not Used…
Nginx User Applica on
memcached bash 3rd Party Applica ons
libssl Libraries libc
ssh init Services
ext4 netfront blkfront Kernel Tinyx: Keep Only What’s Needed
Nginx User Applica on
memcached bash 3rd Party Applica ons
libssl Libraries libc
ssh init Services
ext4 netfront blkfront Kernel Tinyx: Taylor-made Distro
Nginx User Applica on memcached bash 3rd Party Applica ons
libssl Libraries libc
ssh init Services
netfront blkfront Kernel ext4 Tinyx: Taylor-made Distro
Nginx User Applica on memcached bash 3rd Party Applica ons
libssl Libraries libc
ssh init Services
netfront blkfront Kernel ext4 Tinyx: Taylor-made Distro
Nginx User Applica on memcached bash 3rd Party Applica ons
libssl Libraries libc
ssh init Services
netfront blkfront Kernel ext4 Tinyx: Taylor-made Distro
Nginx User Applica on memcached bash 3rd Party Applica ons
libssl Libraries libc
ssh init Services
netfront blkfront Kernel ext4 Tinyx: Taylor-made Distro
▌ Keep only the Nginx User Applica on necessary bits memcached and pieces bash 3rd Party Applica ons l Specialized kernel build containing only the necessary modules libssl Libraries l Root filesystem libc populated with only necessary ssh services, libraries and 3rd party init Services applications
netfront blkfront Kernel ext4 Virtualiza on Technologies
• “Standard” VM – Standard Debian-based Linux VM • “Tinyfied” VM – Tinyx, based on Linux kernel/busybox • Unikernel – On Xen: MiniOS + miniperf – On KVM: OSv + iperf • Containers – Docker
Virtualiza on Technologies
• “Standard” VM – Standard Debian-based Linux VM • “Tinyfied” VM – Tinyx, based on Linux kernel/busybox • Unikernel – On Xen: MiniOS + miniperf – On KVM: OSv + iperf • Containers – Docker
What’s a Unikernel? • Specialized VM: single applica on + minimalis c OS • Single address space, co-opera ve scheduler so low overheads What’s a Unikernel? • Specialized VM: single
applica on +
minimalis c OS 1 pp a app N app 2 • Single address space, USER SPACE co-opera ve scheduler so low overheads KERNEL SPACE driverN driver1 driver2
GENERAL-PURPOSE OPERATING SYSTEM (e.g., Linux, FreeBSD) What’s a Unikernel? • Specialized VM: single
applica on +
minimalis c OS 1 pp a app N app 2 • Single address space, USER SPACE co-opera ve scheduler so low overheads
app SINGLE ADDRESS SPACE KERNEL SPACE driverN driver1 driver2 vdriver2 Vdriver1
GENERAL-PURPOSE MINIMALISTIC OPERATING SYSTEM OPERATING SYSTEM (e.g., Linux, FreeBSD) (e.g., MiniOS, OSv) Unikernels for Benchmarking
apps
guest OS On Xen
Xen Unikernels for Benchmarking
apps iperf
guest mini OS OS On Xen
Xen Xen Unikernels for Benchmarking
apps iperf
guest mini OS OS On Xen
Xen Xen
apps
guest OS On KVM
KVM Unikernels for Benchmarking
apps iperf
guest mini OS OS On Xen
Xen Xen
apps iperf
guest OSv OS On KVM
KVM KVM Nota Bene…
• Our unikernel numbers include op miza ons to the underlying virtualiza on pla orms (Xen, KVM) – Toolstacks – Back-end stores – Hotplug scripts – Network drivers (on Xen Tx) • No me to go over these… RESULTS Image Size, Memory Usage (log scale)
1000 913 913 img size mem usage 100 112 82 61 52 MB 31 30 10 12 8 3.8 3.7 3.5 2 1 Boot Times (log scale)
10000 6500 2988 ) 1000 1711 1081 ms 431 330 100
Boot Time ( 31 10
1 RTT 40 35 30 34
) 25 ms 20 19 18
RTT ( 15 15 10 5 9 5 4 0 Throughput
60 Tx 50 Rx 40
30 Throughput (Gb/s) 20
10
0 Conclusions
• Common lore: VMs provide good isola on but are heavyweight – Results with standard VMs confirm this • Containers provide lighter-weight virtualiza on – But nyfied VMs and especially unikernels yield comparable performance Conclusions
• Common lore: VMs provide good isola on but are heavyweight – Results with standard VMs confirm this • Containers provide lighter-weight virtualiza on – But nyfied VMs and especially unikernels yield comparable performance Poten al Contribu ons to dra -natarajan-nfvrg-containers-for-nfv-01
2.1.1 Challenges - VNF provisioning me - Run me performance (throughput, scaling up/down)
3. Benefits of Containers - Service agility vs VMs - Containers have be er run me performance - Auto-scaling of VNFs - Cross-VNF compa bility: container unikernel/minimalis c distro - Overall performance: VMs -25% throughput vs containers
5. Conclusion - Containers have significant advantages vs hypervisor-based solu ons