XEN and KVM in INFN production systems and a comparison between them
Riccardo Veraldi Andrea Chierici INFN - CNAF HEPiX Spring 2009 Outline
xen kvm Test description Benchmarks Conclusions
[email protected] - HEPiX Spring 2009 Umea 2 What we have now: Xen
Open source “industry standard” for virtualization Included in all recent linux distributions Supports para and hvm approaches Widely used in HEP community Won’t go into details during presentation
[email protected] - HEPiX Spring 2009 Umea 3 What could be the future: KVM
Kernel-based Virtual Machine Open source included in latest linux kernels: implemented as a module A user space program uses /dev/kvm interface to set up VMs (qemu-kvm) Supports hvm approach Rather new to HEP community Qumranet now owned by Red Hat Foreseen boost in development
[email protected] - HEPiX Spring 2009 Umea 4 KVM virtio
Direct access to hardware Hard disk (/dev/vdx) Ethernet interface Support from kernel 2.6.25 Supported since RHEL 5.3 (virtio backport) kernel 2.6.18-128.1.1 Virtio drivers must be on the VM, not on the host EGEE sw can’t currently benefit from this
[email protected] - HEPiX Spring 2009 Umea 5 KVM seen by system admin
rpms: basically the kernel module and a modified version of qemu (qemu-kvm) Network configuration to be done by hand in order to get public IP bridge-utils and tunctl rpms were used to set-up “tap” interface Script developed to automate the process A modified init.d script to configure software bridge has been developed at CNAF No VM configuration file direct support VMs are launched via a standard UNIX command + command line options libvirt
[email protected] - HEPiX Spring 2009 Umea 6 KVM seen by system admin
No direct interface to manage VMs by command line (e.g. “xm”) Libvirt support Virt-manager working Monitor interface Very powerful, lots of options available qemu already well documented, support available on-line
[email protected] - HEPiX Spring 2009 Umea 7 KVM introduced at CNAF
We use quattor to configure and install grid nodes and virtual machines Profiles describe machine configuration Network boot for installation working Keeps booting from network, need to restart with “-boot c” No need for any modification, just like xen- hvm
[email protected] - HEPiX Spring 2009 Umea 8 Qualitative test
CNAF is running LHCb tier2 site entirely on xen VMs (2 CEs, 1 SE) Changed one Computing Element with a KVM machine Has been working for more that 3 weeks flawlessly Quattor machine profile unmodified, no effort for sysadmin CMS secondary squid server installed on the same host with same result (2 weeks ago) Running a KVM in production since 2 months for INFN-CA backups. Works flawlessly KVM executed (and live migrated!) a win7 VM Hardware used: 1 node, dual E5420, 16GB ram, sata disks via Areca controller
[email protected] - HEPiX Spring 2009 Umea 9 Quantitative test: description
Need some measures to understand what is the best solution Tested 3 classic parameters CPU → hep-spec06 (v1.1), PI test for cpu distribution Network → iperf (v2.0.4) Disk access → bonnie++ (v1.94) Compared Xen (para-virtualized and hvm) with KVM, using non virtualized machine as a baseline
[email protected] - HEPiX Spring 2009 Umea 10 Quantitative test: description
Hardware used: 1 blade, dual E5420, 16GB ram, 10k sas disk via LSI logic raid controller (raid0) Xen-para VM specs: 1 vcpu, 2 GB ram, disk on a file Xen-hvm VM specs: 1 vcpu, 2GB ram, disk on a file, “netfront” network driver KVM VM specs: 1 vcpu, 2GB ram, disk on a file, e1000 network driver emulation (non-virtio tests) Host OS: SL 5.2 x86_64, kernel 2.6.18-92.1.22.el5 VM OS: SLC 4.5 i386, kernel 2.6.9-67.0.15.EL.cern VM OS: SLC 5.3 x86_64, kernel 2.6.18-128.1.1 for kvm virtio tests KVM version: 83 Xen version: 3.2.1
[email protected] - HEPiX Spring 2009 Umea 11 Benchmarks: HEP-Spec06 XEN vs. KVM on dual Intel E5420, single performance measure 11
10.5
10
9.5 Spec06 ‐ 9 HEP
8.5
8
7.5 1478 Number of concurrent VMs
Xen‐hvm Xen‐para KVM E5420
[email protected] - HEPiX Spring 2009 Umea 12 Benchmarks: HEP-Spec06
VMs vs. CPU
70
60
50
40 Spec06 ‐ 30 HEP
20
10
0 1vm 4vm 7vm 8vm
kvm xen‐hvm xen‐para E5420
[email protected] - HEPiX Spring 2009 Umea 13 Benchmarks: HEP-Spec06
8VMs aggregate vs. CPUs
70
60
50
40 Spec06 ‐ 30 HEP
20
10
0 E5410 E5420 E5420kvm E5420xen‐hvm E5420xen‐para E5430
[email protected] - HEPiX Spring 2009 Umea 14 Benchmarks: HEP-Spec06
Virtualization % loss from non Technology emulated CPU (E5420, 8vm) E5420kvm 3,42 E5420xen-hvm 4,55 E5420xen-para 2,02 E5410 vs. E5420 4,07
[email protected] - HEPiX Spring 2009 Umea 15 Benchmarks: PI test
System tools to measure CPU usage are not reliable enough Calculating PI with n arbitrary ciphers Algorithm based on Numerical Recipes Written in C Compiled with options gcc -O3 -ffast-math -finline-limit=1000 Returns the number of CPU cycles An external function included into the code calculates the number of CPU cycles, using rdtsc IA32 registry
[email protected] - HEPiX Spring 2009 Umea 16 Benchmarks: PI test
KVM host vs. guest
3.1E+12
3E+12
2.9E+12
2.8E+12 2.7E+12 difference < 2% cycles
2.6E+12
CPU 2.5E+12
2.4E+12
2.3E+12
2.2E+12
2.1E+12 host guest
[email protected] - HEPiX Spring 2009 Umea 17 Benchmarks: PI test 8 concurrent guest
3.1E+12 3E+12 2.9E+12 2.8E+12 2.7E+12 cycles
2.6E+12 2.5E+12 CPU 2.4E+12 2.3E+12 2.2E+12 2.1E+12 guest guest1 guest2 guest3 guest4 guest5 guest6 guest7 guest8 singolo
• 8 core • Balanced Performance • Every guest uses one core at 100%
[email protected] - HEPiX Spring 2009 Umea 18 Benchmarks: PI test 17 concurrent guests
7E+12
6E+12
5E+12 cycles
4E+12 CPU 3E+12
2E+12
1E+12
• Good balancing on guest • Max difference 14%, average ~5%
[email protected] - HEPiX Spring 2009 Umea 19 Benchmarks: Iperf
KVM Network Performance, e1000
1000
900
800
700
600
500 in
Mbits/sec 400 out
300
200
100
0 1vm 2vm 4vm 8vm host iperf ‐w256k ‐P 5 ‐t 900
[email protected] - HEPiX Spring 2009 Umea 20 Benchmarks: Iperf
KVM vs XEN, 1vm, network performance
1000
900
800
700
600
500 in
Mbits/sec 400 out
300
200
100
0 kvm e1000 kvm virtio xen‐para xen‐hvm pv iperf ‐w256k ‐P5 ‐t 900
[email protected] - HEPiX Spring 2009 Umea 21 Benchmarks: bonnie++, file vs lvm
2GB Ram, 4GB data set, 1vm comparison
160000
140000
120000
100000
80000 K/sec 60000
40000
20000
0 Host 1 kvm file 1 kvm lvm 1 xen‐para file 1 xen‐para lvm 1 xen‐hvm file 1 xen‐hvm lvm bonnie++ on a single VM
seq out block seq out rewrite seq in block
[email protected] - HEPiX Spring 2009 Umea 22 Benchmarks: bonnie++
2GB ram, 4GB data set, single
120000
100000
80000
60000
40000
20000
0 1 kvm vm 4 kvm vm 8 kvm vm 1 xen‐para 4 xen‐para 8 xen‐para 1 xen‐hvm 4 xen‐hvm 8 xen‐hvm
seq out block seq out rewrite seq in block
[email protected] - HEPiX Spring 2009 Umea 23 Benchmarks: bonnie++, kvm virtio 32GB data set, lvm partition, single
140000
120000
100000
80000
K/sec 60000
40000
20000
0 Host hdb vdb Bonnie++ on a single vm
seq out block seq out rewrite seq in block
[email protected] - HEPiX Spring 2009 Umea 24 Benchmarks: bonnie++ 2GB ram, 4GB data set, 8vm aggregate
120000
100000
80000
60000 K/sec
40000
20000
0
seq output block seq output rewrite seq input block
[email protected] - HEPiX Spring 2009 Umea 25 Conclusions
KVM proved good stability and reliability No problems on running production machines for more than 4 weeks CPU performances are extremely good Modern CPU virtualization technologies help! Network performances are fair with e1000, good with virtio Disk I/O seems the most problematic aspect Other solutions have problems too Requires sysadmins only a small effort Even if looking promising, right now xen is the most performing solution
[email protected] - HEPiX Spring 2009 Umea 26 Future work
qemu snapshot features Backup and cloning of nodes
high-level VM managers Ovirt, enomalism, ganeti
[email protected] - HEPiX Spring 2009 Umea 27 Bibliography and links
Xen Xen repository KVM Kvm repository Quattor HEP-SPEC Numerical Recipes: second edition in C Computer Systems: A Programmer's Perspective
[email protected] - HEPiX Spring 2009 Umea 28