XEN and KVM in INFN production systems and a comparison between them

Riccardo Veraldi Andrea Chierici INFN - CNAF HEPiX Spring 2009 Outline

„ „ kvm „ Test description „ Benchmarks „ Conclusions

[email protected] - HEPiX Spring 2009 Umea 2 What we have now: Xen

„ Open source “industry standard” for „ Included in all recent distributions „ Supports para and hvm approaches „ Widely used in HEP community „ Won’t go into details during presentation

[email protected] - HEPiX Spring 2009 Umea 3 What could be the future: KVM

„ Kernel-based „ Open source „ included in latest linux kernels: implemented as a module A program uses /dev/kvm interface to set up VMs (-kvm) „ Supports hvm approach „ Rather new to HEP community „ Qumranet now owned by Foreseen boost in development

[email protected] - HEPiX Spring 2009 Umea 4 KVM virtio

„ Direct access to hardware Hard disk (/dev/vdx) interface „ Support from kernel 2.6.25 „ Supported since RHEL 5.3 (virtio backport) kernel 2.6.18-128.1.1 Virtio drivers must be on the VM, not on the host EGEE sw can’t currently benefit from this

[email protected] - HEPiX Spring 2009 Umea 5 KVM seen by system admin

„ rpms: basically the kernel module and a modified version of qemu (qemu-kvm) „ Network configuration to be done by hand in order to get public IP bridge-utils and tunctl rpms were used to set-up “tap” interface „ Script developed to automate the process A modified init.d script to configure software bridge has been developed at CNAF „ No VM configuration file direct support VMs are launched via a standard UNIX command + command line options

[email protected] - HEPiX Spring 2009 Umea 6 KVM seen by system admin

„ No direct interface to manage VMs by command line (e.g. “xm”) Libvirt support „ Virt-manager working „ Monitor interface Very powerful, lots of options available „ qemu already well documented, support available on-line

[email protected] - HEPiX Spring 2009 Umea 7 KVM introduced at CNAF

„ We use quattor to configure and install grid nodes and virtual machines Profiles describe machine configuration Network boot for installation working „ Keeps booting from network, need to restart with “-boot ” No need for any modification, just like xen- hvm

[email protected] - HEPiX Spring 2009 Umea 8 Qualitative test

„ CNAF is running LHCb tier2 site entirely on xen VMs (2 CEs, 1 SE) „ Changed one Computing Element with a KVM machine Has been working for more that 3 weeks flawlessly Quattor machine profile unmodified, no effort for sysadmin „ CMS secondary squid server installed on the same host with same result (2 weeks ago) „ Running a KVM in production since 2 months for INFN-CA backups. Works flawlessly „ KVM executed (and live migrated!) a win7 VM „ Hardware used: 1 node, dual E5420, 16GB ram, sata disks via Areca controller

[email protected] - HEPiX Spring 2009 Umea 9 Quantitative test: description

„ Need some measures to understand what is the best solution „ Tested 3 classic parameters CPU → hep-spec06 (v1.1), PI test for cpu distribution Network → iperf (v2.0.4) Disk access → bonnie++ (v1.94) „ Compared Xen (para-virtualized and hvm) with KVM, using non virtualized machine as a baseline

[email protected] - HEPiX Spring 2009 Umea 10 Quantitative test: description

„ Hardware used: 1 blade, dual E5420, 16GB ram, 10k sas disk via LSI logic raid controller (raid0) „ Xen-para VM specs: 1 vcpu, 2 GB ram, disk on a file „ Xen-hvm VM specs: 1 vcpu, 2GB ram, disk on a file, “netfront” network driver „ KVM VM specs: 1 vcpu, 2GB ram, disk on a file, e1000 network driver emulation (non-virtio tests) „ Host OS: SL 5.2 x86_64, kernel 2.6.18-92.1.22.el5 „ VM OS: SLC 4.5 i386, kernel 2.6.9-67.0.15.EL.cern „ VM OS: SLC 5.3 x86_64, kernel 2.6.18-128.1.1 for kvm virtio tests „ KVM version: 83 „ Xen version: 3.2.1

[email protected] - HEPiX Spring 2009 Umea 11 Benchmarks: HEP-Spec06 XEN vs. KVM on dual E5420, single performance measure 11

10.5

10

9.5 Spec06 ‐ 9 HEP

8.5

8

7.5 1478 Number of concurrent VMs

Xen‐hvm Xen‐para KVM E5420

[email protected] - HEPiX Spring 2009 Umea 12 Benchmarks: HEP-Spec06

VMs vs. CPU

70

60

50

40 Spec06 ‐ 30 HEP

20

10

0 1vm 4vm 7vm 8vm

kvm xen‐hvm xen‐para E5420

[email protected] - HEPiX Spring 2009 Umea 13 Benchmarks: HEP-Spec06

8VMs aggregate vs. CPUs

70

60

50

40 Spec06 ‐ 30 HEP

20

10

0 E5410 E5420 E5420kvm E5420xen‐hvm E5420xen‐para E5430

[email protected] - HEPiX Spring 2009 Umea 14 Benchmarks: HEP-Spec06

Virtualization % loss from non Technology emulated CPU (E5420, 8vm) E5420kvm 3,42 E5420xen-hvm 4,55 E5420xen-para 2,02 E5410 vs. E5420 4,07

[email protected] - HEPiX Spring 2009 Umea 15 Benchmarks: PI test

„ System tools to measure CPU usage are not reliable enough „ Calculating PI with n arbitrary ciphers Algorithm based on Numerical Recipes Written in C Compiled with options „ gcc -O3 -ffast-math -finline-limit=1000 „ Returns the number of CPU cycles An external function included into the code calculates the number of CPU cycles, using rdtsc IA32 registry

[email protected] - HEPiX Spring 2009 Umea 16 Benchmarks: PI test

KVM host vs. guest

3.1E+12

3E+12

2.9E+12

2.8E+12 2.7E+12 difference < 2% cycles

2.6E+12

CPU 2.5E+12

2.4E+12

2.3E+12

2.2E+12

2.1E+12 host guest

[email protected] - HEPiX Spring 2009 Umea 17 Benchmarks: PI test 8 concurrent guest

3.1E+12 3E+12 2.9E+12 2.8E+12 2.7E+12 cycles

2.6E+12 2.5E+12 CPU 2.4E+12 2.3E+12 2.2E+12 2.1E+12 guest guest1 guest2 guest3 guest4 guest5 guest6 guest7 guest8 singolo

• 8 core • Balanced Performance • Every guest uses one core at 100%

[email protected] - HEPiX Spring 2009 Umea 18 Benchmarks: PI test 17 concurrent guests

7E+12

6E+12

5E+12 cycles

4E+12 CPU 3E+12

2E+12

1E+12

• Good balancing on guest • Max difference 14%, average ~5%

[email protected] - HEPiX Spring 2009 Umea 19 Benchmarks: Iperf

KVM Network Performance, e1000

1000

900

800

700

600

500 in

Mbits/sec 400 out

300

200

100

0 1vm 2vm 4vm 8vm host iperf ‐w256k ‐P 5 ‐t 900

[email protected] - HEPiX Spring 2009 Umea 20 Benchmarks: Iperf

KVM vs XEN, 1vm, network performance

1000

900

800

700

600

500 in

Mbits/sec 400 out

300

200

100

0 kvm e1000 kvm virtio xen‐para xen‐hvm pv iperf ‐w256k ‐P5 ‐t 900

[email protected] - HEPiX Spring 2009 Umea 21 Benchmarks: bonnie++, file vs lvm

2GB Ram, 4GB data set, 1vm comparison

160000

140000

120000

100000

80000 K/sec 60000

40000

20000

0 Host 1 kvm file 1 kvm lvm 1 xen‐para file 1 xen‐para lvm 1 xen‐hvm file 1 xen‐hvm lvm bonnie++ on a single VM

seq out block seq out rewrite seq in block

[email protected] - HEPiX Spring 2009 Umea 22 Benchmarks: bonnie++

2GB ram, 4GB data set, single

120000

100000

80000

60000

40000

20000

0 1 kvm vm 4 kvm vm 8 kvm vm 1 xen‐para 4 xen‐para 8 xen‐para 1 xen‐hvm 4 xen‐hvm 8 xen‐hvm

seq out block seq out rewrite seq in block

[email protected] - HEPiX Spring 2009 Umea 23 Benchmarks: bonnie++, kvm virtio 32GB data set, lvm partition, single

140000

120000

100000

80000

K/sec 60000

40000

20000

0 Host hdb vdb Bonnie++ on a single vm

seq out block seq out rewrite seq in block

[email protected] - HEPiX Spring 2009 Umea 24 Benchmarks: bonnie++ 2GB ram, 4GB data set, 8vm aggregate

120000

100000

80000

60000 K/sec

40000

20000

0

seq output block seq output rewrite seq input block

[email protected] - HEPiX Spring 2009 Umea 25 Conclusions

„ KVM proved good stability and reliability No problems on running production machines for more than 4 weeks CPU performances are extremely good „ Modern CPU virtualization technologies help! Network performances are fair with e1000, good with virtio Disk I/O seems the most problematic aspect „ Other solutions have problems too Requires sysadmins only a small effort „ Even if looking promising, right now xen is the most performing solution

[email protected] - HEPiX Spring 2009 Umea 26 Future work

„ qemu snapshot features Backup and cloning of nodes

„ high-level VM managers Ovirt, enomalism,

[email protected] - HEPiX Spring 2009 Umea 27 Bibliography and links

„ Xen Xen repository „ KVM Kvm repository „ Quattor „ HEP-SPEC „ Numerical Recipes: second edition in C „ Computer Systems: A 's Perspective

[email protected] - HEPiX Spring 2009 Umea 28