Performance Profiling in a Virtualized Environment

Jiaqing Du+, Nipun Sehrawat*, Willy Zwaenepoel+ +EPFL, Switzerland *IIT Guwaha, India Virtualizaon‐based Clouds

• Diverse implementaons – soware: Xen, KVM, VMware, … – hardware: Intel , AMD x86, PowerPC, … • Opportunies for performance profiling & tuning – public cloud: guest OS & applicaons – private cloud: whole soware stack

Profilers based on CPU performance counters do not work well with virtual machines.

Jiaqing Du, HotCloud, June 22, 2010 2 Profiling in a Virtualized Environment

• General challenge • XenOProf: profiling in the VMM – only for paravirtualizaon‐based Xen – require accesses to the VMM • Profiling in the guest – normally no results – “OProfile can't work with VMware when using performance counter interface. ”

Jiaqing Du, HotCloud, June 22, 2010 3 Performance Profiling

• Understand runme behavior • Tune performance • Mature & used extensively – VTune, OProfile, …

%CYCLE Funcon Module 98.5529 vmx_vcpu_run kvm‐intel.ko 0.2226 (no symbols) libc.so 0.1034 hpet_cpuhp_nofy vmlinux 0.1034 nave_patch vmlinux

Jiaqing Du, HotCloud, June 22, 2010 4 Nave Profiling

• Performance monitoring unit (PMU) – a set of event counters – generate an when a counter overflows

• PMU‐based profiler User control – sampling configuraon Kernel – sample collecon Module Interpret* – sample interpretaon Configure Collect

CPU PMU

Jiaqing Du, HotCloud, June 22, 2010 5 Guest‐wide Profiling

• Expose PMU interfaces to the guest

User control process

Kernel Module Interpret* Configure Collect

VMM

CPU PMU

Jiaqing Du, HotCloud, June 22, 2010 6 Guest‐wide Profiling

• Where to save & restore the registers? • CPU switch – only in‐guest execuon is accounted to the guest

guest1 VMM guest2 VMM guest2

save&restore • Domain switch – in‐VMM execuon is also accounted to the guest

guest1 VMM guest2 VMM guest2 domain1 domain2

save&restore Jiaqing Du, HotCloud, June 22, 2010 7 Guest‐wide Profiling for KVM

• Kernel‐based virtual machine (KVM) – a kernel subsystem – a set of kernel modules + QEMU – built on hardware virtualizaon extensions • Intel VT extensions – provide a list of hardware features – facilitate our implementaon • No modificaons to the guest and the profiler

Jiaqing Du, HotCloud, June 22, 2010 8 Profiling Packet Receive

• Experiment – push packets to a Linux guest in KVM – run OProfile in the guest – monitor instrucon rerements

Linux

KVM virtual NIC Linux

Hardware Hardware NIC NIC

Jiaqing Du, HotCloud, June 22, 2010 9 Profiling Packet Receive

CPU Switch Domain Switch INSTR Funcon INSTR Funcon 167 csum_paral 2261 cp_interrupt 106 csum_paral_copy_generic 1336 cp_rx_poll 74 copy_to_user 1034 cp_start_xmit 47 ipt_do_table 421 nave_apic_mem_write 38 tcp_v4_rcv 374 nave_apic_mem_read … … 191 csum_paral 19 cp_rx_poll 105 csum_paral_copy_generic 6 cp_start_xmit 94 copy_to_user 6 cp_interrupt 79 ipt_do_table 3 nave_apic_mem_write 51 tcp_v4_rcv 1184 counter overflows 7286 counter overflows

Jiaqing Du, HotCloud, June 22, 2010 10 Other Things in the Paper

• System‐wide profiling – profiling in the VMM – provide full‐scale view: guest + VMM • Virtualizaon techniques – paravirtualizaon – dynamic binary translaon

Jiaqing Du, HotCloud, June 22, 2010 11 Conclusions

• Profilers do not work well with virtual machines. • We implement guest‐wide profiling in a VMM based on hardware assistance. • Profiling helps understand the real cost of I/O operaons in a guest.

Jiaqing Du, HotCloud, June 22, 2010 12