ESXi Performance Principles DOAG Edition Valentin Bondzio Sr. Staff TSE / GSS Premier Services 2020-01-23 DOAG 2020 │ ©2020 VMware, Inc. Brief Intro DOAG 2020 NOON2NOON │ ©2020 VMware, Inc. 2 Brief Intro @VMware since 2009 Global Support Services / Premier Services Focus on Resource Management, Performance and Windows Internals Originally from Berlin, living in Ireland since 2007 And most importantly … DOAG 2020 NOON2NOON │ ©2020 VMware, Inc. 3 Brief Intro Not an Oracle expert ! DOAG 2020 NOON2NOON │ ©2020 VMware, Inc. 4 Brief Intro Not an Oracle expert ! DOAG 2020 NOON2NOON │ ©2020 VMware, Inc. 5 Agenda CPU Scheduling and Usage Accounting The “basics” “Power Management” The Good, the Better and the Ugly ESXi Memory Management More “basics” Local resource distribution What else is running on ESXi CPU Topology Abstraction CPU Socket != NUMA node DOAG 2020 NOON2NOON │ ©2020 VMware, Inc. 6 Agenda CPU Scheduling and Usage Accounting The “basics” “Power Management” The Good, the Better and the Ugly ESXi Memory Management More “basics” Local resource distribution +I/O stuff What else is running on ESXi CPU Topology Abstraction CPU Socket != NUMA node DOAG 2020 NOON2NOON │ ©2020 VMware, Inc. 7 Agenda CPU Scheduling and Usage Accounting The “basics” “Power Management” The Good, the Better and the Ugly ESXi Memory Management More “basics” Local resource distribution +I/O stuff What else is running on ESXi +vMotion CPU Topology Abstraction CPU Socket != NUMA node DOAG 2020 NOON2NOON │ ©2020 VMware, Inc. 8 Agenda CPU Scheduling and Usage Accounting The “basics” “Power Management” The Good, the Better and the Ugly ESXi Memory Management More “basics” Local resource distribution +I/O stuff What else is running on ESXi +vMotion CPU Topology Abstraction CPU Socket != NUMA node +Backup DOAG 2020 NOON2NOON │ ©2020 VMware, Inc. 9 CPU Scheduler Overview Resource guarantees and weighting (shares) on a per VM or “Resource Pool” level DOAG 2020 NOON2NOON │ ©2020 VMware, Inc. 10 CPU Scheduler Overview What does the scheduler do? Dispatch VMs (its “worlds”) to honor CPU settings (Local) vCPU vCPU vCPU HT / Core vCPU vCPU vCPU DOAG 2020 NOON2NOON │ ©2020 VMware, Inc. 11 CPU Scheduler Overview What does the scheduler do? Dispatch VMs (its “worlds”) to honor CPU settings (Local) • For fairness: select VM with the least (consumed CPU time / fair share) vCPU vCPU vCPU HT / Core vCPU vCPU vCPU DOAG 2020 NOON2NOON │ ©2020 VMware, Inc. 12 CPU Scheduler Overview What does the scheduler do? Dispatch VMs (its “worlds”) to honor CPU settings (Local) • For fairness: select VM with the least (consumed CPU time / fair share) • For priority: run latency-sensitive VM (high) before anyone else vCPU vCPU vCPU vCPU HT / Core IO vCPU vCPU vCPU DOAG 2020 NOON2NOON │ ©2020 VMware, Inc. 13 CPU Scheduler Overview What does the scheduler do? Place the worlds / threads on physical CPUs (Global) HT 0 HT 0 HT 0 HT 0 HT 0 HT 0 HT 0 HT 0 HT 1 HT 1 HT 1 HT 1 HT 1 HT 1 HT 1 HT 1 Core Core Core Core Core Core Core Core LLC LLC DOAG 2020 NOON2NOON │ ©2020 VMware, Inc. 14 CPU Scheduler Overview What does the scheduler do? Place the worlds / threads on physical CPUs (Global) HT 0 HT 0 HT 0 HT 0 HT 0 HT 0 HT 0 HT 0 HT 1 HT 1 HT 1 HT 1 HT 1 HT 1 HT 1 HT 1 Core Core Core Core Core Core Core Core LLC LLC VM VM VM VM • To balance load across physical execution contexts (PCPUs) DOAG 2020 NOON2NOON │ ©2020 VMware, Inc. 15 CPU Scheduler Overview What does the scheduler do? Place the worlds / threads on physical CPUs (Global) HT 0 HT 0 HT 0 HT 0 HT 0 HT 0 HT 0 HT 0 HT 1 HT 1 HT 1 HT 1 HT 1 HT 1 HT 1 HT 1 Core Core Core Core Core Core Core Core LLC LLC VM VM VM VM • To balance load across physical execution contexts (PCPUs) • To preserve cache state, minimize migration cost DOAG 2020 NOON2NOON │ ©2020 VMware, Inc. 16 CPU Scheduler Overview What does the scheduler do? Place the worlds / threads on physical CPUs (Global) HT 0 HT 0 HT 0 HT 0 HT 0 HT 0 HT 0 HT 0 HT 1 HT 1 HT 1 HT 1 HT 1 HT 1 HT 1 HT 1 Core Core Core Core Core Core Core Core LLC LLC VM VM VM VM VM • To balance load across physical execution contexts (PCPUs) • To preserve cache state, minimize migration cost • To avoid contention from hardware (HT, LLC, etc.) and sibling vCPUs (from the same VM) DOAG 2020 NOON2NOON │ ©2020 VMware, Inc. 17 CPU Scheduler Overview What does the scheduler do? Place the worlds / threads on physical CPUs (Global) HT 0 HT 0 HT 0 HT 0 HT 0 HT 0 HT 0 HT 0 HT 1 HT 1 HT 1 HT 1 HT 1 HT 1 HT 1 HT 1 Core Core Core Core Core Core Core Core LLC LLC VM VM VM VM VM VM VM • To balance load across physical execution contexts (PCPUs) • To preserve cache state, minimize migration cost • To avoid contention from hardware (HT, LLC, etc.) and sibling vCPUs (from the same VM) • To keep VMs or threads that have frequent communications close to each other DOAG 2020 NOON2NOON │ ©2020 VMware, Inc. 18 CPU Scheduler Overview How does that look? 10:10:29am up 2 days 48 min, 674 worlds, 1 VMs, 2 vCPUs; CPU load average: 0.02, 0.01, 0.01 PCPU USED(%): 0.3 0.1 0.0 0.3 0.2 0.1 0.0 0.0 0.0 0.2 50 50 4.1 0.1 0.1 0.0 0.0 0.0 0.1 0.0 0.0 0.1 0.0 0.0 AVG: 4.4 PCPU UTIL(%): 0.5 0.1 0.1 0.6 0.2 0.2 0.0 0.2 0.0 0.3 100 100 4.2 0.2 0.1 0.1 0.0 0.0 0.1 0.0 0.0 0.2 0.1 0.1 AVG: 8.6 CORE UTIL(%): 0.6 0.7 0.4 0.9 0.3 100 4.3 0.2 0.0 0.1 0.4 0.7 AVG: 9.1 ID GID NAME NWLD %USED %RUN %SYS %WAIT %VMWAIT %RDY %IDLE %OVRLP 96337 148153 vmx 1 0.02 0.01 0.02 61.82 - 37.86 0.00 0.00 96339 148153 NetWorld-VM-96338 1 0.00 0.00 0.00 99.68 - 0.00 0.00 0.00 96340 148153 NUMASchedRemapEpochInitial 1 0.00 0.00 0.00 99.68 - 0.00 0.00 0.00 96341 148153 vmast.96338 1 0.03 0.05 0.00 99.63 - 0.00 0.00 0.00 96343 148153 vmx-vthread-6 1 0.00 0.00 0.00 99.68 - 0.00 0.00 0.00 96344 148153 vmx-mks:Debian86 1 0.00 0.00 0.00 61.55 - 38.13 0.00 0.00 96345 148153 vmx-svga:Debian86 1 0.00 0.00 0.00 99.68 - 0.00 0.00 0.00 96346 148153 vmx-vcpu-0:Debian86 1 62.35 99.68 0.00 0.00 0.00 0.00 0.00 0.05 96348 148153 vmx-vcpu-1:Debian86 1 62.36 99.67 0.00 0.00 0.00 0.01 0.00 0.05 96347 148153 PVSCSI-96338:0 1 0.00 0.00 0.00 99.68 - 0.00 0.00 0.00 96350 148153 vmx-vthread-7:Debian86 1 0.00 0.00 0.00 99.68 - 0.00 0.00 0.00 DOAG 2020 NOON2NOON │ ©2020 VMware, Inc. 19 CPU Scheduler Overview How does that look? 10:10:29am up 2 days 48 min, 674 worlds, 1 VMs, 2 vCPUs; CPU load average: 0.02, 0.01, 0.01 PCPU USED(%): 0.3 0.1 0.0 0.3 0.2 0.1 0.0 0.0 0.0 0.2 50 50 4.1 0.1 0.1 0.0 0.0 0.0 0.1 0.0 0.0 0.1 0.0 0.0 AVG: 4.4 PCPU UTIL(%): 0.5 0.1 0.1 0.6 0.2 0.2 0.0 0.2 0.0 0.3 100 100 4.2 0.2 0.1 0.1 0.0 0.0 0.1 0.0 0.0 0.2 0.1 0.1 AVG: 8.6 CORE UTIL(%): 0.6 0.7 0.4 0.9 0.3 100 4.3 0.2 0.0 0.1 0.4 0.7 AVG: 9.1 ID GID NAME NWLD %USED %RUN %SYS %WAIT %VMWAIT %RDY %IDLE %OVRLP 96337 148153 vmx 1 0.02 0.01 0.02 61.82 - 37.86 0.00 0.00 96339 148153 NetWorld-VM-96338 1 0.00 0.00 0.00 99.68 - 0.00 0.00 0.00 96340 148153 NUMASchedRemapEpochInitial 1 0.00 0.00 0.00 99.68 - 0.00 0.00 0.00 96341 148153 vmast.96338 1 0.03 0.05 0.00 99.63 - 0.00 0.00 0.00 96343 148153 vmx-vthread-6 1 0.00 0.00 0.00 99.68 - 0.00 0.00 0.00 96344 148153 vmx-mks:Debian86 1 0.00 0.00 0.00 61.55 - 38.13 0.00 0.00 96345 148153 vmx-svga:Debian86 1 0.00 0.00 0.00 99.68 - 0.00 0.00 0.00 96346 148153 vmx-vcpu-0:Debian86 1 62.35 99.68 0.00 0.00 0.00 0.00 0.00 0.05 96348 148153 vmx-vcpu-1:Debian86 1 62.36 99.67 0.00 0.00 0.00 0.01 0.00 0.05 96347 148153 PVSCSI-96338:0 1 0.00 0.00 0.00 99.68 - 0.00 0.00 0.00 96350 148153 vmx-vthread-7:Debian86 1 0.00 0.00 0.00 99.68 - 0.00 0.00 0.00 DOAG 2020 NOON2NOON │ ©2020 VMware, Inc.