ESX Server Architectural Directions
Total Page:16
File Type:pdf, Size:1020Kb
PAC-498B ESX Server Architectural Directions Beng-Hong Lim Director of R&D This presentation may contain VMware confidential information. Copyright © 2005 VMware, Inc. All rights reserved. All other marks and names mentioned herein may be trademarks of their respective companies. Agenda ESX Server overview Technology trends in the datacenter ESX Server architectural directions CPU virtualization I/O virtualization Scalable performance Power efficiency and management Conclusions ESX Server Core Virtualization VMX VMX VMX VMX VM VM VM VM Virtual I/O Stack VMM VMM VMM VMM Machine Service Monitor Console Device Drivers Storage Stack Network Stack Device Drivers VMkernel VMkernel Hardware Interface Hardware ESX Server Enterprise-Class Features Third- Party Agents VMX VMX VMX VMX VM VM VM VM Virtual I/O Stack VMM VMM VMM VMM Machine Service Monitor Console Device Drivers Distributed Virtual NIC & Enterprise Resource VM File System Switch Class Management Virtualization Functionality CPU Scheduling Storage Stack Network Stack Memory Scheduling Storage Bandwidth Network Bandwidth Device Drivers VMkernel VMkernel Hardware Interface Hardware ESX Server in the Datacenter Management Third-party Solutions and Distributed Virtualization Services Third- SDK, VirtualCenter Agent Party Agents VMX VMX VMX VMX VM VM VM VM Virtual I/O Stack Machine Service VMM VMM VMM VMM Device Drivers Monitor VMotion Console Provisioning Distributed Virtual NIC & Enterprise VM File System Switch Class Backup Resource Virtualization Management DRS Functionality CPU Scheduling Storage Stack Network Stack DAS Memory Scheduling Storage Bandwidth VirtualCenter Distributed Network Bandwidth Device Drivers Services VMkernel VMkernel Hardware Interface Hardware Agenda Overview of ESX Server Technology trends in the datacenter ESX Server architectural directions CPU virtualization I/O virtualization Scalable performance Power efficiency and management Conclusions Server Technology Trends Multi-core CPUs 16 to 32 CPU cores per server 64-bit systems Multi-terabytes of memory Power-aware architectures Adaptive throttling of CPUs and server hardware Converged I/O fabrics and interfaces Shared high-speed interface to network and storage Network-based, virtualized storage Stateless servers The Future Datacenter Each server: Many CPU cores 64-bits: lots of memory Shared, high-bandwidth … connection to network … and external storage Stateless Power-hungry Servers Network Storage Virtualization is Key Abundant compute resources on a server Virtualization is inherently scalable and parallel The killer app for efficiently utilizing many CPUs and multi- terabytes of memory Power management increasingly important Higher compute densities and rising utility costs Maximize performance per watt across all servers Distributed resource scheduling can optimize for this metric Transforms system management Breaks the bond between hardware and applications Ease of scale-up management in a scale-out environment The Virtual Datacenter ESX Server virtualizes individual servers VirtualCenter synthesizes virtualized servers into a giant computer ESX Server and … … VirtualCenter map applications and virtual machine topologies onto physical resources A Distributed Virtual Computer Agenda Overview of ESX Server Technology trends in the datacenter ESX Server architectural directions CPU virtualization I/O virtualization Scalable performance Power efficiency and management Conclusions CPU Virtualization VMX VMX VMX VMX VM VM VM VM I/O Stack VMM VMM VMM VMM Service Console Device Drivers Distributed Virtual NIC and Virtual Machine Switch File System Storage Stack Network Stack Device Drivers VMkernel Hardware CPU Virtualization Basic idea: directly execute code until not safe Handling unsafe code Trap and emulate: classic mainframe Avoid unsafe code, call into VMM: paravirtualization Dynamically transform to safe code: binary translation Tradeoffs among the methods Trap and Para- Binary Emulate virtualization Translation Performance Average Excellent Good Compatibility Excellent Poor Excellent Sophistication Average Average High CPU Virtualization Directions New technologies: 64-bit CPUs, VT/Pacifica, Linux/Windows paravirtualization. Many guest OS types Flexible architecture supports mix of guests and VMM types . Separate VMM per virtual VM VM VM VM machine Para BT VT . BT VMM VMM-32 VMM-64 VMM-32 Simultaneously run 32-bit, 64-bit, and paravirtualized VMkernel guests Use most efficient method for the hardware and guest OS Agenda Overview of ESX Server Technology trends in the datacenter ESX Server architectural directions CPU virtualization I/O virtualization Scalable performance Power efficiency and management Conclusions I/O Virtualization VMX VMX VMX VMX VM VM VM VM I/O Stack VMM VMM VMM VMM Service Console Device Drivers Distributed Virtual NIC and Virtual Machine Switch File System Storage Stack Network Stack Device Drivers VMkernel Hardware I/O Virtualization Paths Guest OS VMX Paths to physical device 1. Hosted/Split I/O: via a Device Device Driver Emulation separate host/VM I/O Stack VMM 2. Native I/O: via the Device Service Console Device Driver vmkernel Emulation 3. Passthrough I/O: guest 1. Hosted/ I/O Stack directly drives device Split I/O Device Driver Needs hardware support, VMkernel 2. Native I/O sacrifices functionality. 3. Passthrough I/O Which I/O path to use? Evaluating the I/O paths Compatibility Hardware vendors can re-use existing device drivers Performance (per watt) High I/O performance, low CPU occupancy Isolation Contain device driver faults Virtualization Functionality Virtual machine portability Resource sharing and multiplexing Offloading guest functionality into virtualization layer I/O Virtualization: Compatibility Hosted/Split Native Passthrough Compatibility Good Poor Good Performance Isolation Functionality Hosted/Split and Passthrough can re-use device drivers from existing OSes Native requires new or ported drivers. Provide DDK and driver APIs to ease driver development and porting Hosted/Split I/O Performance Service Console or Trusted Virtual Virtual Virtual Virtual Machine Machine1 Machine2 Machine3 OS OS OS OS Backend Frontend Frontend Frontend Native Device Device Device Device Driver Drivers Driver Driver VMkernel Microkernel-style communication, context switch and scheduling overhead unless CPUs dedicated Scalability limits to Service Console or Driver VM Native I/O Performance Virtual Virtual Virtual Machine1 Machine2 Machine3 OS OS OS Frontend Frontend Frontend Device Device Device Driver Driver Driver Backend VMkernel Native Device Drivers Direct calls between frontend and backend Backend can run on any CPU, scalable Passthrough I/O Performance Virtual Virtual Virtual Machine1 Machine2 Machine3 OS OS OS Device Device Device Driver Driver Driver Interrupt routing VMkernel Guest OS driver drives the device directly VMkernel may have to handle/route interrupts I/O Virtualization: Performance Hosted/Split Native Passthrough Compatibility Good Poor Good Performance Poor Good Good Isolation Functionality Hosted/Split incurs switching and scheduling overheads, or consumes dedicated CPUs Native and Passthrough are efficient, scalable Passthrough avoids an extra driver layer, but runs more code non-natively I/O Virtualization: Isolation, Today Hosted/Split Native Passthrough Compatibility Good Poor Good Performance Poor Good Good Isolation None None N/A Functionality Passthrough allows malicious guest to crash system, so not an option All three methods need I/O MMU to map and protect DMA I/O Virtualization: Isolation, Future Hosted/Split Native Passthrough Compatibility Good Poor Good Performance Poor Good Good Isolation Good Good Good Functionality Hosted/Split and Passthrough can isolate within virtual machine, use I/O MMUs Native can isolate within in-kernel protection domains, use VT/Pacifica and I/O MMUs Not a substitute for testing and qualification I/O Virtualization: Functionality Hosted/Split Native Passthrough Compatibility Good Poor Good Performance Poor Good Good Isolation Good Good Good Functionality Good Good Poor Passthrough precludes offloading functionality from the guest into the virtualization layer, e.g., NIC teaming, SAN multipathing Passthrough sacrifices some key virtualization capabilities; VM portability, VMotion I/O Virtualization Direction Hosted/Split Native Passthrough Compatibility Good Poor Good Performance Poor Good Good Isolation Good Good Good Functionality Good Good Poor Future datacenter implications Power-efficient performance favors Native and Passthrough Stateless servers and converged I/O interfaces: fewer devices to support, eases compatibility I/O Virtualization Direction Guest OS VMX Optimize Native I/O for selected devices, Device Device Driver Emulation driver isolation I/O Stack VMM Device Service Console Device Driver Emulation I/O Stack Device Driver VMkernel I/O Virtualization Direction Guest OS Optimize Native I/O for selected devices, Device Driver driver isolation VMM Migrate from Device Hosted/Split I/O to Emulation Passthrough I/O I/O Stack when hardware ready Device Driver VMkernel I/O Virtualization Direction Guest OS Guest OS Optimize Native I/O I/O Stack for selected devices, Device Driver Device Driver driver isolation VMM Migrate from Device Hosted/Split I/O to Emulation Passthrough I/O I/O Stack when hardware ready Device Driver Can synthesize VMkernel Hosted/Split I/O by proxy through Passthrough