WHITE PAPER The Architecture of VMware ESXi VMWARE WHITE PAPER Table of Contents Introduction ............................................................................................................ 3 Components of ESXi ............................................................................................... 3 VMkernel ............................................................................................................ 4 File System .......................................................................................................... 4 Users and Groups ................................................................................................ 4 User Worlds ......................................................................................................... 4 Direct Console User Interface ............................................................................. 5 Other User World Processes................................................................................ 5 Open Network Ports ........................................................................................... 5 System Image Design ............................................................................................. 6 Startup and Operation ........................................................................................... 6 Management Model for ESXi .................................................................................. 7 State Information ................................................................................................ 7 Common Information Model .............................................................................. 7 VI API ................................................................................................................... 8 Summary ................................................................................................................. 8 About the Author .................................................................................................... 9 2 VMWARE WHITE PAPER The Architecture of VMware ESXi Introduction Components of ESXi VMware® ESXi is the next-generation hypervisor, providing The VMware ESXi architecture comprises the underlying operat- a new foundation for virtual infrastructure. This innovative ing system, called VMkernel, and processes that run on top of architecture operates independently from any general-purpose it. VMkernel provides means for running all processes on the operating system, offering improved security, increased reli- system, including management applications and agents as well ability, and simplified management. The compact architecture as virtual machines. It has control of all hardware devices on the is designed for integration directly into virtualization-optimized server, and manages resources for the applications. The main server hardware, enabling rapid installation, configuration, and processes that run on top of VMkernel are: deployment. • Direct Console User Interface (DCUI) — the low-level con- Functionally, ESXi is equivalent to ESX 3, offering the same figuration and management interface, accessible through the levels of performance and scalability. However, the Linux-based console of the server, used primarily for initial basic configura- service console has been removed, reducing the footprint tion. to less than 32MB of memory. The functionally of the service • The virtual machine monitor, which is the process that console is replaced by new remote command line interfaces in provides the execution environment for a virtual machine, as conjunction with adherence to system management standards. well as a helper process known as VMX. Each running virtual Because ESXi is functionally equivalent to ESX, it supports the machine has its own VMM and VMX process. entire VMware Infrastructure 3 suite of products, including • Various agents used to enable high-level VMware VMware Virtual Machine File System, Virtual SMP, VirtualCenter, Infrastructure management from remote applications. VMotion, VMware Distributed Resource Scheduler, VMware High Availability, VMware Update Manager, and VMware • The Common Information Model (CIM) system: CIM is the Consolidated Backup. interface that enables hardware-level management from remote applications via a set of standard APIs. Figure 1 shows a diagram of the overall ESXi architecture. The following sections provide a closer examination of each of these components. CIM broker vpxa SNMP Third-party VM VM VM hostd DCUI syslog CIM plug-ins VMX VMX VMX VMM VMM VMM User world API Distributed Virtual Ethernet Resource VM le system adapter and sw itch scheduling Storage stack Network stack VMkernel Device drivers Figure 1: The streamlined architecture of VMware ESXi eliminates the need for a service console. 3 VMWARE WHITE PAPER VMkernel Users and Groups VMkernel is a POSIX-like operating system developed by Users and groups can be defined locally on the ESXi system. VMware and provides certain functionality similar to that They provide a way to distinguish users that access the system found in other operating systems, such as process creation and via the Virtual Infrastructure Client, the remote command line control, signals, file system, and process threads. It is designed interfaces, or the VIM API. specifically to support running multiple virtual machines and Groups can be used to combine multiple users, just as in other provides such core functionality as: operating systems. Groups can be used, for example, to set • Resource scheduling privileges for many users at once. There are a few system users • I/O stacks and groups, which are predefined in order to identify certain processes running in the VMkernel. • Device drivers Administrative privileges can be set individually for each user or Some of the more pertinent aspects of the VMkernel are pre- group. User and group definitions are stored on the file system sented in the following sections. in the files /etc/passwd, /etc/shadow, and /etc/ File System group, and as in other operating systems, passwords are gen- VMkernel uses a simple in-memory file system to hold the ESXi erated using standard crypt functions. configuration files, log files, and staged patches. For familiar- User Worlds ity, the structure of the file system is designed to be the same The term “user world” refers to a process running in the as that used in the service console of ESX. For example, ESXi VMkernel operating system. The environment in which a user configuration files are found in and log files /etc/vmware world runs is limited compared to what would be found in a are found in . Staged patches are /var/log/vmware general-purpose POSIX-compliant operating system such as uploaded to . /tmp Linux. For example: This file system is independent of the VMware VMFS file system • The set of available signals is limited. used to store virtual machines. Just as with ESX, a VMware VMFS datastore may be created on a local disk in the host system • The system API is a subset of POSIX. or on shared storage. If the only VMFS datastores used by the • The /proc file system is very limited. host are on external shared storage, the ESXi system does not • A single swap file is available for all user world processes. If actually require a local hard drive. By running diskless setups, a local disk exists, the swap file is created automatically in a you can increase reliability by avoiding hard drive failures and small VFAT partition. Otherwise, the user is free to set up a reduce power and cooling consumption. swap file on one of the attached VMFS datastores. Remote command line interfaces provide file management In short, a user world is not intended as a general-purpose capabilities for both the in-memory file system and the VMware mechanism to run arbitrary applications but provides only VMFS datastores. Access to the file system is implemented via enough of a framework for processes that need to run in the HTTPS get and put Access is authenticated via users and hypervisor environment. groups configured locally on the server and is controlled by Several important process run in user worlds. These can be local privileges. thought of as native VMkernel applications and are described in Because the in-memory file system does not persist when the the following sections. power is shut down, log files do not survive a reboot. ESXi has the ability to configure a remote syslog server, enabling you to save all log information on an external system. 4 VMWARE WHITE PAPER Direct Console User Interface Other User World Processes The Direct Console User Interface (DCUI) is the local user inter- Agents used by VMware to implement certain management face that is displayed only on the console of an ESXi system. capabilities have been ported from running in the service It provides a BIOS-like, menu-driven interface for interacting console to running in user worlds. with the system. Its main purpose is initial configuration and • The hostd process provides a programmatic interface to troubleshooting. One of the system users defined in VMkernel VMkernel and is used by direct VI Client connections as well is dcui, which is used by the DCUI process to identify itself as the VI API. It is the process that authenticates users and when communicating with other components in the system. keeps track of which users and groups have which privileges. The DCUI configuration tasks include: It also allows you to create and manage local users. • Set administrative password • The vpxa process