Mythbusting Goes Virtual Debunking Four Common vSphere “Truths”

Written by Scott D. Lowe

Introduction as truisms nonetheless even as the version count rises ever The information being presented in this paper comes courtesy higher. In this white paper, we will expose four such myths of the great minds of Eric Sloof, a VMware Certified Instructor, about vSphere. vExpert, consultant and active VMware community member; and Mattias Sundling, vExpert and Dell evangelist focused on Myth #1: RDMs have better performance than VMFS. the space. The information presented here was What is RDM? discussed in depth during an April 2, 2012 webcast with Mattias A raw device mapping (RDM) is created when a vSphere Sundling and Eric Sloof. administrator has configured a ’s virtual disk to point directly to, for example, a LUN (logical unit number) on Regardless of the underlying technology solution, as anything a storage array. With an RDM in place, a virtual machine can becomes increasingly popular and widespread in use, certain access storage just like it’s any other disk. pieces of sometimes inaccurate information about that product become permanent fact, often taking on legend-like status. RDMs operate as follows: The virtual machine’s initial access to Moreover, as a product matures, it changes; it evolves by an RDM virtual disk results in the virtual machine being pointed taking on new features, shedding old ones and improving the to a small mapping file. This mapping file is a symbolic link functionality everywhere else. However, no matter how much containing the raw ID of the intended storage on the storage a product matures and no matter how much it evolves, many array. Once it learns that raw ID, the virtual machine points products carry with them myths that follow through the ages. directly to the raw ID on the storage array and no longer needs Myths that may or may not have once been true, but are used to make use of the mapping file, as illustrated in Figure 1. Figure 1. A VM initially accesses an RDM virtual disk using a mapping file, but subsequently uses the raw ID.

The source of the myth which provides the greatest flexibility in Because the virtual machine is accessing managing the volume using native SAN storage directly and not going through tools. However, physical RDMs lose some some of the abstraction that takes of the features found with virtual volumes, place when the is placed including the ability to be snapshotted, in the middle, there is a myth that cloned, made into a template, or migrated RDMs have superior performance over if the migration involves copying the disk. virtual storage devices that make use of vSphere Virtual Machine It’s not unreasonable to make the (VMFS) datastores. Evidence of this assumption that “raw” would translate myth abounds in forum articles and into increased performance of the other resources outlining administrators’ virtual machine, but this myth has attempts to use RDMs to eke out as been well and truly busted and, in much performance as possible for fact, RDMs operate with performance storage-intensive workloads, such as characteristics on par with VMFS those supporting databases. storage. This is demonstrated as one starts to peer under the covers at what’s RDMs have two modes: virtual happening with the host system and, in and physical particular, how these storage calls are When considering the use of RDMs, interacting with the hypervisor kernel. bear in mind that they come in two By monitoring the kernel, the entire different flavors: story of how storage operates becomes • Virtual compatibility mode—When an RDM clear. Through this monitoring, an is configured in virtual mode, it appears to administrator can watch the “hidden” the guest just like a virtual story of storage activities and what disk does when it’s housed inside a VMFS impact these activities have on volume. With this mode, administrators overall performance. are still able to enjoy the benefits that come with the use of VMFS, including Testing the myth advanced file locking and snapshots. To evaluate this myth, Eric performed Further, because virtual mode continues to tests using three distinct scenarios, two provide a level of , it involving RDMs and one using VMFS as is more portable across storage hardware primary storage. The tests use a single than physical mode. virtual machine configured with a SCSI • Physical compatibility mode—When an adapter, but with four different volumes, RDM is in physical mode, the volume each configured like this: the characteristics of the mapped device, • Virtual RDM

2 • Physical RDM The results? • VMDK file on VMFS Busted! • A disk that connects to an iSCSI target through the use of the Microsoft iSCSI In testing, Eric discovered that there was initiator that ships with all current editions very little difference between either of of Windows the RDM configurations and the VMFS configuration. In other words, while Otherwise, the environment was there may be other reasons to choose configured as follows: a RDM-based volume over a VMFS- • vSphere 5.0, virtual machine hardware based volume, doing so for performance version 8. reasons alone isn’t necessary. • The virtual machine was running Windows Server 2008. VMware’s test results • It was configured with 4 gigabytes Even VMware has busted this myth in of memory. While there may a pretty big way, as shown in Figure • A single virtual CPU was added to the 2. The PDF file from which the chart virtual machine. be other reasons was sourced includes a wide variety of • The virtual machine was connected to the test cases that fully debunk the RDM vs. to choose a RDM- local area network’s Cisco 2960 switch. VMFS myth. • The storage being used is an Iomega PX6. based volume over Reasons to choose VMFS over RDMs In measuring directly the latency as a VMFS-based Now, understanding that performance storage commands make their way isn’t a reason to choose RDMs, what are volume, doing so through the kernel, Eric discovered that some better reasons to choose VMFS? there isn’t much of a difference in any VMware has spent years improving for performance of the storage configurations since they VMFS and, with vSphere 5, had made all have to go through the kernel, except reasons alone tremendous improvements to this the iSCSI option, which just goes out robust, cluster-aware file system with isn’t necessary. over the network and connects to an features such as: iSCSI target directly. However, at 1 Gbps, • Storage I/O control iSCSI had a top speed throughput rate of • Storage vMotion 112.6 MBps. • Storage DRS

Figure 2. Random mixed I/O per second (higher is better)

3 • Large volume size: 64 TB blocks in a virtual machine that have • Large VMDK file size: 2 TB changed since a point in time. This • Changed block tracking (CBT) support (CBT feature is incredibly powerful because tracks all of the storage blocks in a virtual backup and replication technologies machine that have changed since a point can rely on vSphere’s own vStorage in time.) advanced programming interfaces (), rather than either on drivers and When to choose RDMs over VMFS software developed from scratch or on Even tough RDMs don’t offer better traditional full and incremental backup performance, there are times when methodologies for data protection. an RDM should be considered. When a virtual machine needs access to a Requirements for using CBT particularly large single volume—one A number of requirements must be met that is greater than 2 TB in size—an Using CBT can for CBT to operate: administrator might consider using a • Since CBT was introduced in vSphere 4, the physical RDM, which provides direct drastically shrink host must be running at least that version access to a volume of up to 64 TB in size of vSphere. backup windows, and is not subject to VMDK file size limits, • CBT must actually be enabled for the virtual which remain at 2 TB. Note that this 64 machine. This will be discussed below. and the disk TB capability is valid only for physical • The virtual machine being tracked must be RDMs; virtual RDMs are still limited to a subsystem will also running virtual hardware version 7 or above. size of 2 TB. get less utilized • The virtual machine must be using a storage mechanism that runs through the vSphere Another time when RDMs may storage stack. Such mechanisms include during backups. reasonably come into play is when there VMFS, NFS and RDMs in virtual compatibility is a need to perform SAN snapshotting, mode. However, an RDM in physical which results in snapshots not supported compatibility mode is not supported. iSCSi by vSphere. Before a SAN can take a initiators installed inside a virtual machine snapshot, the virtual machine must be do not work with CBT, either. quiesced, which means that the virtual machine needs to flush buffered data to Benefits of CBT disk and prepare for the snapshot. If you By using CBT, administrators can are using SAN snapshots, which are not drastically shrink their organization’s communicating with the vSphere layer, backup windows since the backup then you need to use RDM with native application doesn´t need to scan the file systems, such as like NTFS or EXT3. VMDK files for block changes when Another scenario that requires the use doing incremental or differential of RDMs comes when there is a need to backups. Even when a full backup is cluster virtual machines with Microsoft performed, CBT can be useful in that the Clustering Services. Microsoft’s clusters CBT API will remove all the unallocated are not aware of what happens blocks when a full VMDK file is read, with VMFS and expect to see fully so only blocks that have actually been configurable and raw storage in order allocated to the virtual machine will be to operate. In that case, use an RDM processed. On top of reduced backup and present to the virtual machine the window the disk subsystem will also get storage that it’s expecting. less utilized during backups. Myth #2: Changed block tracking causes significant overhead on your CBT is enabled or disabled on a per-VM virtual machines. basis, so you can choose to use it for some virtual machine and not use it for What is changed block tracking? others. By default, this feature is disabled First introduced in vSphere 4, changed in a virtual machine and must be block tracking (CBT) is a VMKernel- proactively enabled by the administrator. based driver that tracks all of the storage Why is this the default? First, common

4 thinking holds that CBT introduces a a snapshot is removed from a virtual small amount of performance overhead machine. All of the changes are written in the virtual machine, which does from memory to the CBT driver, which make sense since there is additional then writes it to the CBT tracking file. tracking taking place. However, the myth Also note that the size of the CBT file goes that CBT introduces significant doesn’t change over time; all of the performance overhead. Further, vSphere space it needs is allocated upfront, so doesn’t know whether or not the backup this is a one-time hit. application in use supports CBT, so it’s possible that the feature remains Not one to simply state something as disabled so that the administrator can fact, Mattias took to his lab and ran make a decision that makes the most side-by-side tests. One test was with sense for the organization. That said, a CBT enabled and the other was with significant amount of today’s modern the feature disabled. He came to the The overhead vSphere-aware backup applications do, conclusion that he was simply unable to in fact, support CBT. detect any measurable overhead caused introduced by CBT by CBT except when it came to the CBT is so small as to Testing the myth file itself. That file he could actually see Mattias managed to locate one of the in the file system, so there was obvious be, for all practical creators of the CBT API at VMware and evidence that it was there, but the size asked him about some of the particulars was negligible. purposes, zero. related to CBT in order to gain a better However, the understanding of the performance The results? implications that come with CBT’s use. Busted! benefits of CBT From this resource, Mattias was able to gather the following tidbits: The conclusion is that the overhead are significant for • Memory impact introduced by changed block tracking organizations • CBT uses a maximum of 256 KB of RAM is so small as to be, for all practical for a 2 TB virtual disk. purposes, zero. However, the benefits of using CBT-aware • About 1.25 KB of RAM is required for CBT are significant for an organization every 10 GB of virtual disk space. that is using a CBT-aware backup and backup and • CPU impact replication tool. replication tools. • Flipping a bit when an I/O request is complete. Myth #3: Resource pools should • Storage always be used to categorize • Space: Change tracking file requires 512 and allocate resources to KB per 10 GB virtual disk. virtual machines. • I/O: When disk is closed, tracking Ways of managing resources in vSphere information is written to disk. There are many different ways that resources can be managed in vSphere. Based on these values, it’s safe to say • Resource pools—With the release of that the actual overhead incurred with VMware ESX 3 came the introduction CBT is just about zero. The memory of resource pools, which can be used impact is negligible. The primary impact to categorize virtual machines based is on storage, where the CBT file is on functionality or designation (such as created. This is a static file that requires production or test), and which can also about 0.5 megabyte for each 10 GB of be used to allocate resources based on a VMDK file. This impact is felt as soon allocated shares. A resource pool can be as CBT is enabled, since the CBT needs used to limit a group of virtual machines for to allocate upfront all the space that’s CPU and memory and to place reservations needed for tracking the information on these resources. The use of shares is of the VMDK file. Further impact is a way to give priority to more important experienced when a disk is closed. virtual machines in case of vSphere host is This may happen when, for example, running out of resources.

5 • vApps—There is another kind of resource when resource pools are being used to pool called a vApp. A vApp is a miniature manage resources for groups of virtual resource pool used to combine multi-tier machines. When a resource pool is web applications—such as a web server, created, it has the same default settings database server and a middleware server— as every other brand new resource pool. into a single vApp, which allows the related Most critically, regardless of how many resources to be managed in a central virtual machines reside in the resource way. With a vApp, you can configure the pool—five or five hundred—every startup order for the virtual machines that resource pool still has the same share are a part of the vApp and can distribute value. Again, that’s interesting because IP address to member machines and not every resource pool will have the configure shared resources, including CPU same number of virtual machines inside. and memory. In the sense that vApps share resources, they are a form of a resource All of these objects are living in the pool, sometimes considered a luxury vCenter inventory. As vApps and resource pool because there are usually resource pools are created, they are just a few virtual machines participating in created as siblings to the existing virtual the vApp. machines and each of these resource • Virtual machines—Another component pool types carries a share value of 4,000. that must be considered when considering For each vCPU that is included in a the use of resource pools is the individual virtual machine, that virtual machine virtual machine. Actually, it is necessary gets 1,000 shares. So, a virtual machine to think about all of the individual virtual with four vCPUs carries a share value of machines since they will participate in 4,000 as well. resource contests with vApps and resource pools. So, this means that each of these objects—a vApp with, say, three virtual So, there are three kinds of objects that machines; a resource pool with 20 are contending for resources at the virtual machines; and a single virtual vCenter level: resource pools, vApps and machine with four vCPUs—all receive virtual machines. the same number of shares, as shown in Figure 3. Every new resource pool has the same share value as any other resource pool Unfortunately, many people don’t realize what’s happening behind the scenes

Sibling CPU Shares Value

vApp = 4000

Resource Pool = 4000

Virtual Machine = number of vCPUs * 1000

Figure 3. A vApp with three VMs, a resource pool with 20 VMs, and a single VM with 4 vCPUs all receive 4000 shares.

6 Now, let’s suppose that an administrator By not paying adequate attention creates what is often referred to as a to these kinds of details, a vSphere “monster virtual machine” with 32 vCPUs. administrator risks creating a situation This virtual machine will consume that is not immediately obvious from a 32,000 shares. If placed at the same troubleshooting perspective and that level as the aforementioned objects, this can have a major negative impact on the means that this single virtual machine entire environment. will get eight times as many shares as the vApp and the resource pool that Bear in mind that this impact happens holds 20 individual virtual machines. only when the default resource pool Therefore, administrators need to act settings are kept in place. However, with caution when placing resources at from a myth perspective, it means that the same level in vCenter. resource pools really don’t make it much easier to manage virtual machine Example: how a pool of 24 Ghz is resources in any comprehensive way. divided among various resource pools Figure 4 illustrates what happens from The results? a per-VM perspective in the situations Busted! that have been discussed thus far. Note Resource pools that all of the objects shown reside at How to use resource pools effectively the same level in vCenter and therefore Obviously, if it’s really necessary, one can really don’t each object divides the available use vApps and resource pools, but it’s make it much resources—24 GHz worth—equally into imperative that the administrator refuse 6 GHz chunks. In each box, you can see to simply accept the default options that easier to manage what that division means for each virtual come with these containers and ensure machine that might be running inside a that the environment remains balanced virtual machine particular object. For the resource pool even while it’s constantly changing with resources in any with eight virtual machines, each virtual the addition and deletion of new virtual machine gets 0.75 GHz of resource machines. This means that the shares comprehensive way. capacity, while the virtual machines with must be manually adjusted for each four vCPUs get a full 6 GHz of resourcing. object in the environment to ensure that resources are allocated in a way that

Figure 4. How a resource pool of 24 Ghz is divided by default among various resource pools

7 • Ubuntu 10.04 or higher makes sense for the business. Further, • 2.6.33 or higher never ignore the impact that larger virtual machines might have on these sibling The origins of the myth objects. The monster VM may ultimately As introduced in vSphere 4, the PVSCSI consume more resources than desired. adapter had quite a number of problems and limitations. For example, upon the Figure 5 provides another example of initial release of vSphere 4, a PVSCSI how resources are divided. In this figure, adapter could not be used as the adapter there are two virtual machines and a for the boot volume of the virtual single resource pool. The resource pools machine. There was also guidance from gets 4,000 shares, the virtual machine VMware indicating that virtual machines with two vCPUs gets 2,000, and the with low I/O demand could experience single-vCPU VM gets 1,000. You can see worse I/O performance with the PVSCSI the corresponding percentages in adapter. The guidance indicated that the figure. the intent behind PVSCSI was to boost Myth #4: LSI Logic SCSI is always the performance of the monster VM better than paravirtualized SCSI. and decrease its impact on the vSphere host. It’s no wonder that the myth that What is the paravirtualized SCSI LSI should be preferred over PVSCSI (PVSCSI) adapter? continued to spread. However, these Introduced in vSphere 4, the issues were resolved in vSphere 4.0 paravirtualized SCSI (PVSCSI) adapter Update 1. was intended to provide increased performance to the storage subsystem Given that the initial problems with in a virtual machine (+12%) while PVSCSI have long been fixed, it’s decreasing the virtual storage adapter’s interesting to note that an LSI Logic impact on the vSphere host (-18%). SCSI adapter is generally the default Supported only in hardware version 7 or option recommended by vSphere when later, the PVSCSI adapter is supported a new virtual machine is created. Most by a subset of VMware’s guest operating administrators have been trained that systems, including the following: deviating from default recommendations • Windows Server 2003 or higher generally doesn’t make sense, so • Red Hat Enterprise Linux 5 or higher many move forward with this • SUSE Linux Enterprise 11 SP1 or higher adapter recommendation.

Figure 5. Shares can be divided appropriately among two VMs and a resource pool.

8 Paravirtualized SCSI is equal to or faster Figure 6. Options for a PVSCSI adapter than LSI SCSI and It’s still important to recognize that Testing the myth requires less the PVSCSI adapter isn’t supported by In order to see for himself how the every guest operating system, but the latest PVSCSI adapter operates, Mattias CPU resources. most common ones support it. When tested them head-to-head. In the same the PVSCSI adapter was introduced lab as was used previously, a vSphere 5 way back in vSphere 4, it was targeted environment, Mattias tested the LSI Logic at those monster VMs that needed controller and the PVSCSI controller particularly fast disk I/O; it was not against his iSCSI SAN environment. recommended for the average He used Windows Server 2003 virtual virtual machine. machines and followed all the best practices; for example, he ensured that there were no disk alignment issues.

Test result LSI Logic vs. PVSCSI LSI Logic PVSCSI

• Very similar disk throughput • Lower Host CPU Host CPU utilization utilization for PVSCSI 90MB/s – 60 000 IOPS

Figure 7. CPU utilization is significantly less for the PVSCSI controller than for the LSI controller.

9 From there, he ran a series of disk The results? performance tests, first with the Busted! LSI Logic controller and with a paravirtualized SCSI controller. As Paravirtualized SCSI is equal to or faster shown in Figure 7, it’s clear that the than LSI SCSI and requires less CPU throughput rates are practically identical resources. vSphere 5 carries none of the but the host utilization for the PVSCSI baggage that it did back in the vSphere controller is quite a bit less than it is for 4.0 days. Now that PVSCSI supports boot the LSI controller. The reason that the volumes and works well for even smaller, graphs don’t show much difference less I/O-intensive volumes, it’s an option in throughout is because the tested that should be carefully considered. disk environment was too small. It had The worst that will happen is receiving only six spindles and was running older identical performance, but it’s more than hardware, so it couldn’t move much likely that the environment will operate faster than seen in the chart. If the test more efficiently than with the default had used an up-to-date SAN with more LSI adapter. Consider using PVSCSI as a disk spindles, there would be clear part of your virtual machine templates throughput benefit. so you automatically get the benefits With the PVSCSI it provides when deploying new virtual However, as mentioned, there is machines. When it comes to boosting adapter, an significantly less CPU utilization with the the performance of and reducing the organization can PVSCSI adapter. As one considers today’s host impact from your existing virtual modern environments and constrained machines, identify your top 5-10 disk fit more virtual budgets, it’s important to achieve the intensive virtual machines and consider highest virtual machine density possible updating them to PVSCSI. machines on a in the virtual environment. With the single vSphere host PVSCSI adapter, an organization can Conclusion fit more virtual machines on a single The myths described in this paper were and push a little bit vSphere host and push a little bit more guidance that made sense at one time. throughput as well. However, as new product versions have more throughput been released and administrators have as well. Disk alignment learned more about exactly how certain Disk alignment remains an important features operate, these myths no longer consideration for older operating systems. represent the truth about vSphere. Let When a virtual disk is not aligned properly, the information in this paper be your there can be significant performance definitive answer to these four myths. degradation, sometimes approaching 25–30 percent. In order to improve performance and recover these lost cycles, vSphere administrators should always make sure that disks are aligned for operating systems that require such alignment, such as versions of Windows prior to Windows Server 2008. With Windows 2008, manual alignment adjustment is no longer necessary.

10 For More Information © 2013 Dell, Inc. ALL RIGHTS RESERVED. This document DELL ASSUMES NO LIABILITY WHATSOEVER AND DISCLAIMS contains proprietary information protected by copyright. No ANY EXPRESS, IMPLIED OR STATUTORY WARRANTY RELATING part of this document may be reproduced or transmitted in TO ITS PRODUCTS INCLUDING, BUT NOT LIMITED TO, THE any form or by any means, electronic or mechanical, including IMPLIED WARRANTY OF MERCHANTABILITY, FITNESS FOR photocopying and recording for any purpose without the A PARTICULAR PURPOSE, OR NON-INFRINGEMENT. IN NO written permission of Dell, Inc. (“Dell”). EVENT SHALL DELL BE LIABLE FOR ANY DIRECT, INDIRECT, CONSEQUENTIAL, PUNITIVE, SPECIAL OR INCIDENTAL Dell, Dell Software, the Dell Software logo and products—as DAMAGES (INCLUDING, WITHOUT LIMITATION, DAMAGES identified in this document—are registered trademarks of Dell, FOR LOSS OF PROFITS, BUSINESS INTERRUPTION OR LOSS Inc. in the U.S.A. and/or other countries. All other trademarks OF INFORMATION) ARISING OUT OF THE USE OR INABILITY and registered trademarks are property of their respective TO USE THIS DOCUMENT, EVEN IF DELL HAS BEEN ADVISED owners. OF THE POSSIBILITY OF SUCH DAMAGES. Dell makes no representations or warranties with respect to the accuracy or The information in this document is provided in connection completeness of the contents of this document and reserves with Dell products. No license, express or implied, by estoppel the right to make changes to specifications and product or otherwise, to any intellectual property right is granted by descriptions at any time without notice. Dell does not make this document or in connection with the sale of Dell products. any commitment to update the information contained in this EXCEPT AS SET FORTH IN DELL’S TERMS AND CONDITIONS AS document. SPECIFIED IN THE LICENSE AGREEMENT FOR THIS PRODUCT,

About Dell Dell Inc. (NASDAQ: DELL) listens to customers and delivers worldwide innovative technology, business solutions and services they trust and value. For more information, visit www.dell.com.

If you have any questions regarding your potential use of this material, contact:

Dell Software 5 Polaris Way Aliso Viejo, CA 92656 www.dell.com Refer to our Web site for regional and international office information.

Whitepaper-VirtualMythbusting-US-KS-2013-02-20