A Dell Technical White Paper 1

THIS WHITE PAPER IS FOR INFORMATIONAL PURPOSES ONLY, AND MAY CONTAIN TYPOGRAPHICAL ERRORS AND TECHNICAL INACCURACIES. THE CONTENT IS PROVIDED AS IS, WITHOUT EXPRESS OR IMPLIED WARRANTIES OF ANY KIND.

© 2010 Dell Inc. All rights reserved. Reproduction of this material in any manner whatsoever without the express written permission of Dell Inc. is strictly forbidden. For more information, contact Dell.

Dell, the DELL logo, and the DELL badge, PowerConnect, and PowerVault are trademarks of Dell Inc. Symantec and the SYMANTEC logo are trademarks or registered trademarks of Symantec Corporation or its affiliates in the US and other countries. Microsoft, Windows, Windows Server, and Active Directory are either trademarks or registered trademarks of Microsoft Corporation in the United States and/or other countries. Other trademarks and trade names may be used in this document to refer to either the entities claiming the marks and names or their products. Dell Inc. disclaims any proprietary interest in trademarks and trade names other than its own.

September 2010

Contents 1. Introduction Enterprises are seeking ways to streamline IT budgets without affecting operations and limiting features to end users. Microsoft Exchange® 2010 introduces new performance enhancements, which can significantly reduce storage requirements, and extends high-availability features. Exchange Server 2010 is officially supported by Microsoft on virtualized infrastructures and if designed correctly, can further reduce the total cost of ownership and maximize return on investment. In this white paper, we design and evaluate a 4000 mailbox Exchange 2010 deployment on Dell’s Virtualization Business Ready Configuration (BRC).

Virtualization Business Ready Configuration’s (BRC) from Dell are tools that help customers to minimize complexity during planning, ordering, deployment and installation of their virtual infrastructure. Dell offers multiple BRC’s that provides the customer a choice between multiple server and storage platforms, and for their virtualization infrastructure software. The BRC is optimized for running virtualized workloads and provide optimal redundancy, scalability and manageability. This document covers an approach to virtualize Exchange 2010 on the Dell Virtualization BRC for a Small Medium Business. The approach used in this document is hypervisor agnostic, and therefore the configuration can be used for hypervisors from Microsoft, VMware or any third party vendor that is validated by Microsoft to be used for Exchange Virtualization.

This white paper describes using the Business Ready Configuration for Virtualization to implement an Exchange Server 2010 messaging infrastructure on a VMware® vSphere™ environment. It documents the Exchange 2010 architecture, server and storage configurations, virtual machine sizing and best practices to enable an efficient virtualized Exchange 2010 messaging infrastructure. The best practices presented are the result of installing, sizing and evaluating Exchange Server 2010 components on a Virtualization BRC at the End-to-End Solutions Engineering Lab. Experimental results from the testing form the basis for sizing and performance guidance presented in this white paper to architect an optimal Exchange 2010 solution 2. Overview Dell offers multiple BRC’s that provide the customer a choice between multiple server, storage and virtualization platforms. Customers use the BRC’s from Dell for quick and efficient ordering, deployment and installation of their virtual infrastructure. The BRC used to host the virtualized Exchange 2010 building block presented in this document is available at http://content.dell.com/us/en/enterprise/d/virtualization/BusinessReadyDataCenter.aspx. This BRC combines the Dell PowerEdge™ M-Series blade servers and EqualLogic™ PS6000 storage area network (SAN) with Cisco® networking switches and VMware® vSphere™ 4 technology to help achieve an automated and efficient data center that can be used to host Exchange 2010 Server roles along with other virtualized workloads.

2.1 Hardware Components

Network

VM VM VM VM

VMware vSphere 4

1 2 3 4 5 6 7 8 9 1 0 1 1 1 2 1 3 1 4 1 5 1 6 P

0 0 0 0 0 0 0 0 M o 6 w 1 e 0 1 1 1 1 1 1 1 1 r

E b l d a g d e e 0 0 0 0 0 0 0 0

M s e 1 r

1 1 1 1 1 1 1 1 0 v 0 e 0 r s e

P o we r E d gM10 e 0 0 e

Storage Fabric E q u a S l e L r o i e g s i c

P A S N 6 0 0 0

Figure 1: Typical vSphere 4 Architecture with PS series SAN

The PowerEdge M610 is a two-socket blade server designed with energy-optimized options supporting the new Intel® Xeon® 5600 series processors, DDR3 memory and advanced embedded management capability.

The Dell EqualLogic PS6000 is an iSCSI storage device from Dell designed for deployment in enterprise business environments. It consists of four 1Gb Ethernet network ports per controller for data, faster processors, 2 GB of cache per controller, support for RAID 6, increased drive capacity, and a new monitoring application, SAN HQ, at no additional cost. For more information on Dell EqualLogic storage, see www.dell.com/equallogic.

In the following table, we provide a summary specification of the pre-configured solution. In the rest of the document, we will discuss reference architectures and best practices for deploying and using these solutions. The solution ID can be used to order or view the complete virtualization solution to deploy the reference architecture.

Table 1: Solution Specification

Hardware Solution ID 1131816.1 Blade Chassis (1) M1000e ESX Blade Server (6) x M610 Storage Device (4) PS6000XV Management Blade Server (1) x M610 Chassis Configuration I/O module for A1, C1, A2, C2 Cisco M 3130G I/O module for B1, B2 Cisco M 3130G with (2) 1000BAS-T SFP modules Stacking cables 6 Management Redundant Chassis Management Controllers (CMC) KVM Integrated Avocent keyboard, video, and mouse (iKVM) switch ESX Blade Server Configuration Blade Server Model M610

Processor 2 x Intel Xeon X5660, 2.8Ghz, 12M Cache,Turbo, HT, 1333MHz Memory

Memory 96 GB (12 x 8 GB, DDR3)

Add-in Controllers (2) Broadcom 5709 Dual Port GbE I/O Card for M-Series Blades

Local storage and controller Diskless configuration

Hypervisor VMware ESXi 4.0 Storage Configuration Storage Device (4) PS6000XV

Drives 16 X 600GB, 15K SAS

Storage Capacity 9.6 Terabyte capacity per Device

Management Blade Server Blade Server Model (1) M610

Processor (2) x Intel Xeon (Nehalem) E5520, 2.26Ghz, 8M Cache

Memory 8 GB

Network Controller (2) Broadcom 5709 Dual Port GbE I/O Card for M-Series Blades

Virtualization Management S/W VMware vCenter 4.0, 3Yr Upgrade Subscription

Software and Services Services  3 Year ProSupport for IT and Mission Critical 4HR 7x24 Onsite Pack

 EqualLogic PS Array Installation

 M1000e Blade Chassis On-site installation service 3. Deployment Best Practices for Virtualized Exchange Server 2010 Microsoft started supporting Exchange Server on Virtualized hardware with Exchange Server 2007. Exchange Server 2010 is supported as well, assuming certain conditions are fulfilled. In terms of hypervisors, Windows 2008 with Hyper-V Technology, Microsoft Hyper-V Server and any third party hypervisor (such as VMware and Citrix offerings) that are validated under Window’s SVVP (Server Virtualization Validation Program) are supported. The guest VM running the Exchange Server 2010 roles has to be deployed on Windows Server 2008 SP2 or R2 to be supported. All server roles are supported in a virtualized environment, except for Unified Messaging role, which has to be on a physical server.

Other important configuration or deployment scenarios that to keep in mind –

 Dynamic VHD disks are not supported; neither are snapshots, differencing or delta mechanisms

 Exchange 2010 HA features like Database Availability Groups (DAGs) cannot be combined with hypervisor based clustering solutions.

 Overprovisioning of virtual processor is not supported for a ratio greater than 2:1

 Minimum size of the fixed disk for the Exchange 2010 operating system virtual disk has to be 15Gb in addition to page file size (equal to the memory allocated for the VM).

 In a supported configuration, server based applications cannot be run on the physical root machine of a server that is hosting the Exchange 2010 roles inside virtual machines. It is okay to run management software like anti-virus, backup software etc on the root partition.

 Operating System disk images for the virtual machines have to be stored on serparate storage array or disks from the Exchange database or log storage.

Sizing the Exchange Server Mailbox Roles/VMs Exchange Server 2010 is not virtualization aware and therefore the majority of the design principles used for a physical deployment will translate to the virtual deployment design. Virtual machines are sized similar to physical servers, depending on the Exchange server role that is being hosted on the virtual machine. The number of cores and memory per virtual machine will depend on the message profiles and mailbox sizes. The following guidelines are recommended by Microsoft when deploying Exchange 2010 on a Hyper-V environment to size virtual machines hosting Exchange roles–

 The Hub and CAS server roles are combined in a single VM. This VM is assigned 4 virtual CPUs and 8GB of memory. The number of multi-role Hub/CAS deployed is the same as the number of Mailbox VMs deployed, i.e. the ratio of Hub/CAS VMs to Mailbox VMs is 1:1.  The Mailbox server role is assigned four virtual CPUs. The approximate maximum number of mailboxes recommended per Mailbox VM is 3000 to 4000 depending on the user profile and concurrency. The memory assigned to each Mailbox VM is 4GB (for OS) + memory needed for mailbox database cache. The requirements per user for mailbox database cache size ranges from 3MB-9MB (light profile to very heavy profile).

In the design presented in this paper, we use a multi-role Exchange VM which combines the HUB/CAS/Mailbox roles in a single machine. The goal of this design is to use a part of the resources available in the Virtualization infrastructure, and share the hardware with other virtualized applications and workloads.

 The number of drives required for database and logs is calculated using the same principles as a physical deployment. The Microsoft storage calculator is used to determine the storage design in terms of the number of drives and RAID type for the solution. For internal or direct attach storage configurations SCSI pass-thru is recommended where the volumes are presented directly to the Mailbox VMs for database and log volumes. For iSCSI SAN, the initiator is configured on the host server disk is again presented to the guest VM as a pass- thru disk to database and log storage. 4. Reference Architecture for 4000 Mailbox Virtualized deployments

The Agile Consolidated model from Dell for Exchange Server 2010 deployments are targeted for the virtualized datacenters that have already invested heavily in virtualization1. These datacenters are already running their Tier-2 and Tier-3 applications on VMs and looking at moving their Tier-1 apps on to VMs as well. The storage infrastructure for this class of deployment is on Dell SAN products such as Dell EqualLogic™ PS arrays and Dell/EMC CX4 and AX4 arrays.

A sample reference architecture for the Agile Consolidated Dell model using internal storage is shown in Error: Reference source not found. This architecture is designed for a 4000 mailbox with an I/O profile of 150 messages per day and mailbox sizes of up to 2GB. Table 2 lists the input variables that were used for this configuration.

Table 2 - Input Requirements for 4000 User Configuration

Input Detail Number of mailboxes 4,000 I/O Profile 150 Messages/Day Size of Mailbox Up to 2GB Average Message Size 75KB Number of Copies 2-copy DAG (1 active, 1 passive)

The high-availability for mailboxes is provided using built-in Exchange resiliency features thru the Database Availabiliy Group (DAG). The 4000 mailboxes are evenly distributed between four Exchange VMs, hosted on two separate physical servers. Two Database Availability Groups (DAGs) are created across the 4 VMs so that the mailbox copy is hosted on a separate physical server. The hypervisors are not part of a cluster since virtualization features are not used in this design to provide high- availability. The hypervisors not hosting Exchange, can be part of a cluster and applications running on them can be load balanced thru the virtualization software.

Each physical server hosts a multi-role Mailbox/Hub/CAS server VM. Error: Reference source not found details the host and VM configurations to support 4000 users. Full provisioning of resources is configured so that any server can handle a scenario where all the passive copies become active in the event of failure happening on a server or individual databases. High Availbility in the local site is provided by Exchange Database Availability Groups (DAGs) which provide protection at the application and server layers as well as the storage and database layers.

1 http://www.dell.com/downloads/global/solutions/security/exchange_2010.pdf M1000e enclosure M1000e enclosure with (6) M610 Exchange VMs blade servers for ESX, (1) M610 for e 1 2 3 d - n n - - n n

a vCenter and (9) half height slot for e l o o o o e e i i i i d B d d s s s s future expansion a a a l r n n n n l l e a a a a B B B

t i i p i p p p n x X X x X x x e S S E E S E E E E C E v 4 5 6 - n - - n n n n

Virtual Machines e o e e o o o o i i i i i d d d s s s s s a a a l n n n n n l l a a a a a B

B B VM Network

i i p p i p p p X x x X x x X x S S E E E E S E E E E

(2) Cisco (4) Cisco Catalyst Catalyst Switches Switches (4) 10Gb Uplinks with with with EtherChannel. Management stacking stacking VM, management and Network

s VMotion network e h

c Separated through VLANs. t i

w (4) more 10Gb uplinks S

(16) 1Gb active and t s

y for future expansion (16) 1Gb passive l a t redundant connection a C

with load balancing c s i C V X

0 LUNs 0 0 6 S P

c i g o L l a u q E

Figure 2 - Agile Consolidated Model on VMware

Figure 3 shows the consolidated hardware infrastructure on which the 4,000 user Exchange 2010 deployment is installed. Two PowerEdge M610 servers are used to host the multi-role Exchange Virtual Machine. Dell EqualLogic iSCSI SAN storage is used to host the Exchange Mailbox database copies, logs, and the VM OS images. The arrays are configured using RAID-10 to maximize performance and provide resiliency against disk failures at the storage level. Multiple volumes are used for storing the Exchange database and log files in addition to other volumes to hold the VM images and storage for other VMs.

Figure 3 shows the layout of the Exchange 2010 server roles and distribution of virtual machines on the servers. The 4,000 users are distributed across four Exchange VMs, each host running two Mailbox VMs. In the event of a server failure, Exchange DAG will activate the passive copy on working server to restore service to the failed mailboxes. Configuration details for the server, storage and virtual machines are shown in . Each PowerEdge M610 is configured with two quad-core CPUs and 96 GB of memory so it can handle additional load in case of server failure on the other host. Each Mailbox VM instance is configured with 4v CPUs and 48GB of memory and handles 1000 active mailbox users and 1000 passive users. The hosts are connected to the iSCSI SAN using the software iSCSI initiator in the host hypervisor.

DAG1 DAG2

Exchange Exchange Exchange Exchange VM1 VM2 VM3 VM4

e e g g d d 0 0 E E 1 1 r r 6 6

e 1 e 1 M M w w o o P P

MB VM1 MB VM2 MB VM3 MB VM4 V X

0 DB1 DB2 DB3 DB4 0 0 6 S P

DB5 DB6 DB7 DB8 c i g o L l

a DB3 DB4 DB1 DB2 u q E

DB8 DB7 DB6 DB5

Active Copy (first copy)

Passive Copy (second copy)

Figure 3 - Exchange 2010 Reference Architecture for 4000 Users

Table 3 – Server and Storage Configurations

Solution Output Detail Host Server Configuration 2 PowerEdge M610 2 x Quad-Core CPUs 6.4 GT/s 96 GB DDR4 Memory Mailbox/HUB/CAS VM 2VM’s per Host 4 vCPU, Memory per VM: 48GB 1000 Mailboxes Active and 1000 Mailboxes Passive per Virtual Machine Storage Configuration 4 x EqualLogic PS6000XV iSCSI Enclosures 16 x 3.5” 600 GB 15K RPM SAS disks per enclosure RAID-10 across the entire array Database Configuration Databases 4 database per Mailbox VM (data and logs combined per database) Total 16 databases (8 active and 8 passive databases) Mailboxes 500 Mailboxes per DB Up to 2GB capacity per Mailbox 4.1 Performance Study A smaller configuration of the reference architecture presented was tested and evaluated in the Dell enterprise solutions lab to validate the architecture and gather experimental data. Figure 4 illustrates the configuration that was tested - VMware ESX Server v4.1 is installed on both Dell PowerEdge M610 servers. Each host is running a single Guest VM installed with Microsoft Windows Server 2008 R2 Enterprise Edition. Exchange 2010 roles are installed on the guest virtual machines and host 1000 active and 1000 passive mailboxes each. The mailbox VM is assigned four virtual CPUs and 24GB of memory. It was configured with 2 vNICs, one connected to the public server network and the other was a private network configured for the DAG (Database Availability Group). The Exchange 2010 database and logs are hosted on volumes created on the externally connected EqualLogic PS6000XV array.

Exchange Exchange VM1 VM2 DAG

e e g g d d 0 0 E E 1 1 r r 6 6 1 e e 1 M M w w o o P P 2 1 M M

V DB1 DB2 V DB2 DB1

EqualLogic PS6000 XV

Active Copy (first copy)

Passive Copy (second copy)

Figure 4 - Performance Study Configuration and Setup

Figure 5 shows the average SendMail and 95th Percentile Send Mail latencies measured when simulating a 1000 user mailbox deployment on each virtual machine. The 95th percentile latencies are well within the 1-second threshold recommended by Microsoft for adequate end user performance. The SendMail latencies are still within the threshold when a Database Availability Group (DAG) is created across the two Exchange VMs and they host the passive copy of the databases. 300 Send Mail Latency 95th Percentile SendMail Latency 245 249 250 223 218

) 200 s m (

c 150 n e t a L 100 77 70 69 71

0 VM1 VM2 VM1 VM2 1K Mailboxes 1K Mailboxes 2K Mailboxes 2K Mailboxes (1K Active + 1K Passive) (1K Active + 1K Passive)

Figure 5 - Microsoft LoadGen Send Mail Latencies

Figure 6 illustrates the CPU and Memory usage for the ESX host and Virtual Machines when running 2000 mailbox LoadGen simulation on the standalone configuration and DAG configuration. As expected there is a minor increase in both CPU and memory usage for the DAG configuration as the Exchange server is hosting the passive copy in addition to the active mailboxes.

Physical Server CPU Utilization VM CPU Utilization Memory Usage - Physical Server Memory Usage - VM 16 30000 25922 ) 25537 25376 25950 s e

14 t ) 25000 y

21029 b % 20658

18948 18777 M ( n

20000 o

10 n i t o i a t z

i 8 15000 p l

i 15 t 13 m

U 6 u 11

10000 s U

4 o C C 5000 2 y 3 2 3 4 r o

0 0 m e

VM1 VM2 VM1 VM2 M 1K Mailboxes 1K Mailboxes 2K Mailboxes 2K Mailboxes (1K Active + 1K Passive) (1K Active + 1K Passive)

Figure 6 - CPU and Memory Usage for ESX Server and Exchange VM

The database and log performance metrics are shown in the following figures. In the stand alone configuration there are no log reads being performed. When a DAG is established, we see log reads on both Exchange VMs as an artifact of the DAG functionality. As expected the DB and log I/O’s are significantly higher in a DAG configuration due to the maintaining a database copy. The DB read latencies are observed to increase in the DAG configuration as well, but are within the recommended thresholds by Microsoft for acceptable performance. The latencies presented in the DAG configuration is a sum of the attached and recovery latencies – the attached read latency is still within 10ms which is half of the recommended threshold by Microsoft for optimal end user Exchange performance. 100 DB Reads/Sec DB Writes/Sec 87 90 86 Log Reads/Sec Log Writes/Sec 80 73 67 69 68 70 60 47 47 50 42 40 30 20 14 11 10 4 2.9 0 0 2.7 0 VM1 VM2 VM1 VM2 1K Mailboxes 1K Mailboxes 2K Mailboxes 2K Mailboxes (1K Active + 1K Passive) (1K Active + 1K Passive)

Figure 7 - Database Throughput Performance

25.0 DB Read Latency DB Write Latency Log Read Latency Log Write Latency 21.1 20.0 ) s 15.0 m 13.3 (

y c n e t 10.0 8.5 9.0 a L

5.0 2.6 2.4 2.1 2.0 1.5 1.01 1.2 0.71 0.78 0.82 0 0 0.0 VM1 VM2 VM1 VM2 1K Mailboxes 1K Mailboxes 2K Mailboxes 2K Mailboxes (1K Active + 1K Passive) (1K Active + 1K Passive)

Figure 8 - Database and Log Latency

Figure 9 shows how the ESX host and Exchange VM are affected when a DB failiure occurs on one of the Mailbox VMs. We simulate a failure on VM2 and this makes the copy on VM1 active, and it is therefore running 2000 active mailboxes. This was done to validate that the VM resources were adequate to handle a failure scenario at the database or host level. The chart plots the impact on resource usage over normal operation when the VM is running 1K active and 1K passive mailboxes. 100% 91.7% 92.1% 90% Relative Performance of VM with 2K active Mailboxes vs. 80% VM with 1K Active + 1K Passive Mailboxes t

c 70% a 63.4% p m

n 50% a m r 40% o f r

e 30% 26.1% P 23.4% 20% 10% 5.4% 2.3% 1.8% 0.0% 0% ESX Host CPU VM CPU Memory Usage Memory Usage MS ExchIS RPC MS ExchIS RPC MS ExchIS MS ExchIS MS ExchIS Utilization Utilization ESX Host Virtual Avg Latency Operations/sec Mailbox Mailbox Mailbox Machine Messages Messages Messages Queued Delivered/Sec Submitted/Sec

Figure 9 - Performance Impact when simulating a database faliure

At the Exchange application level we observe an increase in the RPC latency, operations/sec and messages being queued. This is expected behavior since all mailbox operations that was being handled by the two VMs are now being handled by a single VM. One VM is now hosting 2K active mailboxes, as compared to 1K active and 1K passive mailboxes. Hosting passive mailboxes does not incur RPC or Exchange related load on the VM, but has in impact on the storage subsystem. Since the VM was sized adequately to handle this addition load of 1K active mailboxes, there is only a minor increase in the resource utilization and end user performance is not impacted as verified by the SendMail latencies in the two scenarios. The DB read and write latencies were verified to be less than 10 ms which is well within the recommended threshold by Microsoft, proving that we have enough resources allocated to the VM to handle a database failure.

5. Summary This paper presented a reference architecture for deploying Exchange Server 2010 on virtualized Dell infrastructure. A four thousand user architecture was presented using the agile consolidated model from Dell. The high-availability for mailboxes was provided using built-in Exchange resiliency features thru the Database Availabiliy Group (DAG). The performance of the proposed architecture was evaluated using a scaled-down version and test results showed that the configuration was able to handle a 2000 mailbox deployment efficiently. The test setup can be scaled up to handle 4000 users without adding more blades since the CPU and memory usage was very minimal. Adding additional EqualLogic arrays result in a linear performance increase and will handle the I/O requirements for 4000 mailboxes. This setup can be shared with other virtualized applications, and scaled to accommodate future growth for the organization. 6. Reference and Additional Information

Dell Exchange 2010 Sizing Guide http://www.dell.com/downloads/global/solutions/Introduction_to_Exchange_2010_Sizing.pdf

Microsoft’s Exchange 2010 Mailbox Server Role Requirements Calculator http://msexchangeteam.com/archive/2009/11/09/453117.aspx

Dell Architecture Models for Exchange 2010 http://www.dell.com/downloads/global/solutions/security/exchange_2010.pdf

VMware vSphere™ Reference Architecture for Small Medium Business http://i.dell.com/sites/content/business/solutions/engineering-docs/en/Documents/VMware- vSphere-Reference-Architecture-SMB.pdf

Documentation for VMware vSphere 4 for Dell PowerEdge Servers http://support.dell.com/support/edocs/software/eslvmwre/VS/VS.htm

VMware vSphere 4 iSCSI SAN Configuration Guide http://www.vmware.com/pdf/vsphere4/r40/vsp_40_iscsi_san_cfg.pdf

VMware Network Concepts http://www.vmware.com/files/pdf/virtual_networking_concepts.pdf

Configuring VMware vSphere Software iSCSI with Dell EqualLogic PS Series Storage http://www.equallogic.com/resourcecenter/assetview.aspx?id=8453