“V” TO THE MAX John Bowling Storage Architect First National Technology Solutions [email protected] Table of Contents INTRODUCTION ...... 1 BUSINESS CHALLENGES ...... 1 THE VISION ...... 1 Cost Competitiveness ...... 2
Highest Levels of Reliability ...... 2 Ease of Management ...... 2 High Performance ...... 3 Data Mobility and Flexibility ...... 3 Maintenance and Support ...... 3 Compatibility ...... 3 THE SOLUTION ...... 3 Where We Were ...... 4 Why This Couldn‘t Continue ...... 4
Where We Went ...... 4 EMC VPLEX ...... 6 VPLEX Decision Points ...... 7 VPLEX Architecture Design ...... 7 Host Zoning ...... 8 VPLEX Back-End Connectivity and Zoning ...... 10 VPLEX Encapsulation ...... 13 EMC VMAX\VMAXe ...... 13 VMAX Key Decision Points ...... 14
VMAXe Architecture Design ...... 14 VMAXe SAN Connectivity ...... 15 VMAXe Zone Configuration ...... 16 VMAXe Storage Pool Design ...... 17 VMAXe Virtual Provisioning ...... 21 FAST VP ...... 23 EMC VNX ...... 25
2012 EMC Proven Professional Knowledge Sharing i
VNX Decision Points ...... 25 VNX Architecture ...... 25 VNX Fabric Connectivity ...... 26 VNX Zone Connections ...... 26 VNX Storage Pools ...... 28
VNX FAST VP ...... 30 EMC RecoverPoint ...... 31 RecoverPoint Key Decision Points ...... 31 RecoverPoint Architecture ...... 31 RecoverPoint Overview ...... 31 RecoverPoint Consistency Groups ...... 33 RecoverPoint SAN Connectivity ...... 33 RecoverPoint Zoning Configurations ...... 35 RecoverPoint Masking Configurations ...... 35
SAN fabric powered by a pair of Brocade DCX directors ...... 36 Key Decision Points ...... 36 Conclusion ...... 37 Appendix A – VMAX Engine FA Layout ...... 38 Appendix B – VNX Layout ...... 39 Appendix C – FAST VP ...... 40 Appendix D – References and Further Reading ...... 44
Disclaimer: The views, processes, or methodologies published in this article are those of the author. They do not necessarily reflect EMC Corporation‘s views, processes, or methodologies.
2012 EMC Proven Professional Knowledge Sharing ii
Table of Figures Figure 1 High-Level Storage Architecture Diagram ...... 6 Figure 2 VPLEX Host Fiber Connections ...... 8 Figure 3 VPLEX Zoning for Performance ...... 9 Figure 4 VPLEX Zoning for HA ...... 10 Figure 5 VPLEX-VNX Zoning ...... 11 Figure 6 VPLEX -VMAXe Zoning ...... 12 Figure 7 VMAXe SAN Connectivity ...... 15 Figure 8 VMAXe Host Zoning ...... 16 Figure 9 VMAXe Storage Pool Design ...... 17 Figure 10 VMAXe Storage Pool Data Devices ...... 18 Figure 11. VMAXe LUN (TDev) Example ...... 19 Figure 12 VMAXe LUN MetaVolume Example ...... 20 Figure 13 VMAXe MetaVolumes ...... 21 Figure 14 VMAXe Port Group Layout Example ...... 22 Figure 15 VMAX FAST VP Volume Example ...... 24 Figure 16 VNX SAN Connectivity Example ...... 26 Figure 17 VNX Host Zoning Example ...... 27 Figure 18 VNX Port Groupings ...... 28 Figure 19 VNX Tiered Storage Pools ...... 29 Figure 20 VNX FASTVP ...... 30 Figure 21 CRR Normal Dataflow ...... 32 Figure 22 CRR WAN Failure or Congestion Dataflow ...... 33 Figure 23 RecoverPoint SE (VNX) SAN Connections ...... 33 Figure 24 RecoverPoint EX (VMAXe) SAN Connections ...... 34 Figure 25 Example Brocade DCX Port Layout ...... 36 Figure 26 VMAX Engine FA Ports ...... 38 Figure 27 VMAXe Engine FA Ports ...... 38 Figure 28 VNX Example Design Layout ...... 39 Figure 29 High-Level VMAX FAST VP Pool Example ...... 40 Figure 30 VMAX FAST VP General Policy Example ...... 41 Figure 31 VMAX FAST VP Performance Policy Example ...... 42 Figure 32 VMAX FAST VP Storage Allocation Example ...... 43
2012 EMC Proven Professional Knowledge Sharing iii
INTRODUCTION Virtualization is prevalent in most of today‘s modern data centers and is the engine powering cloud computing infrastructures.
Modern storage platforms must deliver long-term cost-effective solutions in order for businesses to meet the onslaught of data and performance demanded by today‘s virtualized data centers. These solutions must be capable of handling today‘s requirements as well as flexible enough to meet tomorrow‘s emerging requirements.
BUSINESS CHALLENGES Most of the customers we meet all face similar business-related storage challenges:
Exponential Data Growth Flat to Lower Storage Budgets (Doing More With Less) Non-Disruptive Data Migrations Between Storage Subsystems (Technology Refresh) Shrinking to Near-Zero Recovery Point Objectives (RPO‘s) Shrinking to Near-Zero Recovery Time Objectives (RTO‘s) Shrinking to Non-Existing Maintenance Windows Meeting Increasingly Stringent Performance Service Level Agreements (SLA‘s)
This Knowledge Sharing article highlights several storage technologies and techniques implemented to overcome these obstacles, enabling us to meet even the most demanding cloud computing environments.
THE VISION We pride ourselves on designing and implementing enterprise-class could-computing solutions for our customers. We have invested in key infrastructure-related initiatives delivering upon a strategic long-term vision. With this vision in mind, we recently implemented a new infrastructure designed to solve the business challenges listed above.
We had several key criteria in mind when deciding on which platforms to implement to support our offerings:
Cost Competitiveness Highest Levels of Reliability Ease of Management High Performance Data Mobility and Flexibility Maintenance and Support Compatibility
2012 EMC Proven Professional Knowledge Sharing 1
Cost Competitiveness Being cost competitive is paramount in order to build and maintain business. Whether we are facing economic turmoil or economic boom times, we must ensure we offer solutions that fit our customer‘s budgets.
Some of the cost considerations include the following:
Initial Up-Front Infrastructure Cost o SAN fabric (Fiber and\or Network) o Cable Infrastructure o Storage Subsystems o Management Servers o Multi-Path Software (MPIO) Ongoing Sustainment-Related Growth Ongoing Maintenance and Support Staff Facility-Related Cost o Floor Space o Cooling o Power, etc.
Highest Levels of Reliability Offering cost-effective solutions is meaningless if the solution is plagued by outages. As more and more systems become virtualized, the impact from a single outage becomes increasingly devastating so we designed our systems with upmost reliability in mind. Storage infrastructures must be capable of performing even after suffering multiple component failures. The loyalty of our customers will be lost if we fail to meet our availability obligations. We strive to offer products and services that remain operational 24 x Forever.
An added benefit of highly reliable systems is the cost savings that are realized the longer the systems remain operational. Systems capable of remaining in production 7 or more years can yield significant long-term savings and\or profit over those capable of running production workloads for 3 to 5 years.
Ease of Management Following hand-in-hand with being cost effective and reliable, systems need to be as automated and easy to use as possible—being able to do more with less. The more complex the solution, the more resources it takes to maintain and operate over its life cycle—driving overall cost up while driving reliability down. Ensuring staffing levels remain stable in the face of unabated
2012 EMC Proven Professional Knowledge Sharing 2 growth is essential in cost containment and is the main reason ease of management remains a key requirement.
High Performance We are constantly asked to deliver the highest levels of performance even during peak usage periods. The overall solution must be capable of delivering during periods of high usage and must be designed to eliminate congestion points. Delivering solutions that suffer from poor performance frustrates customers and wastes precious time and resources tracking down and resolving performance-related issues.
Data Mobility and Flexibility Data mobility and flexibility is the secret sauce that saves everyone both time and money. Requirements change. Data patterns change. Infrastructure changes. We need solutions that allow us to move data from one tier of storage to another—without disruption. We need solutions that allow us to move data within and between storage arrays—without disruption. We need solutions that provide us the capability to move data between data centers—again, without disruption.
Maintenance and Support Expert support and locally knowledgeable technicians are important for long-term support. Some of the small, and not so small, vendors lack the depth and skill set necessary to tackle some of the more complex issues which leads to extended escalations and support on critical systems.
Compatibility Gone are the days of implementing independent computing silos. In order to maintain aggressive growth objectives, we need to ensure all of the systems being deployed are compatible and work with one another. It‘s expensive and difficult to maintain solutions designed in isolation. Everything needs to work together and scale in order to keep the overall solution as simple and manageable as possible.
THE SOLUTION The overall solution was architected to meet the business challenges while addressing the key criteria outlined above.
2012 EMC Proven Professional Knowledge Sharing 3
Where We Were Without the benefit of having a true long-term vision as guidance, we found ourselves in a situation where everything tended to be based solely on price. This left us in a state where we were faced supporting a complex environment filled with several low-cost storage and SAN fabric devices.
We were left with a myriad of low-end departmental switches tied together in a complex core- edge mesh that was difficult to maintain and support, jeopardizing the overall stability of the fabric.
In addition, we were also faced with several low-end storage arrays each with their own set of management and reporting tools. Data migrations were becoming increasingly difficult requiring longer and longer outage windows.
Why This Couldn’t Continue Overall Complexity o There were numerous interoperability support issues between systems due to the highly heterogeneous environment utilizing some not-so-common niche vendors o Complex disaster recovery (DR) scenarios due to the lack of an enterprise solution capable of supporting heterogeneous subsystems Lack of Enterprise Support o Some of the more obscure vendors lack locally skilled technicians o Some vendors have inadequate support staff capable of resolving some of the more complex issues o Some of our systems were plagued with reliability issues during firmware upgrades and other maintenance procedures Manageability o Manual reporting, procedures, and processes were required due to the disparity between systems o Lack of support for open tools due to the highly proprietary nature of several systems o Increasing staffing requirements to support multiple disparate subsystems Trustworthiness o With all of the issues listed above, we were at risk of losing the trust of our customers if we decided not to make the necessary long-term strategic changes required to sustain our business model
Where We Went With all of this as part of our guiding compass, we set out to design and implement a best-of- breed infrastructure capable of meeting the key business challenges as outlined earlier.
2012 EMC Proven Professional Knowledge Sharing 4
After evaluating several alternatives, we designed our infrastructure based on the following technologies: Cisco UCS, VMware ESXi, Brocade DCX SAN Fabric, EMC VMAX™\VMAXe™, EMC VNX™, EMC VPLEX™, and EMC RecoverPoint.
There are many other technologies that support our cloud environment, but this paper focuses on those directly impacting storage: VPLEX, VMAX, VNX, and RecoverPoint. Refer to Figure 1 for a high-level architecture diagram of the final solution.
We previously didn‘t own EMC arrays. In fact, there were those who flatly discounted EMC due to the misconception: ―they are too expensive.‖ But after evaluating all of the vendors, we actually found EMC was best able to meet our key objectives while also being cost-competitive.
*Please Note - The solutions and best practices applied throughout this document were designed around our specific set of requirements. Please consult with EMC before modifying your environment as best practices could change depending on your set of requirements, specifications, and environment.
2012 EMC Proven Professional Knowledge Sharing 5
Figure 1: High-Level Storage Architecture Diagram
EMC VPLEX We realized a new approach would be required to maintain and manage exponential storage growth. We could no longer afford to solely purchase additional arrays in silos; we needed to virtualize our storage.
EMC VPLEX offers a high performance, enterprise-class, storage virtualization platform that supports many heterogeneous platforms.
Virtualizing storage allows us to move data non-disruptively between different arrays. This addresses several business challenges:
Near-zero maintenance windows Exponential data growth Non-disruptive data migrations, etc.
2012 EMC Proven Professional Knowledge Sharing 6
Without storage virtualization, data migrations could take months due to excessive planning and coordination requirements, lack of adequate maintenance windows, extended system outage requirements, and so on.
An added benefit of the VPLEX architecture is the ability to ―stretch‖ LUNs across multiple data centers. This enables us to non-disruptively migrate computing resources from one data center to another.
VPLEX Decision Points Enterprise storage virtualization solution Heterogeneous solution that offers wide support for many storage vendors Non-disruptive data migrations within and between arrays Easy migration path into EMC VPLEX Supports active-active data center designs Enterprise Call-Home Monitoring and Support via EMC Secure Remote Services (ESRS)
VPLEX Architecture Design EMC VPLEX is offered in the following configurations:
Local – Data Mobility within Data Centers Metro – Data Mobility between Data Centers (Synchronous Distances) Geo – Data Mobility between Data Centers (Asynchronous Distances)
Each VPLEX cluster is capable of scaling from one, two, or four VPLEX Engines and can handle >2,000,000 IOPS and 23GB\s.
Each engine consists of a pair of directors. Each director is capable of supporting 4 x 8GB front- end ports and 4 x 8GB back-end ports for a total of 16 x 8GB fiber connections per engine.
We opted to start off implementing VPLEX Local to support non-disruptive data migrations between arrays within our main data center. VPLEX Local also supports High Availability (HA) LUNs by mirroring LUNs between two different storage subsystems. This allows for critical systems to be mirrored such that an entire storage subsystem could fail without system impact to the host.
EMC VPLEX allows us to zone, ―encapsulate‖, and virtualize existing host LUNs making migrations into the VPLEX environment simple and easy. After encapsulation, we can then add LUNs from other arrays (or within the same array) to seamlessly migrate data behind-the- scenes non-disruptively.
2012 EMC Proven Professional Knowledge Sharing 7
Figure 2: VPLEX Host Fiber Connections
Each engine has 8 x 8GB ports (4 per director) to be used for front-end host connectivity. Slots A0 on the odd (right) director and B0 on the even (left) director are dedicated for host connectivity.
As shown in Figure 2, best practices dictates that each director from each engine should be connected to both fabrics. In this example, all of the 0 & 2 ports are connected to Red Fabric A while all of the 1 & 3 ports are connected to Blue Fabric B.
This allows for the highest levels of availability in the event of a fabric, engine, or director failure.
Each engine is configured in a similar fashion.
Host Zoning There are two methodologies used for zoning hosts:
1. Zoning for performance 2. Zoning for High Availability (HA)
2012 EMC Proven Professional Knowledge Sharing 8
Figure 3: VPLEX Zoning for Performance
Figure 3 shows an example of host zoning optimized for performance. Due to the nature of how VPLEX distributes cache, maximum performance can be realized by zoning hosts across both directors within a single engine.
*Tip - Assigning port groups is an easy way to keep track of host connections. In this example, using the green color code, Windows Group 1 hosts should be zoned and masked to VPLEX Engine 1, ports B0-0, B0-1, A0-0, and A0-1.
2012 EMC Proven Professional Knowledge Sharing 9
Figure 4: VPLEX Zoning for HA
Figure 4 shows an example of host zoning optimized for HA. This provides the highest levels of availability in the unlikely event of an entire engine failure.
You can also see how your port groups differ utilizing the HA methodology.
VPLEX Back-End Connectivity and Zoning Back-end connections are similar to that of host connections except back-end ports utilize slots A1 and B1. Ports 0 and 2 are connected to Red Fabric A while ports 1 and 3 are connected to Blue Fabric B.
2012 EMC Proven Professional Knowledge Sharing 10
Figure 5: VPLEX-VNX Zoning
2012 EMC Proven Professional Knowledge Sharing 11
Figure 6: VPLEX -VMAXe Zoning
2012 EMC Proven Professional Knowledge Sharing 12
Figure 5 shows an example of a two-engine VPLEX system connecting to an EMC VNX subsystem. Each VPLEX director requires a path to each VNX LUN. In this example, the first two ports in each director have zones to each VNX Service Processor (SP) through both fabrics.
In keeping with best practice recommendations, we employ single-initiator zones for VPLEX back-end ports.
VPLEX to VMAXe zones follow a similar pattern used for VPLEX to VNX zones. For the highest levels of availability, each director requires visibility to each LUN through both fabrics as depicted in Figure 6.
VPLEX Encapsulation VPLEX Encapsulation is the process of virtualizing existing host LUNs. This process makes it possible to assign LUNs into VPLEX simply and safely while preserving data.
The high-level encapsulation process:
1. Power down host 2. Remove existing host LUN(s) (zones and masking) 3. Zone back-end storage array into VPLEX 4. Mask LUN(s) to VPLEX 5. Claim LUN(s) and create a single extent within VPLEX 6. Power on host 7. Zone host initiator ports to VPLEX front-end ports 8. Assign virtualized LUNs to the host
EMC VMAX\VMAXe Many of our customers have big time databases and highly intensive applications that require an enterprise-class, nonstop storage subsystem. We chose EMC VMAX\VMAXe as it delivers ultimate performance, scale, and reliability.
We were forced to underutilize some of our existing arrays due to the fact that they suffer from performance-related issues such as storage utilization increases. VMAX allows us to scale while adding performance (if needed). As we add storage bays, we can also add processing, cache, and bandwidth to adequately handle the additional storage characteristics.
Systems requiring ultimate performance and\or ultimate reliability are placed on the VMAX platform.
2012 EMC Proven Professional Knowledge Sharing 13
One key design concept we opted for was selecting the EMC VMAXe and VPLEX combo. In order to reduce our overall enterprise storage-related costs without compromising availability or performance, we selected the VMAXe over the full-blown VMAX. Doing this, we were able to cost-justify the addition of storage virtualization by way of the EMC VPLEX platform. Another justification for selecting VMAXe was our ability to utilize the built-in RecoverPoint splitter for a consolidated enterprise-class disaster recovery (DR) solution.
VMAX Key Decision Points Ultimate Scale and Performance Virtual Provisioning o Easy to use and fast storage provisioning capabilities Fully Automated Storage Tiering Virtual Provisioning (FAST VP) o Automated data migration between different tiers of storage o Non-disruptive data migrations within the VMAX o Automatically moves data to the proper drive tier (Flash, Fiber Channel, SAS, and SATA drives) o Lowers overall storage costs due to increased utilization of 7.2k SATA drives Symmetrix® Performance Analyzer (SPA) o Automated performance monitoring and trending charts Symmetrix Management Console (SMC) and Command Line Interface (CLI) o Easy to use management consoles
VMAXe Architecture Design A single VMAXe scales from 1 to 4 engines and is capable of supporting up to: 1,080 drives (Enterprise Flash Drives (EFD), SAS, and\or SATA) 512GB Cache 8 quad-core 2.4GHz Intel Processors 64 x 8GB front-end fiber ports 32 x 10Gb FCoE and iSCSI ports
2012 EMC Proven Professional Knowledge Sharing 14
VMAXe SAN Connectivity Figure 7: VMAXe SAN Connectivity
VMAXe Fiber Connections
F1 H1 F1 H1 VMAXe F0 H0 F0 H0 Engine E1 G1 E1 G1 2 E0 G0 E0 G0 FA4 FA4 FA3 FA3
H1 H1 VMAXe F1 F1 F0 H0 F0 H0 Engine E1 G1 E1 G1 1 E0 G0 E0 G0 FA2 FA2 FA1 FA1
Fabric A Fabric B
Host 1 Host 2 Host 1
Each engine has four slots capable of supporting 16 x 8GB fiber connections, which was how we opted to configure our VMAXe. There‘s also support for iSCSI and FCoE connection options.
Each ―0‖ port (E0, F0, G0, and H0) is connected to Red Fabric A while each ―1‖ port (E1, F1, G1, and H1) is connected to Blue Fabric B. This provides the highest possible levels of performance and availability within the SAN fabric.
Figure 7 shows SAN fabric connections for Engine 1. All other engines are connected similarly.
2012 EMC Proven Professional Knowledge Sharing 15
VMAXe Zone Configuration Figure 8: VMAXe Host Zoning
VMAXe Zone Configuration
F1 H1 F1 H1 VMAXe F0 H0 F0 H0 Engine E1 G1 E1 G1 2 E0 G0 E0 G0 FA4 FA4 FA3 FA3
H1 H1 VMAXe F1 F1 F0 H0 F0 H0 Engine E1 G1 E1 G1 1 E0 G0 E0 G0 FA2 FA2 FA1 FA1
Fabric A Fabric B
Host 1
Host zoning should be spread across multiple directors and engines whenever possible. Figure 8 shows an example of a typical zone configuration in a 2-engine environment.
Host Zoning Example
RedZone_Host1_VMAXe Host 1 Red WWN VMAXe FA2-E0 WWN VMAXe FA3-F0 WWN
BlueZone_Host1_VMAXe Host 1 Blue WWN VMAXe FA1-E1 WWN
VMAXe FA4-F1 WWN
2012 EMC Proven Professional Knowledge Sharing 16
VMAXe Storage Pool Design Figure 9 shows a typical storage pool design consisting of Flash, Fiber, and SATA drives. Each square represents a physical drive. For optimum performance, each drive type is spread across all available back-end ports.
Figure 9: VMAXe Storage Pool Design
Storage Pool Example
Pool 1 16 x 200GB Flash Drives
Pool 2 84 x 600GB 10k Fiber Drives
Pool 3 62 x 2TB 7.2K SATA Drives
Data Devices (DD‘s) are created by dividing each RAID Group into smaller segments and are numbered across each RAID Group similar to that shown in Figure 10.
2012 EMC Proven Professional Knowledge Sharing 17
Figure 10: VMAXe Storage Pool Data Devices
LUNs, (Thin Devices – TDevs), presented to the host are created by taking 768KB slices from Data Devices in sequence. This spreads the workload over all available drives within a given pool giving you more spindles for a given workload, even for small LUNs.
2012 EMC Proven Professional Knowledge Sharing 18
Figure 11: VMAXe LUN (TDev) Example
Figure 11 shows an example of a 100GB LUN created from the 10K Fiber pool. This volume is created by taking 768KB‘s slices from each DD until the required size is fulfilled, effectively utilizing all of the drives in that pool.
For larger volumes, or volumes that may need to be expanded in the future, TDevs can be combined together to form Meta Volumes, i.e., 4 x 100GB TDevs can be combined to form a 400GB Meta Volume that is presented to a host. See Figure 12.
2012 EMC Proven Professional Knowledge Sharing 19
Figure 12: VMAXe LUN MetaVolume Example
Meta Volumes can be created as concatenated or striped.
Concatenated Meta Volumes effectively fills up each TDev before moving to the next TDev. Since each TDev effectively is spread across the entire pool, this is an effective method that can be expanded very quickly. This is our default method for creating storage volumes for our hosts.
2012 EMC Proven Professional Knowledge Sharing 20
Striped Meta Volumes spread data allocation across each TDev in serial fashion. Once data is written to the last TDev, it starts back to the first TDev. Striped Meta Volumes can provide some performance improvement under certain workloads, but LUN expansion takes longer since data needs to be striped across all available TDevs. Essentially, to expand a TDev, data is copied to a Business Continuity Volume (BCV) device first. The original Meta is expanded and then the data is copied (striped) back to the larger Meta Volume.
Figure 13: VMAXe Meta Volumes
Figure 13 shows how a concatenated volume fills the first 100GB TDev before continuing on the next TDev. It also shows how 100GB segments are spread evenly across TDevs in Striped Meta Volumes.
VMAXe Virtual Provisioning Virtual Provisioning allows for quick and easy storage allocations within the VMAX subsystem.
Virtual Provisioning consists of 4 main components:
Storage Groups Port Groups Initiator Groups Masking Views
Storage Groups Storage groups contain all of the volumes (TDevs, Meta Devices, etc.) for each host or group of hosts.
2012 EMC Proven Professional Knowledge Sharing 21
Example Storage Groups:
ESXPROD_SG – (Volume(s) for the Production ESXi Server Farm) ESXTEST_SG – (Volume(s) for the Test ESXi Server Farm) APP01P_SG – (Volume(s) for APP01P)
Port Groups Port Groups contain the Front-End Adapters (FAs) that hosts use to connect to the VMAXe. Figure 14 shows an example port-group layout for a typical environment. Following best practices, we spread host I/O across all available engines and directors.
Figure 14: VMAXe Port Group Layout Example
Example Port Groups:
ESXPRD_PG – (FA1-G1, FA2-G0, FA3-F1, FA4-F0) AIXPRD01_PG – (FA1-F1, FA2-F0, FA3-G1, FA4-G0)
WINPRD01_PG – (FA1-F0, FA2-F1, FA3-H1, FA4-H0)
2012 EMC Proven Professional Knowledge Sharing 22
Initiator Groups Initiator Groups contain the WWN‘s for Hosts
Example Initiator Groups:
ESXPRD _IG – (WWN‘s for each ESXi Production Server)
AIXPRD01_IG – (WWN‘s for AIXPRD01) WINPRD01_IG – (WWN‘s for WINPRD01)
Masking Views Masking views combines the associated initiator, port, and storage groups. All associated masking and device assignments are automatically completed.
Tip: If you would like to add a host to the ESX farm, just add the WWN‘s to the Initiator Group and the Masking View automatically assigns the FA ports and storage to the new host.
Example Masking View:
ESXPROD_MV – (ESXPROD_IG, ESXPROD_PG, ESXPROD_SG) AIXPRD01_MV – (AIXPRD01_IG, AIXPRD01_PG, AIXPRD01_SG) VMAXe Fast VP WINPRD01_MV – (WINPRD01_IG, WINPRD01_PG, WINPRD01_SG)
FAST VP VMAXe Fully Allocated Storage Tiering Virtual Provisioning (FAST VP) automatically assigns data segments to the appropriate tier of storage at the sub-LUN level.
FAST VP moves segments in 10 x 768KB chunks within a volume based upon the FAST VP policy assigned to the Storage Group. See Figure 15 and refer to Appendix C for additional information.
2012 EMC Proven Professional Knowledge Sharing 23
Figure 15: VMAX FAST VP Volume Example
Take for example a 1TB LUN assigned to an ESX host or SQL Database. Most likely, the entire 1TB doesn‘t require equal performance block by block. There are likely blocks of storage that are very active and others that contain aged data that is accessed infrequently.
FAST VP automatically moves blocks that are highly active to higher tiers of storage while moving those accessed infrequently to lower tiers of storage.
This has enormous long-term impact to the solution‘s overall cost-effectiveness. We have found in testing that over 75% of our data eventually migrated down to our SATA pool while improving
2012 EMC Proven Professional Knowledge Sharing 24 performance. This means we can purchase more SATA drives over the subsystem‘s life cycle while dramatically reducing our overall storage costs.
Refer to Appendix C for test results and configuration of VMAX FAST VP.
EMC VNX In order to keep storage-related costs as low as possible, we determined that in addition to the VMAXe, we needed a general-purpose storage platform that was both highly reliable and could easily scale. We determined that the EMC VNX platform best fits that role.
The VNX platform offers a flexible architecture that‘s easy to use and can scale to 1,000 drives within a single subsystem.
VNX Decision Points FAST Cache Utilizing Enterprise Flash Drives o Extends system-wide cache o Ability to maintain low response times during bursts of activity Virtual Provisioning o Fast and easy-to-use storage provisioning Fully Automated Storage Tiering Virtual Provisioning (FAST VP) o Automated data migrations within the VNX o Moves data non-disruptively to the proper disk technology o Lowers overall storage costs due to better utilization of 7.2K NL SAS drives Unisphere Analyzer o Detailed performance metrics Unisphere Manager o Easy-to-use management console Enterprise Call-Home Monitoring and Support via EMC Secure Remote Services (ESRS)
VNX Architecture A single VNX subsystem can provide both Block and File services. This article focuses on block storage. VNX highlights (VNX 7500):
1,000 drives (Enterprise Flash Drives (EFD), SAS, and NL SAS) 48GB onboard cache FAST Cache extends onboard cache >4TB‘s Intel Xeon 5600 processors 32 x 8GB front-end fiber ports 16 x 10Gb FCoE ports 16 x 1Gb iSCSI or 10 x 10GB iSCSI ports
2012 EMC Proven Professional Knowledge Sharing 25
VNX Fabric Connectivity VNX fabric connectivity is similar to that of the VMAXe. Best practice is to connect both fabrics to both Service Processors (SPs).
Figure 16: VNX SAN Connectivity Example
The example in Figure 16 has even ports (0, 2, 4, and 6) connected to Red Fabric A and odd ports (1, 3, 5, and 7) connected to Blue Fabric B.
VNX Zone Connections VNX Zones are also configured similar to that of the VMAXe. Best practice is to configure each host to both fabrics belonging to both SPs. See Figure 17 for an example of VNX-to-Host Zoning.
2012 EMC Proven Professional Knowledge Sharing 26
Figure 17: VNX Host Zoning Example
Host Zoning Example
RedZone_Host1_VNX Host 1 Red WWN VNX SPA0 WWN VNX SPB0 WWN
BlueZone_Host1_VNX Host 1 Blue WWN VNX SPA1 WWN VNX SPB1 WWN
2012 EMC Proven Professional Knowledge Sharing 27
Figure 18: VNX Port Groupings
Figure 18 shows an example port layout for a typical VNX system. Best practice is to spread I/O across both Service Processors (SPs) connected to both sets of fabrics.
VNX Storage Pools VNX Storage pools are configured a little differently than that of the VMAXe. You can create traditional RAID groups, storage pools containing similar drive types and speeds (homogeneous pools), or pools consisting of multiple drive types and speed (heterogeneous tiered pools).
Since we decided to implement FAST VP, this article focuses on heterogeneous tiered storage pools.
2012 EMC Proven Professional Knowledge Sharing 28
Figure 19: VNX Tiered Storage Pools
Figure 19 shows an example of two storage pools consisting of 192 drives each. The first pool contains two tiers of storage (72 x 300GB SAS drives and 120 1TB NL SAS drives). The second pool consists of three tiers of storage (8 x 200GB Flash, 56 x 300GB SAS, and 128 NL SAS drives).
2012 EMC Proven Professional Knowledge Sharing 29
Best practices dictate the use of RAID6 due to the importance of pool protection when utilizing large drives. 1TB, 2TB, and larger drives take time to rebuild and pools that contain hundreds of drives require protection against double-drive failures.
Refer to Appendix B for a layout design for a typical system.
VNX FAST VP VNX‘s FAST VP is very similar to that of FAST VP for VMAX\VMAXe. Blocks of data at the sub- LUN level are distributed to the drive type based upon performance requirements. More frequently accessed blocks are promoted to EFD and SAS drives while less-frequently accessed blocks are demoted to NL SAS drives. One major difference between VMAX and VNX is that FAST VP for VNX moves blocks of data in 1GB extents as depicted in Figure 20.
Figure 20: VNX FASTVP
2012 EMC Proven Professional Knowledge Sharing 30
EMC RecoverPoint Many of our customers require a cost-effective disaster recovery (DR) option. RecoverPoint offers both an enterprise-capable and heterogeneous solution that is flexible and easy to use.
RecoverPoint Key Decision Points Heterogeneous DR Solution TiVo-like DR operations Ability to test DR while maintaining active replication sets EMC VNX and VMAXe both have built-in splitters that support RecoverPoint Synchronous and Near-Synchronous Capabilities Enterprise Call-Home Monitoring and Support via EMC Secure Remote Services (ESRS)
RecoverPoint Architecture EMC RecoverPoint Appliances (RPAs) provide a scalable DR solution capable of:
4 x 8GB SANFiber Connections per RPA 2 x 1Gb Ethernet WAN Connections per RPA Capable of growing from 2 to 8 RPAs per cluster Capable of protecting up to 300TB‘s per cluster Capable of three protection schemes o Local Replication - Continuous Data Protection (CDP) o Remote Replication – Continuous Remote Replication (CRR) o Concurrent Replication – Continuous Local and Remote Replication (CLR) RecoverPoint Deployment Options o RecoverPoint CL – Capacity-Based License o RecoverPoint EX – Used when deployed with the VMAXe integrated splitter o RecoverPoint SE – Used when deployed with a pair of VNX integrated systems
RecoverPoint Overview RecoverPoint ―splits‖ I/O from a host and sends it to the storage array and to a RPA device. This ―splitting‖ can occur in one of three locations:
1. Host Splitting – Via Host Agents (KDrives installed on the replicated host) 2. Fabric Splitting – Via Intelligent Switch Modules (Cisco and Brocade SAN Fabrics) 3. Storage Splitting – Via Splitters Built Into the Storage Subsystem (EMC VMAXe and EMC VNX)
2012 EMC Proven Professional Knowledge Sharing 31
For the purposes of this article, we focus on storage splitters built into the VMAXe and VNX storage arrays as they offer the most cost-effective and robust method for data replication within our environment.
There are three main volumes that make up a RecoverPoint solution:
Repository Volume – 3GB Volume that stores RPA configuration information Journal Volume(s) – Volumes that store point-in-time images at the target (DR) site Replication Volume(s) – Host LUNs that store data to be replicated
During normal data flow, I/O from a host is sent to both the Host LUN and to the local RPA Cluster. The local RPA cluster then transmits the I/O to the remote RPA cluster and is journaled at the remote RPA cluster. See Figure 21.
Figure 21: CRR Normal Dataflow
During periods of wide area network (WAN) congestion or WAN failure, I/O is sent to the Host LUN as well as a local journal volume. Once WAN communication is reestablished, data is copied from the local journal to the remote journal.
2012 EMC Proven Professional Knowledge Sharing 32
Figure 22: CRR WAN Failure or Congestion Dataflow
RecoverPoint Consistency Groups Consistency Groups are created to provide application consistency across multiple LUNS. You can place LUNs that require a consistent state in the same group, i.e., LUNs from web, application, and database servers can be grouped together to provide a consistent point-in-time across all LUNs.
RecoverPoint SAN Connectivity Each RPA should be connected to both fabrics within your environment. Figure 23 shows an example of a RecoverPoint SE deployment for EMC VNX.
Figure 23: RecoverPoint SE (VNX) SAN Connections
2012 EMC Proven Professional Knowledge Sharing 33
Figure 24: RecoverPoint EX (VMAXe) SAN Connections
2012 EMC Proven Professional Knowledge Sharing 34
RecoverPoint Zoning Configurations Host to Storage Zones – This zone is usually already in place and includes the host WWN and storage port WWNs.
VNX Host Zoning VMAXe Zoning
RedZone_Host1_VNX RedZone_Host1_VMAXe Host 1 Red WWN Host 1 Red WWN VNX SPA0 WWN VMAXe FA2-E0 WWN VNX SPB0 WWN VMAXe FA3-F0 WWN
BlueZone_Host1_VNX BlueZone_Host1_VMAXe Host 1 Blue WWN Host 1 Blue WWN VNX SPA1 WWN VMAXe FA1-E1 WWN VNX SPB1 WWN VMAXe FA4-F1WWN
RPA to Storage Zones – Includes all of the RPA ports and all of the Storage ports.
RPA-VNX Zoning RPA-VMAXe Zoning (2 RPA Example) (2 RPA Example)
RedZone_RPA_VNX RedZone_RPA_VMAXe RPA1 Red WWN RPA1 Red WWN RPA2 Red WWN RPA2 Red WWN VNX SPA0 WWN VMAXe FA2-E0 WWN VNX SPB0 WWN VMAXe FA3-F0 WWN
BlueZone_RPA_VNX BlueZone_RPA_VMAXe RPA1 Blue WWN RPA1 Blue WWN RPA2 Blue WWN RPA2 Blue WWN VNX SPA1 WWN VMAXe FA1-E1 WWN VNX SPB1 WWN VMAXe FA4-F1WWN
RecoverPoint Masking Configurations Gatekeeper (VMAXe), Journal, and Repository Volumes should be masked to each RPA device, not to the hosts.
Replication LUNs should be masked to both the host and RPA devices utilizing the same FA and\or SP ports used by the host and RPA devices.
2012 EMC Proven Professional Knowledge Sharing 35
SAN fabric powered by a pair of Brocade DCX directors Key Decision Points Performance – Each director can support up to 512 8GB ports (non-oversubscribed @ 8GB) Simplifies the overall design by collapsing 20 switches down to 2 Reduces overall port count due to the elimination of several Inter Switch Link (ISL) connections
SAN Fabric Best Practices: Implement an isolated dual-fabric design Spread subsystems and hosts wide across multiple blades Keep firmware up-to-date Implement single initiator zones (when appropriate) Keep storage subsystems and busy hosts as close to the core as possible Ensure you have adequate ISL bandwidth (if necessary) Implement aliases for host and storage WWNs Develop zoning naming standards Regularly backup and audit your fabric
Figure 25: Example Brocade DCX Port Layout
2012 EMC Proven Professional Knowledge Sharing 36
Figure 25 shows an example port layout for VNX, VPLEX, and VMAXe connections within a 4-blade DCX Chassis. For the best availability and performance, connections should be evenly spread across all blades as well as port groups within each blade. The DCX has two ASICs per blade as shown by the highlighted sections within each slot.
Conclusion Virtualization is a theme that permeates our long-term vision and is the key formula behind our success.
Virtualized cloud offerings will continue to grow unabated. Deploying a best-in-class solution utilizing EMC VMAX, VPLEX, RecoverPoint, and VNX subsystems will ensure we continue to deliver uncompromised virtualized solutions. Virtualize everything – ‗V‘ to the Max.
2012 EMC Proven Professional Knowledge Sharing 37
Appendix A – VMAX Engine FA Layout Figure 26: VMAX Engine FA Ports
Figure 27: VMAXe Engine FA Ports
Figures 26 and 27 show FA port numbering schemes along with fabric connectivity; red-colored FA‘s are attached to Red Fabric A while blue-colored FA‘s are connected to Blue Fabric B.
2012 EMC Proven Professional Knowledge Sharing 38
Appendix B – VNX Layout Figure 28 shows the disk, cabinet, SP port, and storage pool layout example for a VNX5700 subsystem.
Figure 28: VNX Example Design Layout
2012 EMC Proven Professional Knowledge Sharing 39
Appendix C – FAST VP Figure 29: High-Level VMAX FAST VP Pool Example
Storage Pools (Data Devices)
FASTVP Volumes FAST Pools 1 2 3 4 (Meta Devices) (Thin Devices) 5 6 7 8
200GB Tier 0 - 4.3TB Tdev 1 EFD R5 - 4.3TB 1TB 10 x 768KB 9 A B C HOST Meta 1 200GB Extents Tdev 2 D E F 10 Tier 1 - 14.7TB 200GB FC R1 - 14.7TB Tdev 3 Fully 11 12 13 14 Automated 200GB Tdev 4 Storage 15 16 17 18 FC R5 - 49.6TB Tier 2 - 49.6TB Tiering Virtual 200GB Tdev 5 Provisioning 19 1A 1B 1C
SATA R6 - 60TB 1D 1E 1F 20 Tier 3 - 60TB
2012 EMC Proven Professional Knowledge Sharing 40
Figure 30: VMAX FAST VP General Policy Example
General Volumes
(Meta Devices) (Thin Devices) FAST Pools
200GB Tdev 1
1TB HOST 10 X 768KB Meta 1 200GB EFD R5 - 4.3TB Tdev 2 Extents
200GB Tdev 3 General Policy 25% Tier0 200GB FC R5 - 49.6TB 75% Tier2 Tdev 4 100% Tier3
200GB Performance Window Tdev 5 M-F 7am – 6pm S 7am – 2pm SATA R6 - 60TB
Move Window 24x7x365
2012 EMC Proven Professional Knowledge Sharing 41
Figure 31: VMAX FAST VP Performance Policy Example
Performance Volumes
(Meta Devices) (Thin Devices) FAST Pools
200GB Tdev 1
1TB HOST 10 X 768KB Meta 1 200GB EFD R5 - 4.3TB Tdev 2 Extents
200GB Tdev 3 Performance Policy FC R1 - 14.7TB 75% Tier0 200GB 100% Tier1 Tdev 4 100% Tier3 200GB FC R5 - 49.6TB Performance Window Tdev 5 M-F 7am – 6pm S 7am – 2pm
Move Window SATA R6 - 60TB 24x7x365
2012 EMC Proven Professional Knowledge Sharing 42
Figure 32: VMAX FAST VP Storage Allocation Example
Figure 32 shows how FAST VP worked on an actual production VMAX system. We started off with no LUN assignments utilizing Tier 3 (SATA) storage. After 7 days, 45TB‘s migrated down from Tiers 1 and 2 into Tier 3 with no impact on overall performance. This has tremendous impact on lowering long-term costs. We were able to free up storage from Tiers 1 and 2. Future purchases will consist of more Tier 3 storage.
2012 EMC Proven Professional Knowledge Sharing 43
Appendix D – References and Further Reading "EMC® Symmetrix® VMAXe™ series with Enginuity™ The integrated RecoverPoint splitter for VMAXe." (2011): 5.
"EMC Recoverpoint Deploying with Symmetrix Arrays and Splitter Technical Notes." REV A05 (2011): 35.
"EMC Recoverpoint Deploying with VNX/Clariion Arrays and Splitter Technical Notes." REV A01 (2011): 30.
"EMC Symmetrix VMAXe Series Product Guide." REV A02 (2011): 112.
"EMC VPLEX 5.0 Architecture Guide." (2011): 40.
"EMC VNX Operating Environment for Block." Rev A06 (2011): 22.
EMC believes the information in this publication is accurate as of its publication date. The information is subject to change without notice.
THE INFORMATION IN THIS PUBLICATION IS PROVIDED ―AS IS.‖ EMC CORPORATION MAKES NO RESPRESENTATIONS OR WARRANTIES OF ANY KIND WITH RESPECT TO THE INFORMATION IN THIS PUBLICATION, AND SPECIFICALLY DISCLAIMS IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.
Use, copying, and distribution of any EMC software described in this publication requires an applicable software license.
2012 EMC Proven Professional Knowledge Sharing 44