SERVICE-ORIENTED STORAGE TIERING: A 2014 APPROACH TO A 2004 PROBLEM Daniel Stafford Senior Systems Engineer EMC Corporation [email protected] Table of Contents Introduction ...... 4
Defining the Sandbox ...... 4
Why Consider Storage Service Tiering on a VMAX? ...... 5
Designing the System ...... 7
Data Capture and Analysis ...... 7
Translating Analysis into Physical Design ...... 8
Translating Design into Cost Model ...... 10
Describing Tiers ...... 13
Configuration ...... 14
VMAX Configuration ...... 14
Host IO Limit Configuration ...... 14
Storage Group Configuration ...... 15
Pool Configuration ...... 17
FAST Policy Configuration ...... 18
Front-End Adapter (FA) Allocation ...... 19
Path Balancing ...... 20
vSphere Configuration ...... 21
VMDK Limits ...... 21
Use of Storage DRS and Datastore Clusters ...... 23
Datastore Size ...... 24
Queue Depth ...... 24
Monitoring and Reporting ...... 25
Native Monitoring in Unisphere for VMAX ...... 25
Performance Troubleshooting ...... 25
Unified Reporting in EMC Storage Resource Management (SRM) ...... 26
Simpler Aggregated Front End Statistics ...... 26
2014 EMC Proven Professional Knowledge Sharing 2
VMDK-Level IO Reporting and Mis-Tiering Detection ...... 27
Tier Capacity Reporting ...... 29
Automation and Orchestration ...... 30
Use of ViPR ...... 30
Acknowledgments ...... 31
Appendix A: Input Data Extraction...... 32
Step 0: Create your database schema ...... 32
Step 1: Parse Storage Group Mappings ...... 32
Step 2: Parse MiTrend Output ...... 36
Step 3: Produce output file ...... 37
Step 4: Load the output data into Excel ...... 41
Glossary...... 42
Disclaimer: The views, processes, or methodologies published in this article are those of the authors. They do not necessarily reflect EMC Corporation’s views, processes, or methodologies.
2014 EMC Proven Professional Knowledge Sharing 3
Introduction Defining the Sandbox The idea of defining IT offerings as a catalog of standard services has been with the industry for many years now. Whether your organization already has a well-defined set of offerings, or if as a Storage/Virtualization architect you are leading your enterprise on this journey for the first time, using the best available tools to provide consistent, competitive service is crucial.
To that end, this article will describe a set of best practices for building a tiered service offering using VMAX® arrays, with special consideration given to how to best integrate with a VMware- based virtualization stack.
Host level integration must be considered in such an environment because making a change to the way storage is presented can often have unexpected consequences up the stack. In some cases these consequences can be dealt with simply. Here we will consider some cross-silo effects of implementing storage tiering, and the best practices that can allow multiple groups to manage the environment together peacefully.
To do this, however, we should first define the ideal characteristics of our end-state.
Tiered Performance Offerings The solution should provide different categories of storage based on the performance required by the application. As enterprise applications vary widely in their performance needs, the model should be capable of running the gamut from low-cost archival-class storage all the way to pure- SSD levels of performance.
Application-level Flexibility Change in an application is inevitable, as are mistakes by the administrators running it. To that end our solution should be forgiving and allow the class of service provided to an application to be modified quickly and without disruption.
Environment-level Flexibility Even the most mature organizations will have trouble defining the total need of all applications. In organizations where defined tiers are new, the very act providing those tiers can and will change the behavior of application owners. The solution should easily allow the definition of a class of service to change, and provide paths to shift capacity between classes of service as demand for each class changes.
2014 EMC Proven Professional Knowledge Sharing 4
Avoiding the Tragedy of the Commons The performance levels published should be enforced. This prevents runaway applications from monopolizing the resources of the stack and reducing service levels for all applications.
Transparent The solution should provide reporting on the usage of the classes of service provided and should provide a view into whether the published service levels are being met. This should provide the basis for a practical chargeback system. This reporting should also help identify applications that may be mis-tiered so that action can be taken to either improve the performance of those applications, or conversely, save the application-owner from unnecessarily high chargeback.
Simplicity The extra engineering required to put a tiered service offering in place and maintain it is non- trivial. Care should be taken to ensure that this extra effort is not too onerous for any group in the organization. Too much complexity of administration not only eats away at the benefits provided, but is also an invitation to an unreliable system.
Automatable The solution should have a clearly defined set of rules that allow it to be driven by an automation/orchestration engine. Where examples are required, we will use VMware’s vCenter Orchestrator (VCO) in tandem with EMC ViPR®.
Why Consider Storage Service Tiering on a VMAX? The Economics of Shared Infrastructure While every large organization has a few applications that require customized engineering of the infrastructure stack, it is not feasible to provide this for every application for a number of reasons:
Customization takes time, but the business wants the application online tomorrow Customization requires engineers, and the business isn’t opening up new job requisitions without a fight Customized stacks waste resources – those custom stacks will often have excess capacity on one or more level, whether that means idle processors, unused storage capacity, or unused bandwidth
2014 EMC Proven Professional Knowledge Sharing 5
In contrast, storage consolidation provides higher utilization, greater flexibility, and allows the construction of an infrastructure stack that often has more 9s of availability than would be possible in isolated islands. Even if the cost of building a Shared Infrastructure were equal to the custom-per-application-stack approach, these benefits would result in shared spend providing greater benefits.
Competition, Internal and External Bringing an application online quickly is not just beneficial to the business in outrunning the competition. It also accrues benefits to the IT organization: more applications end up on the IT- owned stack rather than in rogue infrastructure or with another provider.
A properly executed tiered catalog approach doesn’t just speed up the technical part of provisioning for an application. It also allows predictability in cost, which allows business and application owners to more quickly and easily account for the IT portion of a project’s overall cost.
Comparing Enterprise and Public Service Providers Providing infrastructure-as-a-service offerings to internal customers with enterprise expectations can be very different from providing similar services to the public via a cloud portal. Enterprise application and line-of-business owners tend to have high expectations about the ability of the IT department to size for performance and troubleshoot performance issues when they arise. By contrast, the public cloud model puts this onus on the customer: if performance isn’t up to par, the customer needs to pay for a higher SLO.
Despite these different models and expectations, enterprise IT departments can (and must) learn some lessons about operating standardized, automated infrastructure at scale. In doing so, building service offerings that maintains their best assets – deep troubleshooting of systems- level issues and proactive management to expectations, not just SLOs – will ensure IT is seen as a valuable asset in the eyes of the business.
2014 EMC Proven Professional Knowledge Sharing 6
Designing the System Data Capture and Analysis For the purposes of this article, we will consider a scenario in which an enterprise already has a large dataset residing on a VMAX array (or arrays), meaning STP data is available. However, this same process can be easily modified for a variety of data sources.
Similar to a VMAX Cloud Edition (CE), we will define the performance capability of each service tier in terms of IO density.
By capturing STP data (or the best available equivalent), we can describe the distribution of IO density requirements in the environment. This simple output dataset will allow us to easily experiment with different service tier definitions to find the mix that’s right for the environment.
EMC and Partner System Engineers have access to a tool called MiTrend which produces a spreadsheet that can be used for analysis. For input, a copy of the SYMAPI database is required, as well as at least one week of STP data. Ideally this STP data should capture peaks for critical applications, such as end-of-quarter processing.
Sample scripts for parsing this data can be found in Appendix A. Meanwhile, we will skip straight to the output.
For developing a set of service tiers, it is helpful to use the log of the IO Density rather than using the IO Density directly. This is because a typical large environment may have densities that range over 5-6 orders of magnitude. Using the log compresses resulting graphs and lists to a reasonable length.
In the graph shown in Figure 1, IO Density limits for each of four Service Tiers (Bronze, Silver, Gold, and Platinum) have been selected such that the first three have approximately equal capacity, and Platinum takes a relatively smaller share. This was done with four goals in mind:
2014 EMC Proven Professional Knowledge Sharing 7
Ensure that the most common Service Tiers will have a healthy demand Separate the tier for the most demanding applications so the more expensive, higher performance storage does not result in high blended costs for the more common tiers Separate the IO Density limits of the tiers sufficiently that they provide real differentiation Ensure that all or nearly all of the workload is covered by a Service Tier o In this case, 1.4TB out of 826TB had performance demands above 20 IOPS/GB, and was therefore Out of Bounds
For the sample dataset, these tiers were defined as:
Bronze – Up to 0.03 IOPS/GB Silver – Up to 0.4 IOPS/GB Gold – Up to 3.2 IOPS/GB Platinum – Up to 20 IOPS/GB
Note that each tier is approximately 10x the previous. If your selected tiers increase by less than this (2-3x), it will likely simplify the environment to consolidate Pro Tip: Use a small number similar tiers into a single tier. If application owners are of tiers – 3 or 4 at most presented with more than 3-4 options, they may perceive the selection process as too complex. Additionally, every additional tier creates additional overhead for the administrative team that must manage the environment.
On the other hand, with just one or two tiers there may not be enough differentiation in the cost model for each tier to be competitive with the alternatives. To paraphrase a well-known physicist, the set of available service tiers should be as simple as possible, but no simpler.
Translating Analysis into Physical Design It is important to note that applying a Service Tiering approach does not necessarily change the workload that must be supported. While there may be some intentional down-tiering by application owners who can accept lower performance in exchange for a lower-cost storage tier, it should be assumed that the overall workload will not change when moving into a tiered environment.
2014 EMC Proven Professional Knowledge Sharing 8
With this in mind, all traditional design advice still applies. If a trusted target VMAX design is already available, this is likely the best starting point for a physical design. The tiering approach described in the coming sections is intended to Tier Combined Peak % of overlay existing best practice VMAX IOPS Total configurations. Bronze 1,384 0.18% Suppose a starting VMAX disk configuration as follows, which uses capacity tier ratios typical Silver 51,294 6.71% across many environments: Gold 359,050 46.98% 17.2TB EFD – 3.5% Platinum 352,576 46.13% 137.6TB 15k – 27.1%
343.8TB SATA - 69.4% Out of 74,413 9.74% Bounds We will divide these high level capacity tiers into smaller units to support the Service Tiers Total 764,304 100% developed above. Using the same skew summary data, we can get an idea of the peak IO that will be demanded of each tier (Figure 2). We know that these peaks will not happen at the same time, and can assume that this factor is constant across the input array(s). In this case, the sum of the individual volume IO peaks (764k IOPS) is about 5.5x the peak IO observed on the array as a whole (140k). This means we can assume, for example, that the Bronze tier will only peak at 250 IOPS rather than 1383.
Based on this, we will assign portions of the high level capacity tiers to the Service Tiers. Since Platinum has a relatively smaller capacity, we will combine it with Gold.
Table 1: Relative Shares of Capacity Tiers assigned to Service Tiers
EFD 15k SATA
Gold/Platinum 80% 80% 12%
Silver 20% 20% 38%
Bronze 0% 0% 50%
2014 EMC Proven Professional Knowledge Sharing 9
This results in three ‘virtual arrays’ with different capacity ratios. Note that these ratios correspond roughly to what an EMC/Partner Systems Engineer might design if asked to design a VMAX array for that tier alone. (For example, Silver is 240TB with 2x thin provisioning and a peak of ~10k IOPS might result in the disk mix shown in Table 2)
Table 2: Capacity Ratios of Each Service Tier Pro Tip: Treat your Tiers like ‘virtual arrays’ when Gold/Platinum Silver Bronze assigning disk ratios
% EFD 8.3% 2.1% 0%
% 15k 66.7% 17% 0%
% SATA 25% 80.9% 100%
Translating Design into Cost Model One of the most important parts of a cost model is that it needs to be competitive with alternatives. If an application requires 20TB of Bronze-class storage, it may be very problematic if they can build a dedicated array with similar characteristics for substantially less than the chargeback. For this reason, the method of developing a cost model shown here may be very different based on the circumstances in your enterprise.
First, we will define which components of the storage infrastructure it makes sense to charge an application owner for. At a high level, this may be:
TB of Capacity Subscribed per Tier o Using subscribed rather than allocated capacity can allow some flexibility in setting chargeback, as well as providing a ‘profit margin’ if this is necessary to your accounting/budget structure SAN Switch ports o This can be one of many incentives to virtualize or use dense blade-based infrastructure, where switch port cost can be defrayed across many applications TB of Capacity Replicated Remotely o Via SRDF®, RecoverPoint, etc TB of Capacity Replicated Locally o Via TimeFinder® Clones/Snaps, or RecoverPoint Local
2014 EMC Proven Professional Knowledge Sharing 10
A few simple rules can be used to arrive at starting chargeback price for most of these items using the Bill of Materials for the target VMAX
The cost of components shared across tiers (such as engines, racks, or base software) is allocated to each tier based on the percentage of peak IO expected o Extending our example, the Silver Tier would bear 6.71% of the cost of the VMAX engines, whereas the Gold Tier would bear 46.98% of that cost o Again, it may be helpful to combine Gold and Platinum at this point for a total of 93.11% o This also applies to items directly on the system BOM, such as the costs of floor space and power/cooling. This accounts for the relatively higher density and lower power utilization of lower storage tiers. Components that provide actual capacity will be charged to a tier based on the percentage of that tier allocated o Here, the Silver tier would bear 38% of the cost of the SATA disks, while the Bronze tier would pay for 50% of them Software costs for a particular feature (i.e. TimeFinder) are charged only to the usage of that feature. If using Raw capacity licensing, it may be helpful to estimate what the total usage will be and use this to determine a per-TB cost for this feature.
Table 3: Sample Calculations Demonstrating Cost Distribution Rules
Gold/Platinum Blend Silver Bronze
$1.5M Shared Cost $1.39M $100k $2700
$500k EFD Cost $400k $100k $0
$500k 15k Cost $400k $100k $0
$250k SATA Cost $30k $95k $125k
Tier Subtotal $2.22M $395k $127.7k
@ 498.6TB Usable $13,452/TB $2444/TB $743/TB
2014 EMC Proven Professional Knowledge Sharing 11
$13.13/GB $2.39/GB $0.73/GB
@ 850TB Usable $7,937/TB $1441/TB $438/TB after Thin $7.75/GB $1.41/GB $0.43/GB
We can then further break down the Gold/Platinum cost into charges for those two tiers based on the facts that 1) Each accounts for an even share of performance, but 2) Platinum only accounts for 16% of the Gold/Platinum capacity.
Table 4: Distributing Cost Among Gold/Platinum
Gold Platinum
$2.22M Tier Cost $1.11M $1.11M
@ 165TB Usable 139TB @ $7986/TB 26TB @ $42,692/TB
$7.80/GB 41.69/GB
@ 280TB Usable after Thin 235TB @ 4273/TB 45TB @ $24,667/TB
$4.61/GB $24.09/GB
Again, an actual enterprise may have need to tweak these models further. Examples might include: Providing a discount to storage allocated to VMware to encourage a virtualization strategy Raising or lowering tier costs to meet previously set expectations Rounding to whole numbers for simplicity
A common question about this sort of Service Tiering model is “What is stopping an application owner from just doubling their Silver order to get to the total level of IO needed based on the published density?” Fortunately, the model accounts for this Pro Tip: Performance difference behavior – we have chosen Service Tiers that are roughly 10x between Tiers should be greater separated from the adjacent tier in performance capability, but than their cost difference
2014 EMC Proven Professional Knowledge Sharing 12 are only 3-5x separated in cost. This means it will almost always be in the application owner’s interest to choose a smaller amount of higher-performance storage if a decision is on the edge.
Describing Tiers It’s one thing to describe a service tier in terms of expected IO Density capability to a seasoned storage engineer. It’s quite another to try the same thing with a Database Administrator or Project Manager. For this reason, it is important to define a more readable catalog describing the performance levels offered. For example:
Tier Tech Specs Cost Description
Bronze 0.03 $0.60/GB Low performance bulk storage, used for things that IOPS/GB aren’t accessed much. If you would feel comfortable with the performance of your home PC’s hard drive for this data, this is the appropriate tier. Examples include scratch space, temporary files, backup dump volumes, and archival data.
Silver 0.4 $2/GB Medium performance, for applications where some IOPS/GB delay is acceptable, especially those where end-user response or completion time is not critical. Tier 2 Databases, file shares, application hosts that don’t rely heavily on storage, archive logs, and low-end virtual machines would reside here. This is a good general- purpose tier.
Gold 3.2 $6/GB This is the performance tier most appropriate for things IOPS/GB like high-end databases, critical boot devices, redo logs, and other demanding applications.
Platinum 20 IOPS/GB $30/GB For applications with the most critical, demanding needs. If the application vendor mentions that All-Flash Arrays are common in successful implementations, this is the right tier to choose.
2014 EMC Proven Professional Knowledge Sharing 13
Configuration VMAX Configuration Host IO Limit Configuration In the previous section, we arrived at a set of IO Densities for each tier. In order to enforce these performance policies at the VMAX, we will use a feature introduced in Enginuity 5876 called Host IO Limits. These limits are applied at the VMAX Storage Group level, and can be defined as an IOPS limit, a MBps limit, or both.
At the very simplest level, the formula we are applying when setting this parameter is:
However, a number of other factors will necessarily modify this in a real-world environment.
Host IO Limit Granularity Host IO Limits must have a minimum granularity of 100 IOPS. Suppose a Bronze Tier with an IO Density of 0.03 IOPS/GB, and a requested Bronze capacity of 1TB. While the simple formula would suggest this Storage Group would be assigned a limit of 30 IOPS, we must assign it 100 IOPS. Pro Tip: Round Host IO Limits up to the nearest While it might seem that such a deviation from policy is not hundred desirable, in practice this is a positive development. Its consequence is that if an application owner requests low-performance storage, it will typically behave as if they were using a locally attached consumer hard disk. This can be a useful description when marketing the tiers internally, because even less storage-savvy application owners can intuitively grasp it: A tier with IO density of less than about 0.1 IOPS/GB is appropriate for workloads that would perform acceptably on your home PC.
Performance of Bursty Workloads The Host IO Limit feature is built on a “leaky bucket algorithm”. This means that new IO may be accepted at the rate it comes in (subject to all the normal VMAX cache and queuing rules), but it is serviced no faster than the IO limit. There is currently some limited bursting functionality built into the algorithm to allow IO to be briefly serviced faster than the limit, but this is primarily useful for very short (<1-2s) bursts.
2014 EMC Proven Professional Knowledge Sharing 14
Pro Tip: Add 25-40% to While most workloads will not notice this change in servicing the published limit when behavior, those that typically operate near the limit will see provisioning additional latency injected, especially if they have a bursty rather than steady profile. To ensure that we can demonstrably reach the limit, as well as mitigate impact for applications that operate near it, a common practice is to add a percentage (25-40%) on top of the published limit.
In practice this means that the Service Catalog might publish 0.4 IOPS/GB for Silver and 3.2 IOPS/GB for Gold, but the actual limits might be set at 0.6 IOPS/GB and 4.0 IOPS/GB, respectively. Besides ensuring that applications with a long history prior to implementation of Service Tiering are not impacted, this also can provide an important psychological buffer for application owners that might be suspicious of a new limit. If the suspicious application owner requests a 3.2 IOPS/GB tier and sees 3.5 IOPS/GB in testing, he or she is much less likely to raise a red flag than if the volumes only achieved 3.1 IOPS/GB.
RecoverPoint Replicas A RecoverPoint replica should have the same or higher IO limit which is applied to the production device. This is because it will have a similar workload in most cases. For devices that experience high levels of writes (>50%), a higher limit may need to be applied.
For journal devices, it may be possible to use a lower tier than is assigned to the replica. This is because most IO to journal devices is very large block (~1MB). As long as the block size of the production workload is small (<64kB), a lower Service Tier is not likely to cause replication impact. If throughput (MBps) limits are applied in addition to IOPS limits, this recommendation does not apply.
In a recovery scenario, it may be advisable to remove limits entirely from the replica and journal devices. This will prevent Host IO Limits from slowing down distribution to the chosen point in time.
Storage Group Configuration Rules for Host IO Limits on Devices and Storage Groups There are a few rules that must be considered when provisioning. Awareness of these can simplify changes down the road.
2014 EMC Proven Professional Knowledge Sharing 15
A volume can only have one active Host IO Limit o “Active” means “Assigned to a Storage Group in a Masking View” If a volume is in a masked Storage Group with a Host IO Limit, it cannot be in another masked Storage Group without a limit The Host IO Limit quota is distributed and enforced over the FA ports in the masking view
The effective result of these rules is that if you wish to place a Host IO limited volume in multiple masking views, those Masking Views must share the same storage group and same port group.
These rules do not prevent an administrator from creating an unmasked Storage Group for the purpose of collecting performance data on a set of volumes. For example, it would be legal to create a Storage Group with every R1 device on the frame to allow easy capture of input load on SRDF, since that group would not be in a Masking View.
Using Cascaded Storage and Initiator Groups A very useful pair of Enginuity features has flown under the radar for the past few years: Cascaded Storage and Initiator Groups. These allow the configuration of parent/child relationships in Storage and Initiator Groups.
The use of Cascade Storage groups is very helpful here. However, there are some changes in methodology required from many of the traditional documents describing Cascaded SGs.
For a host or cluster, create a parent storage group. This should not have any applied limits. Create a child storage group for each required Tier o Masking views must be associated with the Child Storage groups to ensure maximum flexibility o If you use child storage groups to organize different application components (for example, separating database and redo volumes on an Oracle host), this does not need to change. Simply apply appropriate limits on these existing child groups Pro Tip: Mask using For clusters, create a parent initiator group for the cluster Child Storage groups to and a child group for each cluster node. The parent group maximize flexibility can be used for masking volumes common to the entire cluster.
2014 EMC Proven Professional Knowledge Sharing 16
The association of masking views with Child rather than Parent Storage Groups is the biggest change here. While this sacrifices some of the convenience of masking to the Parent SG, it is necessary to ensure that volumes have mobility between Tiers.
Using the VMAX RecoverPoint Splitter As long as the above rules are followed, there are no conflicts in using Host IO Limits with RecoverPoint. When masking volumes to RecoverPoint, the same Storage Group and Port Group used for masking to the production host/cluster should be used. The only difference will be the use of the RecoverPoint cluster initiator group.
Pool Configuration This is an area where we will trade a little complexity for some greatly simplified reporting and monitoring capability. However, an administrator willing to put the right level of automation into managing creation of meta-storage-groups (groups used just for reporting purposes) as well as a deep analytics tool like EMC SRM might be able to eliminate this modest amount of complexity and realize the best of both worlds.
Each Service Tier will consist of its own set of pools. These pools will be composed of TDATs horizontally striped across all available disks of the tier. While this introduces some possibility for pool workloads to compete, the benefits gained through better DA balance and higher peak performance.
In the case of the example we have been developing, a total of seven pools would be constructed.
Gold/Platinum Tier Silver Tier Bronze Tier
EFD G_EFD == 80% of cap S_EFD == 20% of cap None
15k G_15k_Bind == 80% S_15k == 20% of cap None of cap
SATA G_SATA == 12% of S_SATA_Bind ==38% of B_SATA_Bind == 50% of cap cap cap
2014 EMC Proven Professional Knowledge Sharing 17
Note that one pool from each tier is designated by name as the binding pool. This means this is the default binding location for volumes on this tier.
Why go to the trouble to configure these additional pools?
Checking the total capacity usage of a tier becomes a simple matter of summing the pools If a disk group becomes busy, the pools provide a Unisphere® statistics construct that allows an administrator to quickly determine from which tier the pool workload is originating Because a VMAX is capable of draining TDATs from a pool, if the pool makeup needs to change later we have a mechanism with which to do this
FAST Policy Configuration One of the bywords in this storage design is consistency. When designing the FAST Policy, rather than using a simple 100/100/100 policy (or variation thereof) as is common on many arrays, we will base the FAST Policy for a tier on the underlying disk resources. In order to provide some leeway, we will add a few percentage points to each component of the ratio, with additional points on the lowest tier to provide FAST with the means to down-tier things such as un-reclaimed zeroes.
Gold/Plat Gold/Plat Silver Disk Silver Bronze Bronze Policy Ratios Policy Disk Ratios Policy Disk Ratios
% EFD 8.3% 12% 2.1% 5% 0%
% 15k 66.7% 70% 17% 22% 0% N/A
% SATA 25% 50% 80.9% 100% 100%
This policy design ensures that the higher-performance TDAT pools will fill relatively evenly. While this may result in lower initial performance for some volumes (those that might have monopolized the EFD tier), it provides a means for all volumes to maintain consistent performance over time, even as the system fills, presuming the system does not encounter some other bottleneck.
2014 EMC Proven Professional Knowledge Sharing 18
Front-End Adapter (FA) Allocation As with VMAX configurations generally, performance in a Service Tiered environment will be best if as many FAs as possible are in use. Depending on the IO density required for a storage group, more or fewer FAs may be required. As a rule of thumb:
0.1 IOPS/GB or less – Two FAs sufficient 2.0 IOPS/GB or less – Four FAs sufficient Greater than 2 IOPS/GB – 8 or greater FAs
It is also generally recommended that Port Groups supporting Pro Tip: Use non- overlapping Port Groups different applications and service levels be striped among the when presenting multiple FAs available in the array. While two FAs might be sufficient tiers to a host. for the entire Bronze workload of even a large array, in practice isolating the workload to these processors would both impact the other service levels (as those FAs might have unused processing power that could be used for Silver, Gold, etc.), but also will tend to run into other issues such as Channel Address limitations.
Adding or Removing FAs When Migrating Between Tiers Let us consider a Service Tiering environment in which Bronze volumes are masked to two FAs and Gold volumes are masked to eight. Suppose a host can see ten Bronze volumes (in one masking view which uses a child storage group) and ten Gold volumes (in a second masking view which uses another child storage group of the same parent). Now suppose that the application owner requests that a single one of the Bronze volumes be converted to Gold.
In order to do this non-disruptively, the Bronze volume in question must briefly reside in both masking views while the host rescans to see the newly added paths. For this reason, we have a specific rule for FA assignments to ensure this move can always be done non-disruptively:
When a host or cluster has visibility to multiple Service Tiers, each tier should be masked using unique, non-overlapping Port Groups
This ensures that the volume can exist in multiple masking views at the same time without the possibility of HLU conflicts.
2014 EMC Proven Professional Knowledge Sharing 19
Explicit Port Commitment Tracking
Pro Tip: Track IO If Host IO limits are applied to all workloads on a frame, this provides a Limit commitment to unique opportunity to implement rules-based load balancing across automate port ports. Rather than needing to look at performance and utilization metrics selection on a FA over time, the Host IO Limit commitment of all ports can be queried directly, and the Port Group with the lowest level of commitment selected when new workload is added.
Sample code to implement this algorithm is too long to include in an Appendix, so we have instead posted it at https://community.emc.com/people/DannoOfNashville/blog/2013/12/25/sidewndr .
Path Balancing When a Host IO Limit is applied to a Storage Group, it is distributed as a quota across all the ports in the associated Masking View. This means if a Storage Group with a limit of 1,000 IOPS is masked to four ports, each will attempt to service the volumes in the Storage Group at no faster than 250 IOPS. Pro Tip: Use SAN While Enginuity 5876.229.145 introduced some features (Director Path Load Balancing Down, Dynamic Rebalance) which should be strongly considered for best results for default enablement in a Service Tiered environment, it is still strongly recommended that proper path load balancing be implemented.
The most generally effective way to do this across a wide variety of host types is the use of EMC PowerPath® which uses an active load balancing algorithm to send IO to the lowest- latency path. PowerPath also has many advantages in troubleshooting instrumentation and mass management.
If not using PowerPath, the use of a Round-Robin style load balancing algorithm is recommended. If the number of IOs that will be sent down a given path is configurable, as in VMware, a setting of ‘1’ is recommended.
2014 EMC Proven Professional Knowledge Sharing 20
Migration Between Tiers If a volume requires migration between tiers in this solution, the workflow goes as follows:
1. Determine new target pools and port groups for subject volume. a. New port group should be non-overlapping. b. If host already has volumes from target tier, existing port group in use for that tier has preference, subject to port group commitment rules 2. Strip FAST Policy from subject storage group 3. If migrating a single volume from a multi-volume storage group: a. Add subject volume to target SG b. Create or Refresh Masking View to include target SG c. Rescan at host/cluster d. Remove volume from original SG/MV 4. Check free space on target pools versus source devices a. This is intended to avoid a migration causing a pool full condition 5. Start a symmigrate operation on the subject volume(s) a. If moving from one multi-tier Service Tier to another, use multiple ‘selective- source’ moves. This moves only the tracks in a given pool, giving FAST placement a head-start in the new location b. If moving to/from a single-tier Service Tier, use a standard symmigrate 6. Apply FAST Policies appropriate to Service Tier a. To the target SG b. To the original SG if volumes remain in it vSphere Configuration VMDK Limits Setting VMDK Limits vSphere provides a feature very similar to the VMAX’s Host IO Limits (VM.Disk.DiskLimitIOPerSecond) which operates on a similar algorithm. The primary difference is that it operates at the VMDK level, providing a much greater level of granularity. This parameter can be set using the path listed above, or in the GUI in the Resources tab of VM Settings.
2014 EMC Proven Professional Knowledge Sharing 21
Using VMDK Limits in Conjunction with VMAX Limits Rather than treating VMDK or VMAX IO limits as an either/or decision, a large environment will often have reasons to consider using both at the same time. Each feature can be used together in such a way as to take advantage of the best of both. Pro Tip: Use VMAX limits First, designate datastores by Service Tiers. This can be done for macro-tracking and in the datastore naming scheme. At the VMAX level, a group of control, VMDK limits for datastores assigned to a cluster can be placed in a child granular control Storage Group and be given a single Host IO Limit that covers the entire group. In our example environment, a cluster with three 1TB Gold datastores would have a child Storage Group with a limit of 12,288 IOPS (4 IOPS/GB * 1024GB * 3 Datastores).
Second, set the IO limit of each VMDK in the datastore based on the tier in which it resides. In most environments it will make sense to add an additional overage factor (going from 4 to 6 IOPS/GB for Gold) or a minimum granularity (ie, 50 IOPS per VMDK in Bronze, or roughly the performance of a 5400rpm hard disk). This can be scripted to apply in existing environments, but ultimately should be made part of the orchestration workflow.
It is important to note that kernel-level IO such as Storage vMotion is not restricted by this limit. The setting of a limit for an entire group of datastores (or datastore cluster) on the VMAX works out well here. This macro-level limit typically allows enough overhead to perform these operations without performance impact.
Setting the macro-limit on the VMAX also allows tracking of IO commitment, which is important to automating provisioning in a balanced fashion.
2014 EMC Proven Professional Knowledge Sharing 22
VMDK IO Limit Scalability The limit for each VMDK uses a lightweight algorithm enforced locally on each vSphere host. This has allowed it to be successfully deployed in environments with tens or hundreds of thousands of virtual machines.
Datastores: To Tier or Not to Tier? In some organizations the idea of dividing datastores by Service Tier may be a controversial topic, especially if the VMware administrators are accustomed to having a homogeneous storage environment and manually mixing high and low performance VMDKs to balance performance and queue limits. There are two primary arguments for tiering datastores that should be considered:
As the storage landscape evolves, it is a certainty that datastores will be placed on non- VMAX storage. This might be a flash-optimized array like XtremIO, a massively distributed non-array like ScaleIO, or some other up-and-coming technology that disrupts traditional expectations in a positive way for your organization. Ultimately, a management structure needs to be in place to deal with differentiated tiers gracefully. Manual management of IT environments is going the way of dial-up Internet. If a process cannot be automated, it must be replaced for an IT organization to remain competitive.
Troubleshooting VM Performance Issues Because we have set a macro-level VMAX limit that will typically be higher than the combined demand on all datastores in a cluster, this provides the VMware administrator some freedom in troubleshooting. If it is suspected (or observed) that a virtual machine is experiencing performance issues because it is reaching the IO limit, the administrator can temporarily increase or remove this limit. This can provide immediate relief until the VMDK can be Storage vMotioned to a datastore cluster in a higher Service Tier.
As with VMAX Host IO limits, be aware that issues arising from these limits may happen at smaller timescales than are visible in many reporting engines. The use of real-time monitoring or short-term, highly granular statistics from vCenter is recommended.
Use of Storage DRS and Datastore Clusters The use of Datastore Clusters can simplify administration of a large number of datastores. In the case of a Service Tiered environment, datastore clusters can be based on the tier and further
2014 EMC Proven Professional Knowledge Sharing 23 subdivided based on other LUN-level storage services offered. For instance, Bronze Datastore Clusters for both replicated and non-replicated volumes.
The use of this approach does not change the standard recommendations about Storage DRS in a FAST VP environment. While it is acceptable to configure Storage DRS to help with initial VMDK placement, as well as migrate VMDKs in order to meet free-space requirements, it is NOT recommended to use Storage DRS for performance. The movement by Storage DRS of a VMDK from between datastores based on latency thresholds invalidates FAST VP statistics, which makes both algorithms less effective.
Note that interoperability of SRDS and Site Recovery Manager was released in SRM 5.5. If you intend to use these features together, SRM 5.5 or higher and a compatible SRA is required.
Datastore Size One of the key factors driving datastore size is the ability of both hosts and arrays to drive IO to a single LUN. There can be many limiting factors here: every queue between the VM Guest and the physical disk drive is potentially a bottleneck. For this reason, administrators should strongly consider making datastore size inversely proportional to IO density. A 0.05 IOPS/GB datastore may perform acceptably at a size of tens of TB, while a 20 IOPS/GB datastore may see issues at just a few hundred GB.
Queue Depth Administrators may find that these higher performance (Gold and Platinum) datastores will find bottlenecks in the stack that weren’t previously taxed. One common place this occurs is with HBA queue depth. A higher than default setting such as 256 may be required to maximize performance of Platinum datastores. Refer to VMware KB1267 (http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&ext ernalId=1267) for more details.
2014 EMC Proven Professional Knowledge Sharing 24
Monitoring and Reporting Native Monitoring in Unisphere for VMAX Unisphere for VMAX provides a number of pool level metrics that can be useful in characterizing performance of a tier. In some cases, these may be metrics that were not often used before the configuration of multiple logical pools per disk tier.
Capturing Back-end IO Statistics for a Tier The Virtual Pool Tier and Thin Pool constructs maintain many of these statistics. This can be useful in gauging what portion of a physical disk tier’s IO comes from which Service Tier.
Capturing Front-End IO Statistics for a Tier The simplest way to capture front-end IO statistics for an entire tier is to use a non-masked Storage Group to aggregate this data. For example, you might maintain a storage group called GOLD_SG that includes all volumes with gold policy, which allows reporting on total IO to all Gold devices, data on front-end response time, cache hit ratio, and all other such Storage Group level metrics.
If you maintain a naming scheme for child storage groups that indicates the Service Tier they fall in (i.e. appending ‘_GOLD_CSG’ to those with Gold policies applied), it is a relatively simple matter to regularly parse the output of a symsg command to update the tiered storage groups.
Performance Troubleshooting If an application complains of slow performance and suspects the issue resides with storage, there is now a new set of tools at your disposal.
2014 EMC Proven Professional Knowledge Sharing 25
First and foremost, the fact that the Service Tiers have very explicit performance SLAs provides you with a quick check. If a Storage Group is servicing IO at or near the IO Density limit, this is a sign that the offending volumes need to be placed in a higher tier (and therefore are subject to a higher chargeback).
This solution will also make a certain category of problem more common: limits on short term bursts causing application issues. If an application has occasional bursts that last, for example, one second, and a newly applied IO limit extends that time to five seconds, many of the traditional performance metrics which operate on timescales of a few minutes will not look much different. However, to the application this may be very noticeable.
Fortunately, the Host IO Limit feature added some new parameters to Unisphere (and STP) to detect this scenario. At the Storage Group level you can now view IO Limit % Delayed IOs and Host IO Limit Delayed Time (ms). If both of these parameters are high, this can be a sign that IO bursts are bumping up against the configured limit.
Another traditionally available tool that should not be forgotten is Unisphere for VMAX’s Real Time Monitoring. If an application complains of impact in a specific scenario, you can use real- time monitoring to see IO statistics at a five second granularity.
Unified Reporting in EMC Storage Resource Management (SRM) EMC SRM allows the construction of custom reporting. In this section, we will discuss some sample reports that may be useful.
Simpler Aggregated Front End Statistics In an environment with only Unisphere for VMAX available, the use of non-masked storage groups was recommended to get good tier-level reporting. SRM provides some much more powerful tools for doing this sort of reporting.
In this case, maintaining a naming scheme for all SGs with limits applied is still recommended. SRM adds the ability to aggregate statistics on the fly based on this naming scheme.
For example, an SRM administrator might build a filter that would limit input data to the set of Storage Groups on an array that match the pattern ‘%GOLD_SG’. This would allow the administrator to report on any Storage Group level statistics on the entire tier in ways that might not otherwise be possible.
2014 EMC Proven Professional Knowledge Sharing 26
VMDK-Level IO Reporting and Mis-Tiering Detection It is a relatively simple matter to determine the IO density of a volume used by a physical host – the scripts in Appendix A handle this mapping with just a few pages of code. When working in a VMware environment, the use of datastores introduces a layer of abstraction that can make this mapping more difficult. To determine the appropriate tier for a VMDK and provide mass reporting, the ability to track and report on IO at the VMDK level and map this from datastore to symdev is required. This functionality was added to SRM in the 3.0 release – ensure that vCenter Statistics Collection is set to Level 3 or better to ensure that VMDK-level IO stats are available in vCenter outside of Real Time mode.
Whether building this report for VMDKs or for physical volumes, the configuration is similar.
For the object in question (whether a VMDK, symdev or possibly a similar construct):
Determine the field that defines the IO limit set In a Custom Report, create calculated fields for o IOPS – On some types of devices, like Storage Groups, SRM tracks ReadRequests and WriteRequests, and it may be necessary to perform a math.Spatial.Sum of these metrics o Configured IO Density – Divide set IO limit by most recent capacity o Observed IO Density – Divide peak IOPS by most recent capacity o Usage Ratio –Configured IO Density divided by Observed IO Density
From here, display the resulting data in a Table style report, sorted by IO Density Observation to Quota Ratio. A sample report is shown below sorts by the Usage Ratio to find the Storage Groups that could be moved to a lower tier without impact.
2014 EMC Proven Professional Knowledge Sharing 27
This sample report requires some customization of the default emc-vmax collector provided with SRM Suite 3.0.1, as the Host IO Limit parameter is not imported by default. To make these changes:
First, make a copy of /opt/APG/Collecting/XML-Collector/emc-vmax/xmlcollector- vmax-topo.xml . If any mistakes are made, this will allow easy rollback. Edit this file with your favorite text editor (vim is pre-installed on the SRM 3.0.1 vApp) Find the section which parses the output of vmax.sh symsg -output XML -sid (host) list -v . This should start around line 1057. To the property-set symProperties, add the following XML tag
Once you have done this, you will need to restart the VMAX collector and run the import- Properties-Default task. Both of these can be done from the Centralized Management console. After the next collection is complete, a new metric called ‘iolimit’ will be associated with VMAX Storage Groups.
The Usage Ratio can be used to create Best/Worst reports for administrative action. There are two thresholds to consider:
Usage Ratio <1.5: This means that the volume is experiencing peak IO near the configured limit, which means the application owner may wish to upgrade to a higher tier of service. Usage Ratio > 10: This means that the volume is experiencing peak IO much lower than the configured limit, and the application owner may wish to downgrade to a less costly tier of service. o 10 is chosen as the threshold here because our example Service Tiers are all approximately 10x apart. A different threshold may be necessary if your environment requires different service levels.
2014 EMC Proven Professional Knowledge Sharing 28
Tier Capacity Reporting It is possible to configure EMC SRM Suite to report on a Service Tier as a logical unit. The figure to the right shows a graph that sums (using the math.Aggregation function) the total and usable capacities of the three pools that make up the Gold Tier on a VMAX array. This sum can have trend lines applied in order to estimate when the tier will fill up.
If you have used the naming scheme described in the Pool Configuration section above, creating the SRM filter for such a report can be as simple as parttype==’StoragePool’ & part=’G_%’ (to select only the pools associated with the Gold Tier, for example).
Tier Performance Reporting Similarly, aggregate performance of Storage Groups associated with a pool can be reported. For example, filtering for parttype=’Storage Group’ & part=’%BRZ%’ and then performing a math.Aggregation on name==’ResponseTime’ with the aggregation parameters set to AVG will return a graph describing the overall latency of the Bronze Tier. This can be useful for determining if an overall SLA is being met.
Future Service Design Once a tool like SRM is in fully embedded, much of the complex legwork and scripting described in Appendix A becomes unnecessary. Rather, the monitoring and reporting tool can quickly aggregate months of data about performance and help determine the impact of a change in design. Imagine, for instance, being able to definitively say “If you move this application from Gold to Silver, it will only impact 1.5% of IO based on the last six months of history, which will reduce your chargeback by $5,000/month (and allow me to purchase a much lower cost array at tech refresh time)”
2014 EMC Proven Professional Knowledge Sharing 29
Automation and Orchestration Use of ViPR As of this writing, EMC ViPR is at release 1.0. As anyone with experience in IT can attest, a 1.0 release means there will be use cases that are currently unsupported. As such, we will discuss the best way to set up a Service Tiered environment using ViPR, as well as note the current limitations. As new releases of ViPR become available, this set of limitations provides a set of metrics which an organization wishing to deploy a common software defined storage middleware can use to evaluate these releases.
Defining vpools in ViPR The use of FAST policies with ratios tied to the underlying disk ratio serves double duty here. In order to maintain consistency in binding, only those pools designated as binding pools will be added to Service Tier vpools. The binding pool should fill to its limit (typically 80% when thin provisioning is used) at around the same time as the other pools.
If multiple tiered VMAXs are deployed, the matching binding pools on each can be combined into a single vpool.
Setting Host IO Limits ViPR currently does not have a native method of assigning a Host IO Limit to a VMAX Storage Group. The best way to do this currently is to capture the storage group name created by ViPR and assign the Host IO Limit.
Migration Between Tiers In ViPR 1.0, the only supported vpool migration (using the PUT /block/volumes/{id}/vpool method) is when those vpools reside on a VPLEX cluster. The ability to migrate between vpools residing on VMAX storage is required to implement a Service Tiered approach as described here.
Port Overlapping There is currently no elegant way in ViPR to avoid overlapping ports used by different tiers presented to the same hosts. This means that even if an administrator uses ViPR for initial provisioning and accepts that changes may orphan entries in the ViPR database, there is no way to guarantee that a Service Tier can be changed non-disruptively.
2014 EMC Proven Professional Knowledge Sharing 30
Acknowledgments Thanks to everyone who helped to work out this set of practices over the last few months, including Derek Miller, Lee Cooper, Michael Smith, and Roger Kaeser of HCA, Jonathan Miller and Patrick Carmichael of VMware, and Doug Pardue, John Adams, John Madden, Rick Brunner, and Chris Conniff of EMC.
2014 EMC Proven Professional Knowledge Sharing 31
Appendix A: Input Data Extraction Three applications will be needed to use these scripts as-is.
A PHP interpreter (http://www.php.net)
A Perl interpreter (http://strawberryperl.com/ or http://www.activestate.com/activeperl)
A MySQL compatible database server (https://mariadb.org/ or http://www.mysql.com/)
Sections that may require modification are highlighted in red.
Step 0: Create your database schema In using this example code, it is often helpful to create a database for each analysis you wish to keep.
CREATE DATABASE 1464;
CREATE TABLE `luns` ( `sgid` INT(10) NULL DEFAULT NULL, `symdev` TINYTEXT NULL, `capacitymb` INT(11) NULL DEFAULT NULL, `maxiops` FLOAT NULL DEFAULT NULL ) COLLATE='latin1_swedish_ci' ENGINE=InnoDB;
CREATE TABLE `storagegroups` ( `sgid` INT(10) NOT NULL AUTO_INCREMENT, `sgname` TEXT NOT NULL, PRIMARY KEY (`sgid`) ) COLLATE='latin1_swedish_ci' ENGINE=InnoDB;
Step 1: Parse Storage Group Mappings If using a fresh install of PHP, you may need to edit your php.ini file to point to the ‘mysqli’ extension. This can be found in the ‘extensions’ section of the file.
To run this script, modify the first few lines to point to your output file and database server, then run it with: php step1-parse-symsg.php
2014 EMC Proven Professional Knowledge Sharing 32
Once this is complete, run a test SELECT statement (such as ‘SELECT * FROM storagegroups’ and ‘SELECT * FROM luns’) to ensure that the data was imported correctly.
--Begin step1-parse-symsg.php--
# Preferred input is # symsg -sid xxxx list -v -output XML
$file = "symsg1464.xml"; $database = "vmax1464"; $mygroups = simplexml_load_file($file); $link = mysqli_connect("127.0.0.1","root","password",$database) or die("Error " . mysqli_error($link));
$lunsadded = 0; $lunlist=array(); $i = 0; foreach($mygroups as $group) { if($group->SG_Info->SG_group_info->count == 0) { if($group->SG_Info->Masking_views == 'Yes') # avoids RDF_SG and FAST_SG { # Non-cascaded storage group $name = $group->SG_Info->name; print "Found non-cascaded group ".$name."\n"; $insertquery = "INSERT INTO storagegroups SET sgname='$name'"; print "$insertquery\n"; $link->query($insertquery); $sginsertid = $link->insert_id;
$devs = getDevs($group); if($devs != 0) { #print "Includes devices "; foreach($devs as $device) { if(!in_array((string)$device,$lunlist)) { $insertquery = "INSERT INTO luns SET symdev='$device',sgid=$sginsertid";
2014 EMC Proven Professional Knowledge Sharing 33
print "$insertquery\n"; $link->query($insertquery); $lunlist[$lunsadded] = (string)$device; $lunsadded++; } } } # end if devs not zero } } else { foreach($group->SG_Info->SG_group_info->SG as $sg) { if($sg->Cascade_Status == "IsParent") # first pass we just make a list of parents { $parentnames[$i] =(string)($sg->name); $i++; } # end check for IsParent } # end foreach group_info as mysubgroup } # end if sg_group_info->count not equal to 0 } // end foreach mygrouups as group print "Initiating second pass \n"; foreach($mygroups as $group) { $thisname = (string)$group->SG_Info->name; if($group->SG_Info->SG_group_info->count != 0) { if(in_array($thisname,$parentnames)) { print "Found parent group $thisname\n"; $insertquery = "INSERT INTO storagegroups SET sgname='$thisname'"; print "$insertquery\n"; $link->query($insertquery); $sginsertid = $link->insert_id;
$devs = getDevs($group);
#print "Includes devices "; foreach($devs as $device)
2014 EMC Proven Professional Knowledge Sharing 34
{ if(!in_array((string)$device,$lunlist)) { $insertquery = "INSERT INTO luns SET symdev='$device',sgid=$sginsertid"; print "$insertquery\n"; $link->query($insertquery); $lunlist[$lunsadded] = (string)$device; $lunsadded++; } } } # end if name is in list of parents else { print "Found child group $thisname\n";
} # end else not in list of parents
} # end if count not zero } # end foreach mygroups as group function getDevs($group) { $i=0;
foreach($group->DEVS_List->Device as $device) { $devs[$i] = $device->dev_name; $i++; } # end foreach if(isset($devs)) { return $devs; } else { return 0; } } # end function getdevs ?>-- End step1-parse-symsg.php –
2014 EMC Proven Professional Knowledge Sharing 35
Step 2: Parse MiTrend Output If using a fresh Perl install, you may need to install the Spreadsheet::BasicRead and DBI modules. If using ActiveState Perl, this can be done with the Package Manager GUI. If using Strawberry Perl, both ppm and cpan are available.
The $file below should be the Storage Details output from a Mitrend analysis. Note that the column IDs in Mitrend output are not static over time. If you receive unexpected results, you may need to confirm that the column and sheet numbers are still valid for the most recent Mitrend output.
--Begin step2-parse-mitrend.pl-- use Spreadsheet::BasicRead; use DBI;
$file = "1464Details.xlsx"; $dsn="DBI:mysql:database=vmax1464;host=127.0.0.1;port=3306"; $dbh = DBI->connect($dsn, 'root', 'password');
# File is expected to be the "[Customer] Storage Details.xlsx" from a Mitrend report # Sheet LUNS is 6, zero-indexed # symdev is col 1 # Storage groups is col 2 but is useless # Capacity (MB) is col 5 # Max IOPS is col 8 # 95th percentile Response time is 10 # 95th percentile Read Size is 21
$ss = new Spreadsheet::BasicRead($file) || die "Couldn't open $file\n";
$ss->setCurrentSheetNum(6); # puts us on the LUNs sheet
$ss->getNextRow(); # let's go ahead and pop off the header row
$i=0; while(my $data = $ss->getNextRow()) { $symdev = $data->[1];
$capmb = $data->[5];
2014 EMC Proven Professional Knowledge Sharing 36
$maxiops = $data->[8]; $responsetime = int ($data->[10]); $readsize = int ($data->[21]); $selectquery = "SELECT maxiops FROM luns WHERE symdev='$symdev'"; $sth = $dbh->prepare($selectquery); $rv = $sth->execute(); $row = $sth->fetchrow_arrayref(); $oldmaxiops = $row->[0]; if($maxiops > $oldmaxiops) { $updatequery = "UPDATE luns SET capacitymb=$capmb,maxiops=$maxiops WHERE symdev='$symdev'"; print "$updatequery\n"; $dbh->do($updatequery); } else { print "Skipped lun $symdev\n"; } $i++;
} # end while print "Processed $i lines \n";
$dbh->disconnect(); --End step2-parse-mitrend.pl—
Step 3: Produce output file This script has more input parameters than the previous ones. You’ll want to define:
The IO Density limits you wish to be displayed Matching friendly names for those IO Density limits o Note that the last limit shown here, Out-of-Bounds (OOB) is intended to be effectively ‘infinity’ to capture everything above Platinum Set the filter if desired to process only a subset of the input data o Note that only the ‘NONE’ and ‘BY_SG_NAME’ options have been implemented in this example. ‘BY_DEVICE_FILE’ is provided as an example of another filter that a user might wish to implement. If using the ‘BY_SG_NAME’ filter, set the $sgnamefilter variable
2014 EMC Proven Professional Knowledge Sharing 37
o In the example below, we are filtering for only Storage Groups that include the substring ‘vms’, which in a particular environment captured all vSphere clusters to the exclusion of physical hosts Names for the two output text files.
While the first two steps need only be run once for a dataset, it is intended that this script be run multiple times with different IO Density limits and filters to allow the user to get an idea of the distribution of IO within the source dataset.
The output device file ($outputdevicefile) includes a list of each symdev and associated details. It is intended to be helpful in migration scenarios when an existing environment is being moved into an environment with tiered service levels.
The output skew file ($outputskewfile) is a list of IO density levels, sorted by the log10 of the IO density and rounded to the nearest 0.1. This is intended to allow macro-level design assistance, answering questions like “How large would my bronze tier be if I set its IO Density limit to 0.03?”
--Begin step3-query-skew.pl-- use DBI;
# Configuration parameters $dbhost = "127.0.0.1"; $dbuser = "root"; $dbpass = "password"; $database = "1464";
$limits[0] = 0.03; $limits[1] = 0.4; $limits[2] = 3.2; $limits[3] = 20; $limits[4] = 100000;
$tiername[0] = "Bronze"; $tiername[1] = "Silver"; $tiername[2] = "Gold"; $tiername[3] = "Platinum"; $tiername[4] = "OOB";
#$FILTER_TYPE = "NONE";
2014 EMC Proven Professional Knowledge Sharing 38
#$FILTER_TYPE = "BY_DEVICE_FILE"; #$FILTER_TYPE = "BY_SG_NAME";
$sgnamefilter = "LIKE '%vms%'"; $devicefilename = "devices.txt";
$outputdevicefile = "devicetiering.txt"; $outputskewfile = "skew.txt"; # end configuration parameters open($OUTSKEW, ">", $outputskewfile) or die "Cannot open $outputskewfile\n"; open($OUTDEVS, ">", $outputdevicefile) or die "Cannot open $outputdevicefile\n"; print $OUTDEVS "SGName, symdev, Max IOPS, Capacity (GB), IO Density, Tier\n"; print $OUTSKEW "IO Density, log10(IO Density), Capacity (GB), Max IOPS, Tier\n"; #$iodensity, $aggcapgb, $aggio,$tier);
$dsn="DBI:mysql:database=$database;host=$dbhost;port=3306"; $dbh = DBI->connect($dsn, $dbuser, $dbpass);
#SELECT S.sgname,L.symdev,L.capacitymb,L.maxiops FROM storagegroups S INNER JOIN luns L ON L.sgid=S.sgid WHERE S.sgname LIKE '%vms%';
$query = "SELECT S.sgname,L.symdev,L.capacitymb,L.maxiops FROM storagegroups S INNER JOIN luns L ON L.sgid=S.sgid"; if($FILTER_TYPE eq 'BY_SG_NAME') { $query .= " WHERE S.sgname ".$sgnamefilter } print "Executing $query\n";
$sth = $dbh->prepare($query); $sth->execute();
$dataref = $sth->fetchall_arrayref();
$i = 0; foreach(@{$dataref}) {
2014 EMC Proven Professional Knowledge Sharing 39
@data = @{$_}; #print "Checking row $i\n"; #print "Storage group is ".$data[0]."\n";; $maxiops = $data[3]; $capgb = $data[2] / 1024; if($capgb > 0) { $iodensity = $maxiops / $capgb; } else { $iodensity = 0; } if($iodensity > 0) { $log = log10($iodensity); } else { $log = -10; } #print "Log was $log\n"; my $logbucket = sprintf("%.1f", $log);
$logcapstore{$logbucket} += $capgb; $logiostore{$logbucket} += $maxiops;
$j=0; $tier = $tiername[0]; while($iodensity>$limits[$j]) { $tier = $tiername[$j+1]; $j++;
} # end for j
printf $OUTDEVS ("%s,%s,%.2f,%.2f,%.2f,%s\n",$data[0],$data[1],$maxiops,$capgb,$iodensity,$tier); } # end foreach $dataref
$tier = 0;
2014 EMC Proven Professional Knowledge Sharing 40 foreach $key (sort {$a <=> $b } (keys %logcapstore)) { $iodensity = 10**$key; $aggcapgb = $logcapstore{$key}; $aggio = $logiostore{$key};
$j=0; $tier = $tiername[0]; while($iodensity>$limits[$j]) { $tier = $tiername[$j+1]; $j++;
} # end for j
printf $OUTSKEW ("%.5f,%.2f,%.2f,%.2f,%s\n",$iodensity, $key, $aggcapgb, $aggio,$tier); }
$dbh->disconnect(); sub log10 { my $n = shift; return log($n)/log(10); } --End step3-query-skew.pl—
Step 4: Load the output data into Excel The output text files from Step 3 are comma separated. To open them in Excel, it is important to define fields with hex values (like Symmetrix device IDs) as text so they are not interpreted incorrectly.
2014 EMC Proven Professional Knowledge Sharing 41
This avoids Excel interpreting symdev IDs as numbers expressed in scientific notation, or erasing leading zeroes.
Glossary Enterprise Flash Disk (EFD) A term used to describe a Solid State Disk (SSD) which would be used in an enterprise storage array.
Initiator Group (IG) A logical collection of Host HBA WWNs used for Masking/Mapping
Logical Unit (LUN) A raw SCSI device defined in a storage array such as a VMAX and presented to a host. Often colloquially referred to as simply a volume.
Masking View Used for presenting VMAX LUNs to hosts out a given set of VMAX ports. This is the intersection of a Storage Group, Port Group, and Initiator Group
2014 EMC Proven Professional Knowledge Sharing 42
Port Group (PG) A logical collection of VMAX front-end ports used for Masking/Mapping
Service Level Objective (SLO) A metric which describes the service level a service catalog resource should meet. This might be availability (99.999% available), performance (at least 1000 IOPS at a better than 5ms average), or any number or combination of other measurable parameters.
SRDF Symmetrix Remote Data Facility, an array replication technology used on the VMAX and other Symmetrix-class arrays
Storage Group (SG) A logical collection of LUNs on a VMAX used for Masking/Mapping
STP Data Statistics data describing the performance of a VMAX system over time
Symmetrix Device (symdev) A VMAX LUN, or a subcomponent component of a VMAX LUN.
VMDK A file representing a virtual hard disk used by a VMware Virtual Guest.
VMFS The filesystem applied to a raw LUN by a VMware vSphere host. Files such as VMDKs reside here. A VMFS-formatted LUN is known as a datastore.
EMC believes the information in this publication is accurate as of its publication date. The information is subject to change without notice.
THE INFORMATION IN THIS PUBLICATION IS PROVIDED “AS IS.” EMC CORPORATION MAKES NO RESPRESENTATIONS OR WARRANTIES OF ANY KIND WITH RESPECT TO THE INFORMATION IN THIS PUBLICATION, AND SPECIFICALLY DISCLAIMS IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.
Use, copying, and distribution of any EMC software described in this publication requires an applicable software license.
2014 EMC Proven Professional Knowledge Sharing 43