Copyright

Information in this document, including URL and other website references, represents the current view of Commvault Systems, Inc. as of the date of publication and is subject to change without notice to you.

Descriptions or references to third party products, services or websites are provided only as a convenience to you and should not be considered an endorsement by Commvault. Commvault makes no representations or warranties, express or implied, as to any third party products, services or websites.

The names of actual companies and products mentioned herein may be the trademarks of their respective owners. Unless otherwise noted, the example companies, organizations, products, domain names, e-mail addresses, logos, people, places, and events depicted herein are fictitious.

Complying with all applicable copyright laws is the responsibility of the user. This document is intended for distribution to and use only by Commvault customers. Use or distribution of this document by any other persons is prohibited without the express written permission of Commvault. Without limiting the rights under copyright, no part of this document may be reproduced, stored in or introduced into a retrieval system, or transmitted in any form or by any means (electronic, mechanical, photocopying, recording, or otherwise), or for any purpose, without the express written permission of Commvault Systems, Inc.

Commvault may have patents, patent applications, trademarks, copyrights, or other intellectual property rights covering subject matter in this document. Except as expressly provided in any written license agreement from Commvault, this document does not give you any license to Commvault’s intellectual property.

COMMVAULT MAKES NO WARRANTIES OF ANY KIND, EXPRESS OR IMPLIED, AS TO THE INFORMATION CONTAINED IN THIS DOCUMENT.

©2018 Commvault Systems, Inc. All rights reserved. Commvault, Commvault and logo, the “C” hexagon logo, Commvault Systems, Solving Forward, SIM, Singular Information Management, Simpana, Commvault Galaxy, Unified Data Management, QiNetix, Quick Recovery, QR, CommNet, GridStor, Vault Tracker, InnerVault, QuickSnap, QSnap, Recovery Director, CommServe, CommCell, APSS, Commvault GO, Commvault HyperScale, IntelliSnap, ROMS, Commvault OnePass, CommVault Edge, Edge Drive, and CommValue, are trademarks or registered trademarks of Commvault Systems, Inc. All other third party brands, products, service names, trademarks, or registered service marks are the property of and used to identify the products or services of their respective owners. All specifications are subject to change without notice. Confidentiality

The descriptive materials and related information in the document contain information that is confidential and proprietary to Commvault. This information is submitted with the express understanding that it will be held in strict confidence and will not be disclosed, duplicated or used, in whole or in part, for any purpose other than evaluation purposes. All right, title and intellectual property rights in and to the document is owned by Commvault. No rights are granted to you other than a license to use the document for your personal use and information. You may not make a copy or derivative work of this document. You may not sell, resell, sublicense, rent, loan or lease the document to another party, transfer or assign your rights to use the document or otherwise exploit or use the Manual for any purpose other than for your personal use and reference. The document is provided "AS IS" without a warranty of any kind and the information provided herein is subject to change without notice.

©1999-2018 Commvault Systems, Inc. All rights reserved.

V11 SP11 Performance April 2018

ABOUT THIS DOCUMENT This document is intended for all audiences and is current as of the software version and service pack stated in the top left corner of the page. This document is updated every three to six months depending on feature changes to Commvault® software. The date of publish is within the document title, e.g. 170530 indicating a publish date of May 30, 2017, and the date appears in the top right-hand side of each page. New and updated sections are indicated in the revision history section with hyperlinks to each section and appear with a RED heading and darker text for easy identification. For updated versions of this document, contact us at: [email protected] Whether an employee, partner, or customer; we all want to work collectively to provide the best technical education material possible. If you have ideas to improve this document or corrections to existing content, please contact us as: [email protected]

AUTHORS

Frank Celauro, Irene Grimaldi, Carl Brault

Edited by: Madelyn Moalam

REVISION HISTORY

Link Date Contributing Notes Author May 30, 2017 Initial document release Unbuffered I/O for UNIX/Linux March 28, 2018 Carl Brault SP10-11 revision Network Equipment Optimization Recommended Filters Tape Chunk Size Graphic Updated Copyright HyperScale Storage

For comments, corrections, or recommendations for additional content, contact: [email protected]

Commvault® Education Services Page 3 of 36 V11 SP11 Performance April 2018

Contents About this Document ...... 3 Authors ...... 3 Revision History ...... 3 Performance Overview ...... 5 Performance Benchmarks ...... 5 Environment Considerations ...... 6 Commvault® Stream Management ...... 9 Commvault® Performance Tunables and Best Practices ...... 11 File System Backup ...... 11 Virtual Server Agent Backup ...... 14 Database Agents ...... 15 Microsoft Exchange Database Agent ...... 16 Network Settings ...... 17 Disk Storage ...... 19 Tape Storage ...... 23 Cloud ...... 26 Auxiliary Copy Performance ...... 27 Commvault Features Assessment Charts ...... 28 Storage Policy Configurations ...... 28 Subclient Configurations ...... 32 Deduplication Configurations ...... 33 SILO Storage Configurations ...... 35

Commvault® Education Services Page 4 of 36 V11 SP11 Performance April 2018

Performance Overview Commvault® software is a high-performance solution for protecting all data in any environment within defined protection windows. The software also provides many settings to improve performance. Before considering tuning Commvault software, it is important to understand capabilities and limitations of all hardware and software deployed within an environment.

There is no such thing as a static data center. Network infrastructures are constantly changing, new servers are added, mission critical business systems are moving to hybrid cloud, or public cloud infrastructures. Before considering Commvault tunables, it is first important to understand your environment including the capabilities and limitations of the infrastructure; specifically, the ability to transfer large amounts of data of production or backup networks.

When making modifications to an environment, changes that may positively impact one aspect of the environment can negatively affect another aspect. This is also true about Commvault settings. For example, enabling multiplexing when writing to tape drive can improve backup speeds. However, it may have a negative impact on restores if dissimilar data types are multiplexed to the same tape. Another example is using Commvault deduplication and setting a high number of data streams. Since client-side deduplication is being used, there will be a low impact to the network. But if the deduplication database needs to be sealed, the next set of backup operations may result in oversaturating the network while re-baselining blocks in storage.

Performance Benchmarks Benchmarks can be divided into two kinds, component and system. Component benchmarks measure the performance of specific parts of a process, such as the network, tape or hard disk drive, while system benchmarks typically measure the performance of the entire process end-to-end.

Establishing a benchmark focuses your performance tuning and quantifies the effects of your efforts. Building a benchmark is made up of the following 5 steps:

 Understand the process  Identify the resources involved  Minimize outside influence  Periodic test  Write it down Understand the process

You can’t document or improve something if you don’t know what’s going on. More importantly, you need to understand what phases a job goes through and how much each phase affects the overall outcome.

For example, a backup job over a network to a tape library takes two hours to complete. You think it should take a lot less and you spend time, effort, and money to improve your network and tape drives and parallel the movement of data. The job now takes 1.8 hours to complete. You gained a 10% improvement.

Looking at the job in more detail we find that the scan phase of the job is taking 1.5 hours and the rest is the actual data movement. Switching the scan method reduces the scan phase time to 12 minutes. The job now takes .4 hours. You gained a 78% improvement.

Knowing what phases a job goes through and how much each phase impacts the overall performance can help you focus your time, effort, and money on the real problems.

Identify the resources involved

Each hardware component is going to have a theoretical performance limit and a practical one. Attempting to get improvement beyond these limits without changing the resources involved is a waste of time. Consider using newer vs. older technologies, such as tape drives.

Commvault® Education Services Page 5 of 36 V11 SP11 Performance April 2018

Minimize outside influence

Large data movements are usually done during non-production hours for two reasons – one, they can degrade production work, and two, production work can degrade the movement of data. You want to minimize competition for resources to get a fair benchmark of what performance is achievable. In those cases, where competition cannot be eliminated, you must accept the impact to performance or invest in more resources.

Periodic Test

A single measurement is not a benchmark. Tape devices have burst speeds that are not sustainable over the long run. Networks have various degrees of bandwidth availability over a period of time. A single snapshot check of bandwidth will not give you a realistic expectation. Do periodic testing over the actual usage of a resource to determine its average performance. Try to level out the peaks and valleys - or at least try to identify what causes these variations.

Multiple measurements scattered over a day can also help in establishing if an unexpected external process is impacting the environment. For example, if you have a database server that is slowly backing up at night, but when you sample during the day, it is achieving expected performances, you can suspect an external process impacting the backup, such as a database administrator dumping the database and copying it to another server at the same time in this example.

Write it down

The hardest lessons are the ones you must learn twice. Once you’ve established your acceptable and/or expected performance levels for each resource and end-to-end, write them down and use them as the baseline for comparing future performance.

Environment Considerations Before modifying Commvault® software settings to improve performance, consider environmental capabilities and limitations. Ensure the environment is optimized to the best of your team’s abilities. Commvault software can move data at high rates of speed, but it will ultimately be limited by bottlenecks on servers and network devices.

TCP/IP TCP/IP is the most common network transmission protocol. Factors that can degrade TCP/IP performance are:

 Latency - Packet retransmissions over distance take longer and negatively impact overall throughput for a transmission path.  Concurrency - TCP/IP was intended to provide multiple users with a shared transmission media. For a single user, it is an extremely inefficient means to move data.  Line Quality - Transmission packet sizes are negotiated between sender/receiver based on line quality. A poor line connection can degrade a single link’s performance.  Duplex setting - Automatic detection of connection speed and duplex setting can result in a half-duplex connection. Full duplex is needed for best performance.  Switches - Each switch in the data path is a potential performance degrader if not properly configured.  Firewalls – Firewall is the first line of defense against hackers, malware, and viruses. There are hardware firewall appliances and software firewalls, such as operating system firewalls. Firewalls can have minor to moderate impacts on transfer performances.

Vendor documentation for most network interface cards (NIC), switches, and router provide useful information on optimization of their equipment.

Commvault® Education Services Page 6 of 36 V11 SP11 Performance April 2018

SCSI/RAID SCSI is the most common device protocol used and provides the highest direct connection speed. An individual SCSI drive’s speed is determined by spindle speed, access time, latency, and buffer. Overall SCSI throughput is also dependent on how many devices are on the controller and in what type of configuration. The limitation of SCSI is the distance between devices and the number of devices per controller.

 RAID arrays extend the single addressable capacity and random access performance of a set of disks. The fundamental difference between reading and writing under RAID is this: when you write data in a redundant environment, you must access every place where that data is stored; when you read the data back, you only need to read the minimum amount of data necessary to retrieve the actual data--the redundant information does not need to be accessed on a read. Basically – writes are slower than reads.

 RAID 0 (striping) or RAID 1 (mirror) or RAID 1+0 with narrow striping are the fastest configurations when it comes to sequential write performance. Wider striping is better for concurrent use. A RAID 5 configured array can have poor write performance. The tradeoff in slower write performance is redundancy should a disk fail.

Fine tuning a RAID controller for sequential read/write may be counterproductive to concurrent read/write. If backup/archive performance is an issue, a compromise must be arranged. iSCSI/Fibre Channel iSCSI or Fibre Channel protocol (FCP) is essentially serial SCSI with increased distance and device support. SCSI commands and data are assembled into packets and transmitted to devices where the SCSI command is assembled and executed. Both protocols are more efficient than TCP/IP. FCP has slightly better statistics than iSCSI for moving data. Performance tuning is usually setting the correct ‘Host Bus Adapter’ configuration (as recommended by the vendor for sequential I/O) or hardware mismatch. Best performance is achieved when the hardware involved is from the same vendor. Given that configuration and hardware is optimum, then for both iSCSI and FCP, performance is inhibited only by available server CPU resources.

Disk I/O Performing I/O to disks is a slow process because disks are physical devices that require time to move the heads to the correct position on the disk before reading or writing. This re-positioning of the head is exacerbated by having many files or having fragmented files. You can significantly improve read performance of the source data by de-fragmenting the data on a regular basis.

Anti-Virus Anti-viruses are intelligent software protecting a system against corrupted data by periodically scanning files systems and ensuring that every file accessed or opened by any processes running on the system is a legitimate file (and not a virus). You can easily imagine that when a backup runs and protects every system files, the anti-virus validation significantly decrease backup performances. It might also access and lock Commvault files, such as log files. It is recommended on all systems on which Commvault software is installed, to add exclusions to the anti-virus software for Commvault® software folders, so that when Commvault related processes are in action, they do not trigger the anti-virus validation process.

Commvault® Education Services Page 7 of 36 V11 SP11 Performance April 2018

Windows Recommended anti-virus exceptions

For V10 and earlier environments, any Content Store directory must be replaced by ‘Simpana.’ For V11 environments, any Content Store directory must be replaced by ‘ContentStore.’

CommServe, Software Installation Path client, and *:\Program Files\CommVault\ContentStore\** MediaAgent Updates Cache folder with CVPackages and CVUpdates subfolders on the CommServe Installation Paths C:\Program Files\CommVault\ContentStore\SoftwareCache DR backup set directories on the CommServe C:\Program Files\CommVault\CS_DR Job Results folder C:\Program Files\CommVault\ContentStore\iDataAgent\JobResults Index Directory folder C:\Program Files\CommVault\ContentStore\IndexCache Additional Magnetic libraries MediaAgent *:\**\CV_MAGNETIC\** Paths Deduplication databases Obtain the deduplication database location from the CommCell® console, from the Copy Properties dialog box of the primary copy, located in the Deduplication tab. SharePoint Temp folder path Agent C:\Users\Commvault Services account\AppData\Local\Temp

Content Indexing Exclude entire CI Engine install folders: and Search C:\Program Files\CommVault\ContentStore\CIServer C:\Program Files\CommVault\ContentStore\CVCIEngineSolr folder path C:\Program Files\CommVault\ContentStore\CVCIEngine\solr CI Index folder path C:\Program Files\CommVault\ContentStore\CVCIEngine\solr\CIIndex

UNIX, Linux and Macintosh Recommended anti-virus exceptions

CommServe, Software install directory client, and */opt/commvault/** MediaAgent Job Results directory Installation Paths /opt/commvault/iDataAgent/jobResults Index directory /opt/commvault/IndexCache Directory to extract installation binaries /tmp/.gxsetup

Additional Magnetic libraries MediaAgent */CV_MAGNETIC/** Paths Deduplication databases Obtain the deduplication database location from the CommCell® console, from the Copy Properties dialog box of the primary copy, located in the Deduplication tab.

Commvault® Education Services Page 8 of 36 V11 SP11 Performance April 2018

Commvault® Stream Management Data Streams are used to move data from source to destination. The source can be production data or Commvault protected data. A destination stream will always move to Commvault protected storage. Understanding the data stream concept will allow a CommCell® environment to be optimally configured to meet protection and recovery windows.

Stream settings are configured in various places within the CommCell® console including the storage policy, MediaAgent, subclient, and library. The system always uses the lowest setting. If a MediaAgent is configured to receive as many as 100 streams and one storage policy is writing through the MediaAgent and is configured to use 50 streams, then only 50 streams will be sent through the MediaAgent.

During a data protection job, streams originate at the source file or application that is being protected. One or more read operations is used to read the source data. The number of read operations is determined by the number of subclients and within each subclient, the number of data readers or data streams, depending on which agent is managing the data. Once the data is read from the source it is processed by the agent and then sent to the MediaAgent as job streams. The MediaAgent then processes the data, arranges the data into chunks and writes the data to storage as device streams. The data is written to storage based on the number of writers, for a disk library, or devices (tape drives) for a tape library.

Stream management high level overview

Commvault® Education Services Page 9 of 36 V11 SP11 Performance April 2018

Stream Settings Summary Table Features & Description Functionality

Subclients  Subclients are independent jobs, meaning each subclient will have one or more streams associated with each job.

Multi-stream  Most subclients can be multi-streamed. For subclients that do not support multiple subclients streams, multiple subclients are used to multi-stream data protection jobs.  Data readers are configured in the General tab of the subclient.  Data Streams are configured in the storage device tab for MS-SQL and Oracle subclients.

Non-Subclient  Agents such as the new Exchange Mailbox agent manage streams at the object based agents level. For Exchange, each mailbox is protected as a single stream.  The default subclient data readers setting is still used as the primary stream governor for the maximum number of concurrent objects that can be protected.

Job Streams  Job streams are active network streams moving from source (client or MediaAgent) to destination (MediaAgent).  The Job controller shows the total number of job streams currently in use in the bottom of the window and the job stream ‘high watermark’ for the CommCell environment.  Add the ‘Number of Readers in Use’ field in the job controller to view the number of streams being used for each active job. Device Streams  Configured in the Storage Policy properties.  Determines how many concurrent write operations will be performed to a library. This number should be set to equal the number of drives or writers in the library to maximize throughput.  Multiplexing is used to consolidate multiple job streams into single device streams.

Drives  For a removable media library writing data sequentially to devices, there will be one device stream per drive.

Writers  For a disk library where random read/write operations can be performed the number of writers should be set to allow the maximum throughput without creating bottlenecks in your network, MediaAgents, or disks.

Commvault® Education Services Page 10 of 36 V11 SP11 Performance April 2018

Commvault® Performance Tunables and Best Practices

File System Backup Consider the following key points when backing up the File System Agent:

 For backups on Windows operating systems, ensure source disks are defragmented.  Ensure all global and local filters are properly configured. Consult Commvault online documentation for recommended filters.  If source data is on multiple physical drives, increase the number of data readers to multi-stream protection jobs.  For larger high-speed disk, a maximum of two data readers can be set for an individual disk. Enable ‘Allow Multiple Data Readers within a Drive or Mount Point’ to allow multiple streams on a single disk.  If source data is on a RAID volume, create subclient(s) for the volume and increase the number of data readers to improve performance. Enable the ‘Allow Multiple Data Readers within a Drive or Mount Point’ option.  Consider using synthetic full, or better, DASH Full backups over traditional full backups.  Consider using the Commvault OnePass® agent to archive older ‘stale’ data.  For large volumes containing millions of objects use the File System Block-Level Backup.  Consider using multiple subclients and stagger backup operations over a weekly or even monthly time period.  For supported hardware, consider using the Commvault IntelliSnap® feature to snap and backup volumes using a proxy server.  Increase the ‘Application Read Size’ from the default of 64KB to 512KB.

Data Readers Disk I/O is the most costly, time-consuming portion of a data movement job. Using multiple data readers (also called data streams) can improve performance.

Conditions that can degrade performance for the File System Agent:

 In some configurations, such as concurrent backups that use embedded agents on multiple virtual machines (VMs) in a hypervisor environment, using multiple data readers for each backup might overwhelm the disk I/O and degrade performance. In this situation, using only one data reader for each VM might achieve the best performance.  Internal algorithms determine the maximum number of data readers that can read concurrently from a single physical drive. Too many data readers on a single physical drive can degrade performance.  Subclient content is divided between data readers based on physical drives. Thus, the first data reader reads from the first physical drive, the second data reader reads from the second physical drive, and so on. By default, only one data reader is allowed per physical drive, regardless of how many data readers are configured. Often, a data reader completes before the other data reader completes, which reduces the performance gain of using multiple data readers.

Commvault® Education Services Page 11 of 36 V11 SP11 Performance April 2018

Allow Multiple Readers within a Drive or Mount Point For the File System Agent, the Number of Data Readers value determines the number of parallel read operations from the data source.

The ‘Allow multiple data readers within a drive or mount point’ option helps you to use data readers more efficiently. For example, if you have subclient content that spans 4 physical drives, and you configure 8 data readers. Each physical drive gets 2 data readers. When one data reader completes its task, it assists another physical drive. This process continues until all data is read. This process maximizes the time that multiple data streams are moving data, which can improve performance.

Setting the number of readers and multiple readers within a drive or mount point

Application Read Size The application read size is the size of the application data that is read from the clients during backup jobs.

Values for the application read size must be in the power of 2; the minimum value is 64 KB, and the maximum value is 4,096KB (4MB).

Recommended values for Application Read Size

 NTFS volume 512KB  ReFS volume 2,048KB When the size of the application data that is read during backup jobs matches the source application’s internal buffer allocation, the overhead is minimized, and performance is improved. To achieve the optimal rate of data transfer during backup jobs, configure the application read size based on the source application's internal buffer allocation. You can increase the application read size to reduce the amount of data that is read from the given application. Reducing the amount of data that is read also reduces the number of I/O jobs that are performed against the application. As a result, overall backup performance might improve. However, backup memory usage might also increase, which might inadvertently consume additional resources from the application.

Commvault® Education Services Page 12 of 36 V11 SP11 Performance April 2018

Commvault recommends that you set the application read size at either the default value or at the cluster size that is directed by the application.

Microsoft NTFS uses a default cluster size (allocation unit) of 4KB by default. The 4KB cluster size was established when 2GB disks were considered large. Today, Microsoft recommends using a cluster size of 16KB or higher for NTFS volumes on servers. Commvault recommends that you use 64KB clusters, which matches the Microsoft ReFS default cluster size. With source data on volumes that have a 64KB cluster size, Commvault recommends using an application read size of at least 2,048KB for NTFS and ReFS.

For information about cluster sizes, see the Microsoft support article “Default cluster size for NTFS, FAT, and exFAT”.

Setting the Application Read Size

Commvault® Education Services Page 13 of 36 V11 SP11 Performance April 2018

Virtual Server Agent Backup General guidelines

 To optimize virtual environment data protection and recovery performance, contact Commvault Professional Services for the latest guidance and assistance.  Use the Commvault Virtual Server Agent (VSA) to protect most VMs. Specific I/O intensive VMs may require more advanced protection methods.  Use backup set or subclient VM filters to filter VMs that don’t require protection.  Use subclient VM rules to group priority VMs for protection. For example, use the power state rule to set infrequent schedules of VMs that are not powered on.  Maximize VM backup concurrency by increasing the ‘Data Readers’ option. Use caution as setting the readers option too high can cause performance degradation on backups and datastores or volumes hosting the VMs. As a general starting point, start with two VM backups per datastore or volume.  It is preferred to use physical VSA MediaAgent proxies versus virtual server MA proxies.  Ensure there are enough proxies to handle data movement load.  Use Commvault® software client-side deduplication and DASH Full backups.  For larger VMs, consider using the Commvault OnePass® feature to archive older ‘stale’ data.  Consider using multiple subclients and staggering schedules for when incremental and full or synthetic (DASH) full backups run.  Ensure Change Block Tracking (CBT) is enabled for all virtual machines, when applicable.

VMware specific guidelines

 Ensure VSA proxies can access storage using the preferred transport mode. SAN transport and HotAdd will fall back to NBD mode if they cannot access VMs from the SAN or DataStore. When protecting applications in a virtual environment:

 Using the VSA to protect applications without the Application Aware feature or agents installed within the VM may result in crash consistent backups.  For low to medium I/O applications, use the Application Aware feature. Check the Commvault Online Documentation for a list of applications supported by the VSA Application Aware feature.  For I/O intensive applications, it is still preferred to use application agents installed in the VMs. Commvault IntelliSnap® for VSA:

 Use IntelliSnap for VSA to protect I/O intensive VMs.  Define subclients by datastore affinity. When hardware snaps are performed the entire datastores is snapped regardless of whether the VM is being backed up.  For smaller Exchange or MS-SQL databases (less than 500GB), application consistent snapshots can be performed using the IntelliSnap feature and VSA.

Commvault® Education Services Page 14 of 36 V11 SP11 Performance April 2018

Database Agents General Guidelines

 For large databases that are being dumped by application administrators, consider using Commvault database agents to provide multi-streamed backup and restores.  When using Commvault database agents for instances with multiple databases, consider creating multiple subclients to manage databases.  For large databases, consider increasing the number of data streams for backing up database. For multi-streamed subclient backups of SQL and Sybase databases, the streams should not be multiplexed. During auxiliary copy operations to tape if the streams are combined to a tape, they must be pre-staged to a secondary disk target before they can be restored.  For MS-SQL databases using file/folder groups, separate subclients can be configured to manage databases and file/folder groups.

Database Agent Streams Disk I/O is the most costly, time-consuming portion of a data movement operation. Using parallel data readers (also called data streams) can improve performance. For databases, the Number of Data Readers value determines the number of parallel read operations that are requested from the database application.

Before you modify the number of data readers, Commvault recommends recording baseline throughput performance using the default settings, which are the recommended settings. You can then modify the number of data readers until you achieve the fastest throughput performance.

SQL data streams configuration

Commvault® Education Services Page 15 of 36 V11 SP11 Performance April 2018

Microsoft Exchange Database Agent

Application Read Size The performance of both regular backup operations and IntelliSnap backup operations of an Exchange Database can benefit greatly from an application read size of 4MB (4,096 KB). The default value is 64KB.

For most Data Availability Group (DAG) environments, backup operations are performed on the passive node, and memory usage for the application read size is not a concern. If production performance problems occur, then you can decrease the application read size.

Multi-streamed Exchange Database Backups Multi-streamed backups of Exchange database reduce backup time by allocating streams on a per database level. The maximum number of streams that is used by a backup is determined by the number of databases in the Exchange environment. If a subclient’s content contains four databases, then four streams could be used – each stream protecting one database.

In a DAG environment, the stream allocation is based on the number of nodes. When the job starts, the stream logic automatically assigns one stream to each node. If there are additional streams remaining, they are allocated based on which node has the most databases. The stream allocation process continues in order of Exchange servers in the DAG environment containing the most databases to fewest in a prioritized round-robin method until all streams are allocated.

Configure Multi-Streamed Exchange Database Backups

Commvault® Education Services Page 16 of 36 V11 SP11 Performance April 2018

Network Settings

Pipeline Buffers By default, Commvault software establishes 30 Data Pipeline buffers for each data movement connection. You can increase the data transfer throughput from the client by increasing or even decreasing the number of Data Pipeline buffers. The number of the Data Pipeline buffers depends largely on the transport medium.

To set the number of pipeline buffers, use the ‘nNumPipelineBuffers’ additional setting.

Although the maximum value for ‘nNumPipelineBuffers’ is 1,024, if you use a value that is greater than 300, you should consult with Commvault Support. When you increase the number of Data Pipeline buffers, the client or MediaAgent consumes more shared memory. When available memory is low, this consumption of shared memory might degrade the server performance for other operations.

Recommended values for nNumPipelineBuffers:

 Internet - 30 buffers  100BASE - 30 buffers  1000BASE - 120 buffers

Add Pipeline buffers additional setting

Commvault® Education Services Page 17 of 36 V11 SP11 Performance April 2018

Network Agents Network agents are threads or processes that transfer data to and from the network transport layer. Each network agent spends half its time reading and half its time writing. For higher speed networks, having multiple networks agents can improve performance.

Default values and valid values for the number of network agents:

 Windows default – 2. Valid options 1 – 4  Unix default – 1. Valid options 1 – 2 Network Agent configuration for a Windows subclient

Commvault® Education Services Page 18 of 36 V11 SP11 Performance April 2018

Disk Storage

Chunk Size Chunk sizes define the size of data chunks that are written to media and is also a checkpoint in a job. The default size for disk is 4GB. The default size for tape is 8GB for indexed based operations or 16GB for non-indexed database backups. The data path ‘Chunk Size’ setting can override the default settings. A higher chunk size results in a more efficient data movement process. In highly reliable networks, increasing chunk size can improve performance. However, for unreliable networks, any failed chunks must be rewritten, so a larger chunk size could have a negative effect on performance.

Chunk size recommendation for disk storage

Storage media Job type Default Recommended chunk size chunk size Disk All data protection jobs 4 GB 512 MB – 8 GB Direct-attached NDMP All data protection jobs 8 GB N / A

Commvault HyperScale® All data protection jobs 8 GB N / A scale out storage

Chunk size settings for a disk data path

Commvault® Education Services Page 19 of 36 V11 SP11 Performance April 2018

Chunk size configuration for MediaAgents Use the ‘DMMBCHUNKSIZE’ additional setting to control the chunk size of the data write jobs that go to the MediaAgent on which the additional setting is created.

The chunk size that you specify in the additional setting overrides the values that you specify in the chunk size that you specify for the CommCell® in the Media Management configuration.

Configuring MediaAgent chunk size

Commvault® Education Services Page 20 of 36 V11 SP11 Performance April 2018

Block Size MediaAgents can write to media that is formatted with different block allocation sizes or file allocation sizes if the MediaAgent operating system supports those sizes. Using a larger block size for disk library volumes can reduce overhead and thus increase the speed of write operations to media.

Linux ext3 and Microsoft NTFS use a default block (allocation unit) of 4KB. The 4KB block size was established when 2GB disks were considered large. Today, Microsoft recommends using at least a 16KB block size or higher for NTFS volumes. Commvault recommends that you use 64KB, which matches the Microsoft default value for the ReFS block size.

You can increase the Linux ext3 block size only on an Itanium system. For other file systems, consult your OS vendor documentation for your file system’s available block sizes.

Block size settings for a disk data path

Commvault® Education Services Page 21 of 36 V11 SP11 Performance April 2018

Unbuffered I/O for Windows® MediaAgent If the source copy is on disk and is managed by a Windows MediaAgent, then enable the Use Unbuffered I/O option for each mount path. Using unbuffered I/O can significantly improve performance.

To increase the speed of jobs that access the mount path, you can configure the MediaAgent to bypass the Microsoft Windows file system buffering.

You can make this configuration for Windows MediaAgents and for disks that are mounted directly (not for UNC paths).

Unbuffered I/O configuration for Windows MediaAgent

Unbuffered I/O for Unix/Linux MediaAgent A similar option is available for UNIX/Linux based MediaAgent, however, it must be enforced at the operating system level and not through the Commvault® software GUI. It can be achieved using two methods:

 Method one – Use the GFS tool provided by most UNIX/Linux based OS. This tool sets a direct I/O flag to a directory and all its current subdirectories and files. Once enabled, any new directory or files created will also inherit the direct I/O attribute. It can be turned on (using the setflag parameter) or off (clearflag) as desired.

 Method two – Use the Unbuffered I/O configuration for Linux MediaAgent:  Gfs_tool setflag inherit_directio MyDirectory Mount the NFS filesystem using the force direct I/O flag (forcedirectio). For as long as the filesystem is mounted, it will bypass the operating system buffer.

For more information on the GFS tool or the mount direct I/O option, refer to your operating system vendor’s documentation.

Commvault® Education Services Page 22 of 36 V11 SP11 Performance April 2018

Tape Storage

Chunk Size A chunk is the unit of data that the MediaAgent software uses to store data on media. For sequential access media, a chunk is defined as data between two file markers. By default, the chunk size is configured for optimal throughput to the storage media.

Job type Default chunk size Recommended chunk size

Granular (index based) job 8 GB 8 – 32 GB

Database (non-indexed) job 16 GB 8 – 32 GB

Chunk Size for tape libraries can be modified on the data path for a specific tape library, or globally, using the Media Management applet. Global chunk size settings are configured per agent type.

Chunk size settings for a tape data path

Commvault® Education Services Page 23 of 36 V11 SP11 Performance April 2018

Global chunk size settings for tape media

Block Size Before changing tape block size, ensure that the following criteria are satisfied:

 Block size is supported by the MediaAgent OS, Host Bus Adapter (HBA), and the tape device.  All the MediaAgents that are associated with a storage policy support the block size that is configured on that storage policy. Consider the support and the compatibility of MediaAgent platforms at any disaster recovery site.  If you use different MediaAgents for backup operations and restore operations, and if the backup MediaAgent has a higher block size, then ensure that the restore MediaAgent can read data that is written with a higher block size. Many streaming tape drives perform a read-after-write check. If the drive detects a bad block, then the drive puts a discard token after the block, and repeats the entire buffer write. If the drive detects a discard token, then the read cycle has corresponding logic to replace the bad block with the replacement block.

All tapes will have media defects. If you write 1,024KB blocks instead of 256KB blocks, then the chance of any block spanning a media defect are increased by a factor of 4. Because of the larger block size, the rewrite time is 4 times as long as well.

Increasing block size can improve the performance of writing to tape by minimizing the overhead associated with accessing and recording each block. If you select the data path’s Use Media Type Setting option, then the data path’s default block size for tape is 64KB. Refer to the Commvault Online Documentation: Use Media Type Setting section for more information.

Important notes on configuring tape block size:

 Use caution when you select large block sizes. Large block sizes can vastly increase error rates and retries.  Block size applies only to tape media in direct-attached libraries.  Changes to the block size settings take effect when the next spare tape media is used.  Ensure hardware at data center and other location, including DR sites support higher block sizes.

Commvault® Education Services Page 24 of 36 V11 SP11 Performance April 2018

Block size settings for a tape data path

Commvault® Education Services Page 25 of 36 V11 SP11 Performance April 2018

Cloud

Deduplication Block Size and Async IO Commvault recommends that you always use deduplication for efficiency and performance when writing to cloud libraries. The default deduplication block size is 128KB, but for the fastest performance to cloud libraries, Commvault recommends using a deduplication block size of 512KB.

Commvault recommends using the ‘SILookAheadAsyncIOBlockSizeKB’ additional setting to set the block size that is used by Async IO in the look-ahead reader. The recommended value is 2 times the deduplication block size.

The default value of ‘SILookAheadAsyncIOBlockSizeKB’ is 128KB for cloud libraries. For the recommended deduplication block size of 512KB, set the AsyncIO block size to 1,024KB.

Configure the ‘SILookAheadAsyncIOBlockSizeKB’ additional setting on all source data mover MediaAgents that are associated with the source storage policy copy. For instructions about adding additional settings from the CommCell® console, refer to the Commvault Online Documentation: Add or Modify an Additional Setting section for more information.

Configuring the Async IO look ahead reader block size for cloud libraries

Commvault® Education Services Page 26 of 36 V11 SP11 Performance April 2018

Auxiliary Copy Performance If the network utilization during a DASH copy job or a backup job with deduplication is low, then apply all the following settings on all source data mover MediaAgents that are associated with the source storage policy copy. For auxiliary copy operations, these settings are enabled by default.

For instructions about adding additional settings from the CommCell® console, refer to the Commvault Online Documentation: Add or Modify an Additional Setting section for more information.

Additional Settings for Auxiliary Copy performance

Additional Setting Description Values DataMoverUseLookAh Enables the Default value: 1 (enabled) eadLinkReader reading of multiple data signatures from the deduplication database SignaturePerBatch Ensures that the Modify this value only if there is a large latency DDB signature between the client and the deduplication look-ups are MediaAgent. batched with  Default value: 1 multiple signatures per look-up  Maximum value: 32  Recommended values: 16 and 32. If the latency is between 100 and 200 ms, then use 16. If the latency is more than 200 ms, use 32. DataMoverLookAhead Performs more  Default value: 16 LinkReaderSlots lookups on the destination  Valid values: 16, 32, 64, 128, and 256 deduplication  Recommendation: Set the look-ahead slot database for value to as high as the MediaAgent deduplication memory and concurrent operations allow. block signatures Each look-ahead slot uses a memory size of one block of deduplicated data. So, with a default deduplication block size of 128KB and 16 look- ahead slots, each reader stream in an auxiliary copy operation uses 2MB of memory. Setting lookahead slots to 128 uses 16MB of memory for each reader stream. So, if 200 auxiliary copy streams run on the MediaAgent, then the total memory overheard for the lookahead slots only is 3.2GB.

Commvault® Education Services Page 27 of 36 V11 SP11 Performance April 2018

Commvault Features Assessment Charts Commvault® software provides many features and settings that can improve data protection and recovery performance. It is important to note that, in some cases, these features or settings can have a positive impact in certain aspects, but negative impacts in other aspect.

The following section provides several charts that compare and contrast the pros and cons of features and settings within Commvault software:

 Each feature explains potential positive and negative effects on protection windows, media management, and recovery objectives.  Some features may not have any affect or minimal effect so no information is listed.  Each analysis is rated as ‘May have’, ‘Will have’, or ‘Significant’ effect. An explanation is also provided explaining why the effect may occur and which setting causes the effect to take place.  Additional considerations are listed for some features.

The following charts highlights the positive and negative effects of various Commvault protection methods and configuration options. Charts and explanations assume a solid level of competency with the Commvault Product suite.

Storage Policy Configurations

Feature / Option Protection Windows Media Recovery Objectives Other / (Primary / Secondary) Management Considerations Implementation Methods

Device Streams May have positive effect Library writers or by running more jobs drives should equal concurrently. number of storage policy device streams. May have negative effect When using if infrastructure cannot deduplication, ensure handle higher data loads. Media management ‘Maximum number of Will have positive effect parallel data transfer when used in conjunction operations for with client side deduplication deduplication. database’ is adjusted accordingly.

Erase Data May have a negative effect on restore as Media Explorer, Catalog or restore by job features cannot be used on media written by an erase data enabled storage policy.

Incremental May have a positive effect May have a positive effect on Storage Policy on incremental or log restore performance by restoring inc / backup performance by logs from fast disk storage. writing to a separate disk library.

Commvault® Education Services Page 28 of 36 V11 SP11 Performance April 2018

Primary Copy Configurations

Option / Protection Windows Media Management Recovery Objectives Other Value (Primary / Secondary) Considerations

Device May have positive effect Library writers or Streams by running more jobs drives should equal concurrently. number of storage policy device May have negative effect streams. if infrastructure cannot When using handle higher data loads. deduplication, ensure Media Will have positive effect management when used in conjunction ‘Maximum number with client side of parallel data deduplication. transfer operations for deduplication database’ is adjusted accordingly.

Multiplexing Will have positive effect May have negative effect on on backups when protecting RTO if streams are not required multiple slower job streams for restore and they are to a single device stream. multiplexed to same tape with streams that are required for restore.

Retention May have negative effect on storage if retention is improperly configured.

Spool Copy Will have positive effect when first spooling to disk then writing to tape when protecting large amounts of small files.

Extended May have negative effect on Retention tape consumption if jobs with Rules extended retention are mixed on tape containing jobs with no extended retention.

Will have a positive effect on retention when grandfather / father / son tape rotation is required.

Significant negative effect if using deduplication. A significant increase for both the disk library size and the deduplication database size that could potentially lead to a deduplication database seal and restart.

Commvault® Education Services Page 29 of 36 V11 SP11 Performance April 2018

Option / Protection Windows (Primary / Media Recovery Objectives Other Value Secondary) Management Considerations

GridStor™ Will have a positive effect on Will have a negative effect on RTO Round Robin primary backup by load balancing when using deduplication writing to across multiple MediaAgent. shared SAN library.

GridStor™ Will have a positive effect on Failover primary backup by providing a failover path to different MediaAgent, drive pool, scratch pool or library.

Data path chunk Will have a positive effect for size indexed jobs by increasing chunk size.

May have a positive effect for indexed jobs over unreliable network (WAN) by decreasing chunk size.

Data path block May have a positive effect on Significant negative effect on size backup performance by increasing restore if hardware does not support block size. higher block size including NIC, HBA, library, drive and MA OS.

Hardware May have negative effect on backup Provides high May have negative effect on restore encryption performance by additional level of performance. (LTO4 - LTO7) processing to encrypt data. security.

Commvault® Education Services Page 30 of 36 V11 SP11 Performance April 2018

Secondary Copies Configurations

Feature / Protection Windows Media Recovery Objectives Other Option (Primary / Management Considerations Secondary)

Inline copy May have positive effect by running primary and secondary jobs concurrently.

May have negative effect since jobs are streamed and will only run as fast as slowest device.

Combine to May have a negative Will have a May have a negative effect on RTO if streams streams effect if configured positive effect for restore are mixed with streams not required for using tape improperly. on media usage restore. media by combining streams to Will have negative effect on RTO for multi- fewer tape stream backups as opposed to placing each stream media. on separate tape. Note this only applies if using restore by job.

Will have negative effect on RTO if different data sets required for restore are on same tape. Only one data set can be restored at a time.

Multiplexing May have a positive with effect if source of combine to auxiliary copy is disk streams with multiple mount paths.

Software May have a negative encryption effect on performance depending on MediaAgent resources and size of data in secondary copy.

Using Significant Significant negative effect on RTO. Less restore Global positive effect jobs can run concurrently as data in stored on Secondary on tape usage same tapes, as opposed to different storage Copy since many policies tape copies isolated on their own media. storage policy secondary copies are concatenated on the same set of media.

Commvault® Education Services Page 31 of 36 V11 SP11 Performance April 2018

Subclient Configurations

Feature / Option Protection Windows Media Management Recovery Objectives Other (Primary / Considerations Secondary)

Custom Subclients Will have positive Will have positive effect by providing effect by allowing greater scheduling specific content to flexibility. be retained

separately. Will have positive effect by running May have multiple streams negative effect for concurrently. secondary copies if stream management settings are improperly configured.

Data readers Will have positive effect by running multiple streams concurrently.

Multiple readers in Will have positive drive or mount point effect by running multiple streams concurrently on RAID volumes

Data Streams for Will have a positive May have a Will have a positive effect on RTO from SQL, DB2 and effect on performance negative effect for tape with streams on separate tapes. Sybase by multi-streaming tape copy since each backup jobs. stream must go to Significant negative effect if multi- separate tape. stream backup was combined to tape. Tape copy requires pre-staging to disk prior to recovery.

Filters Will have a positive Will have a positive effect on restores effect on backup and since unwanted data was not backed up. auxiliary performance by eliminating May have a negative effect on unwanted data from restorability if filters are improperly jobs. configured and filter important data.

UNC path for file Will have a negative system subclient effect on primary contents backup. Path will be from UNC share to the Client and then to MediaAgent.

Commvault® Education Services Page 32 of 36 V11 SP11 Performance April 2018

Deduplication Configurations

Feature / Protection Windows Media Management Recovery Objectives Other Option / (Primary / Secondary) Considerations Implementation Methods

Isolating data in Will have positive effect May have negative effect May have positive effect Best practice is to separate policies to improve performance with marginal decrease in on restore performance by isolate data types to based on data when using multiple dedupe efficiency in grouping data blocks different policies type and block MediaAgent using primary disk storage. relative to data type being providing greater size dedicated DDB for each protected in their own scalability. policy. May have negative effect deduplication store folders. for secondary copies if May have negative using tape media. impact if more than two DDB loaded on single MediaAgent.

Client side Significant positive Significant positive May have negative RTO For WAN backups using deduplication effect after the first full by effect reducing size of data by slower disk read access client side sending only change data on disk. (due to fragmentation of deduplication, enable blocks to MediaAgent. data blocks on disk). the client side disk Will have positive effect cache. Will have positive effect on retention by making on secondary copies if more efficient use of disk using DASH Copy. media.

Block size Larger block size may May have negative effect Follow Commvault best have positive effect by if using small block size practices when reducing size of dedupe <128k (due to determining block size database making hash fragmentation of data settings. comparisons faster. blocks on disk).

Compression May have negative May have positive effect Will have positive effect Compression takes effect due to increase CPU since compression adds on RTO since data is place before hash so usage on client. considerable space savings decompressed on client. duplicate blocks will on deduplication depending generate same hash. Will have positive effect on data type. requiring less bandwidth If application for backup. compresses data, disable compression in dedupe SP.

Feature / Protection Windows Media Management Recovery Objectives Other Option / (Primary / Secondary) Considerations Implementation Methods

Do not Will have negative effect Temporary negative effect This option is for block deduplicate at point specified as all as duplicate block will be integrity by periodically against objects duplicate blocks older than written to dedupe store. (based on days set) older than specified number will be rewrite blocks to disk. backed up again.

Commvault® Education Services Page 33 of 36 V11 SP11 Performance April 2018

Seal deduplication Will have negative Temporary negative effect In a well-designed store effect at point store seals until the old store ages and environment sealing as all blocks will be written pruned. the store should not be to disk. necessary unless it meets specific protection needs.

Sealing the store may become necessary if dedupe database grows too large or if Commvault best practices are not followed.

Global Will have positive effect Will have positive effect May have negative effect Global deduplication deduplication on performance by on disk consumption by on performance that may policies are intended to policy deduplicating against deduplicating blocks from degrade by slower disk consolidate remote site blocks from multiple multiple storage policies read access (due to backups to a central primary copies into single into single store. fragmentation of data disk location. store. blocks on disk). May have negative effect This is not intended to May have significant since each individual SP centralize data for all negative effect on will still have to create policies in an scalability and individual secondary copies environment. performance if not requiring dedicated tape configured and for each policy copy. implemented correctly. (See other considerations).

DASH Full Significant positive effect on full backups after the first full is performed.

DASH Copy Significant positive Will have positive effect effect on off-site copies to on off-site RPO since data secondary disk target. can be copied off-site more frequently versus sending tapes off-site.

Commvault® Education Services Page 34 of 36 V11 SP11 Performance April 2018

SILO Storage Configurations

Feature / Protection Media Management Recovery Objectives Other Option / Windows (Primary Considerations Implementation / Secondary) Methods

Using SILO Will have positive Significant positive effect by May have negative impact on effect on writing dedupe data to tape RTO since volume folders must performance since without rehydration. be pre-staged to disk before data data does not can be recovered. require rehydration. Will have positive effect disk consumption by pruning old May have significant volume folders from disk. negative effect if large amounts of data need to be recovered.

Will have positive effect by copying all dedupe data to SILO it can improve RPO with more recovery points available as opposed to periodically performing full auxiliary copies.

SILO & global Significantly reduce tape May have negative impact on deduplication consumption if multiple policies are restore since volume folders policy writing to a global dedupe. must be pre-staged to disk before data can be recovered. Using SILO in place of traditional secondary copies will allow data May have negative impact on from multiple storage policies to be RTO. If large amounts of data stored on fewer tapes. Traditional need to be recovered, RTO aux copy would require separate performance can be significantly tapes for each secondary copy. degraded.

Significant positive impact on RPO. By copying all dedupe data to tape, more recovery points will be available as opposed to periodically performing full auxiliary copies.

Number of Silos May have a negative impact on Significant positive effect on to be kept in storage. Will Increase disk RTO. Improve RTO if volumes cache consumption by keeping SILO data required for restore are in cache. in disk cache.

Commvault® Education Services Page 35 of 36 V11 SP11 Performance April 2018

COMMVAULT.COM | 888.746.3849 | EA.COMMVAULT.COM

©2017 COMMVAULT SYSTEMS, INC. ALL RIGHTS RESERVED.