OK, As Mentioned in the Previous Post, I Didn T Really Get Around to Thoroughly Testing

The unofficial CommVault tuning guide

First; the challenge: We have a pretty large (at the moment, 2 TB) Oracle database on a HP BL685c G6 blade running Redhat Enterprise Linux 5.5 (64 bit) with Oracle 10.2.0.5.0. This is a pretty beefy machine. It uses Fibre Channel and multipathing to address it’s primary storage. It has 8 active paths to it’s storage, which multipath groups into 2 path groups. The client is therefore using 4 paths active-active to address it’s storage. The client has it’s own tape drives on the ESL 712e.

Second; a couple of warnings: - Please document all settings as they are currently set. A audit report will NOT display all settings changed. - Always keep in mind the limits of your environment. Keep performance stats handy for both your network performance (if doing LAN- based backups) and / or your primary storage. - Once you overcome one bottleneck, you’re sure to run into the next one. - Changes on one client CAN and will impact other clients using the same shared resources (such as your primary storage). - Some of the settings in this document are guess work. The settings have not been thoroughly tested. - The author is not affiliated with CommVault Systems Inc. or any of its partners - Settings in this document may not be supported by CommVault support. Please, always rollback your changes, and see if you can reproduce the error you’re getting. - Settings in this document are implemented at your own risk. See the copyright disclaimer at the end of this document. - All my settings are based on the “LAN-Free” method of backup. In this manner, you install both a client and a MediaAgent on the same machine, and setting the “Data Path Configuration” to “Use preferred data path”. - Always, always use common sense.

And now, for the actual tuning.

In all things regarding tweaking, there are 3 things to consider: - Read speed for the client (from primary storage) - Buffering of the data read - Writing the data to secondary storage (in our case, tape)

Let’s start with the first thing. Let’s see if the client can actually read the data fast enough. I use a tool called “IOzone” (www.iozone.org) to test what performance I can get, and what the ideal blocksize is to read from the primary storage. As the client (Oracle) I’m trying to backup, is using direct I/O, I’m using the “-I” flag to iozone to have it also use direct I/O.

The following is the output for iostat: Iozone: Performance Test of File I/O Version $Revision: 3.394 $ Compiled for 64 bit mode. Build: linux

Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins Al Slater, Scott Rhine, Mike Wisner, Ken Goss Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR, Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner, Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy, Dave Boone, Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root, Fabrice Bacchella, Zhenghua Xue, Qin Li, Darren Sawyer. Ben England.

Run began: Wed May 4 17:29:14 2011

Excel chart generation enabled O_DIRECT feature enabled File size set to 1048576 KB Record Size 4 KB Record Size 16 KB Record Size 256 KB Record Size 1024 KB Command line used: iozone -I -s 1g -r 4k -r 16k -r 256k -r 1024k -r 2048k -r 4096k -8192k -r 16384k -i 0 -i 1 -i 2 -i 3 -i 5 -i 6 -i 7 -i 8 -i 9 -i 10 -i 11 Output is in Kbytes/sec Time Resolution = 0.000001 seconds. Processor cache size set to 1024 Kbytes. Processor cache line size set to 32 bytes. File stride size set to 17 * record size. KB reclen write rewrite read reread read write read rewrite read fwrite frewrite fread freread 1048576 4 9510 9739 39830 40380 844 8277 17027 1345 889123 984691 2034604 2425323 1048576 16 27479 22374 59392 64053 2747 24885 11687 10766 932245 1110729 2404477 2635069 1048576 256 66115 66640 133319 139195 34788 67691 69131 164154 1037669 1189047 2928287 2973216 1048576 1024 101449 99394 222478 248884 245761 98841 250000 253888 926448 1140963 2787121 2955716 1048576 2048 70763 100007 121962 134538 93643 87468 176558 179893 808908 954603 2622587 2757923

As can be seen from the above output, the ideal blocksize for the primary storage in this example, is 1 MB. This will give me a read speed of about 222 MB/s (In Commvault metrics, this is 780 GB/hr). The read block size seems to be controlled by the Cvd/nBUFSIZE registry key, which seems to be specified in KB. The default is 64k, which is not ideal. So I’ve set this to 1024. The nBUFSIZE seems to make a big difference in how fast data is being read. The setting Cvd/nMAXBUFF seems to specify the maximum size for 1 buffer, so this should also be set to something equal or larger then 1024 (I’ve set this to 1024 too).

In the Client Properties, the “Backup Arguments” tab, you can also set the “Data Files per BFS”, where BFS is a BackupFileSet. This will basically determine the read multiplexing for RMAN while reading. As I’m using 4 active paths to the primary storage, I want the “Data Files per BFS” devided by “Max Open Files” setting equalling 4. The 32 / 32 / 8 default seems to work pretty good, though I’ve had improvements while using 64 / 64 / 16. This depends heavilly on the actual number of data files in your database (see the “Content” tab for a rough estimate, or ask your friendly DBA). Basically, you want to have as few BFS as possible, but enough to fill the number of streams you’re using. So on a database with 256 datafiles, and 4 reader streams (see the “Writing” part for how many streams you will need), you want 64 files per BFS. Multiplexed by 4, so 64 / 4 = 16 open files.

Next; the buffering.

CommVault did document the Cvd/nNumPipeLineBuffers parameter. The default (30) is too low, and it should be in multiples of 30. I’ve noticed, that above a certain setting, no notable performance gain can be reached. I’ve set this setting to 300, but it’s a bit of trial and error where you’ll hit your maximum. The Cvd/nNUMBUFFERS registry key seems to also be related. The default for this setting is 64. Increasing it to 256 also gave me a performance boost.

These settings will (presumably) trade memory for buffers. Make sure you have plenty of memory available for caching this much data (nNUMBUFFERS * nBUFSIZE = size in KB).

One disadvantage I noticed by increasing these registry keys is that “simpana restart” doesn’t work. It seems to stop, the logging shows the daemon starting, but it is not working. Using “simpana stop && sleep 5 && simpana start” fixes this. This seems to be related to some cleanup routines the daemon is executing when stopping.

Last; the writing.

Keep in mind that your writer (tapedrive) also has a maximum speed, as does your reader. See http://en.wikipedia.org/wiki/Linear_Tape_Open#Ultrium for the maximum uncompressed speed. In my case (LTO-3), this is 80 MB/s, so 160 MB/s. As my reader can do 222 MB/s (see above) I need 2 tapedrives allocated for maximum speed (running at 111 MB/s). The ESL seems to be happiest when writing at a block size of 256k. Set this in your storage policy. While you’re at it, change the chunk size to 16384MB. This insanely high chunk size seems to improve write speed drastically. Hardware compression in the storage policy does not seem to be a slowing factor. In fact, with this turned off, I’d need 222/80 = 3 tapedrives. So leave the hardware compression enabled.

Also, do enable multiplexing on your streams. As I have 4 active paths to the primary storage, I’ve set the multiplexing factor to 4. To use 2 tapedrives, I will therefor need 8 readers to fill both tapedrives.

So, in the SP set the number of device streams to a minimum of 8. Check the “Enable Stream Randomization”. This will slow your restores, but at the moment, I’m looking for raw throughput during backup operations. Disable all compression and deduplication on the subclient. Set the “Number of Data Backup Streams” to 8 and the “Number of Archive Log Backup Streams” to 8 (if you’re also doing an archive log backup).

Another easily tunable parameter is the “Block Size”, found in the “Details” in the subclient properties. CommVault recommends setting this to 1 MB or more for LTO drives. As we’re using a block size of 256k in the Storage Policy, I’ll set this to 1MB (1048576), so one RMAN block equals 4 blocks to the tapedrive.

Epilogue:

As warned, most of the above is purely guesswork, and I have to admit that some of the reasoning above is based on nothing, a feeling, or unfounded calculations. Try these settings, but be prepared to rollback your changes. As warned, always keep a close eye on network / storage / client performance.

COPYRIGHT DISCLAIMER

THIS ADVICE IS DISTRIBUTED IN THE HOPE THAT IT WILL BE USEFUL, BUT WITHOUT ANY WARRANTY. IT IS PROVIDED "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THIS DOCUMENT MAY BE REDISTRIBUTED FREELY, AS LONG AS THIS COPYRIGHT DISCLAIMER IS PRESERVED IN FULL.