Scheduling Operations in NetWorker

Aaron Kleinsmith

EMC Proven Professional Knowledge Sharing 2010

Aaron Kleinsmith P&E Consultant, EMC Education EMC² [email protected] Table of Contents

Scheduling in NetWorker ...... 3 Different ways to backups ...... 3 Group resource ...... 3 Scheduling the Group backups ...... 4 On-demand Group backup ...... 5 Restarting a Group backup ...... 5 Savegrp ...... 6 Savefs and save ...... 8 or Unix/Linux cron ...... 9 External scheduling applications ...... 9 Using Schedules effectively ...... 9 Re-occurring events in the Schedule ...... 10 Week or Month ...... 10 Schedule Overrides ...... 10 Backing up intervals longer than a single day ...... 12 Planning for dependent save sets ...... 14 Retention Policy ...... 15 How to use Directives effectively ...... 15 Excluding files from a backup ...... 16 Use Null or use Skip? ...... 16 Using Groups, Schedules and Directives for Application Backups ...... 17 Cloning and staging ...... 18 Overview of Cloning ...... 18 Backing up and Cloning with Disk ...... 19 Scheduling cloning jobs ...... 20 Cloning as part of a backup Group ...... 21 Cloning using nsrclone ...... 21 Examples using nsrclone ...... 23 Retentions on Cloned Save Sets ...... 24 Using nsrclone with mminfo ...... 25 Cloning by ...... 26 Recovering from a Cloned Save Set ...... 26 Overview of Staging ...... 26 Staging Resource...... 27 Using nsrstage command ...... 27 Other ways to schedule scripts in NetWorker ...... 28 Conclusion ...... 28

Disclaimer: The views, processes, or methodologies published in this compilation are those of the authors. They do not necessarily reflect EMC Corporation’s views, processes, or methodologies.

2010 EMC Proven Professional Knowledge Sharing 2 Scheduling in NetWorker

Many activities need to be scheduled in NetWorker™ but performing a backup is the most important. There are many different requirements depending on what of system, what type of data, or who owns the data. These requirements you to decide when to start the backup, how long you have to complete the backup, how long the data needs to be recoverable, and whether or not the data will need to be transported offsite. There are many technologies available to change the way data is backed up and stored. This great amount of flexibility complicates the way we manage backups. This article describes strategies that you can use to schedule backup and cloning tasks in NetWorker.

Different ways to start backups

Group resource

Configuring a Group resource to “Autostart” at a specific is the most common method to start a backup in NetWorker. It is straight-forward; just create a Group resource with a configured start time listed in 24-hour time (for example, a setting of 19:00 for a start time of 7:00 pm), enable the Autostart attribute, and ensure one or clients are in the group. This type of backup is a -initiated backup since the NetWorker server requests all the clients in a group to send data for backup.

Which save sets that client will be sending for backup are configured In each client resource. When performing backups for clients, the NetWorker server will initiate separate save commands on the clients to start for each save set that needs to be backed up. Prior to starting the save commands, start the savefs program on each client to first verify save set paths or discover the save sets paths if a save set keyword of “All” was configured for a client. The savefs command running on each client helps build a worklist of save sets that will be sent for backup. The NetWorker server will collect the worklists of these save sets from each client, and organize the data to the needed storage nodes.

2010 EMC Proven Professional Knowledge Sharing 3 Creating multiple groups to begin at staggered start times, and dividing clients between these groups is a good strategy when scheduling the backups. The size of the environment, the number of clients, start time requirements, and backup windows will help you to decide how many groups and when they should be configured to start. When you stagger the groups, the NetWorker server will not be left with too many clients and save sets to queue and manage at one time. This allows higher-priority clients to be backed up first by placing them in the groups that are scheduled to start first.

Scheduling the Group backups

Once a day backup

The NetWorker Group resource can be left with the default “Interval:” attribute of “24:00” that will run the group once every 24 hours if the group backup is required to be performed once every day at the same time. Even though a once a day backup is common, there are some other strategies for scheduling the groups more or less frequently than once a day. Backing up more frequently than once a day is covered in the next section. The Group will need to be combined with an appropriate Schedule if there is a need to back up less frequently than once a day.

Backing up at an interval time

If you must do a specific backup more frequently than every 24 hours, the Interval attribute can be shortened to the necessary requirement such as 30 minutes (00:30) or every 2 hours (02:00). This scheduling works well when you need to back up data that changes frequently such as database logs or other types of transactional logging data from important applications.

Backing up at an interval during certain hours

Starting a backup at an interval can be very useful, but in certain circumstances there may be a window of time you don’t want the backups to occur – a black-out window. A group can start at a specific interval, but NetWorker does not allow you to stop that interval for a period of time. For example, if you need to back up transaction logs of a database during business hours and not at all during its nightly, level full, 8-hour backup

2010 EMC Proven Professional Knowledge Sharing 4 window, the interval does not help to prevent any overlap between these two backups. In this case, you can configure multiple groups with needed start times and select them in each client resource so the client can start backing up at these required times.

So if the requirement was to perform a backup every 2 hours during normal business hours (9am-5pm), five Group resources could be created each with an interval of 24 hours and different start times: 09:00, 11:00, 13:00, 15:00, 17:00. A client can be made part of multiple groups so the client resource that needs the frequent backup can be made part of each of these groups. A nightly, once a day backup for an overnight backup would also be needed, so include one more group for that.

Create more groups if you need to perform the backup more frequently than every 2 hours. If you need to perform backups every hour, create groups at 1 hour intervals during the needed window. For example, there may need to be 9 or 10 groups such as 8:00, 9:00, 10:00, 11:00, 12:00, 13:00, 14:00, 15:00, 16:00, 17:00.

Use a name that describes the purpose and time the group runs; avoid using spaces or special characters when deciding on group names. When you use spaces in group names, and it needs to be called from the command line, surround it with double-quotes. Consider whether the savepnpc (save Pre aNd Post Command) will ever be needed when choosing a group name. This feature uses the group names to create and name a configuration file that will be created on the client. NetWorker will have no trouble creating files for this feature if there are no spaces or special characters in the names.

On-demand Group backup

Once you create a Group resource, start it from the Monitor section of the NetWorker management console Graphical User Interface (GUI) window under the Groups tab. Right-click the Group and choose ‘Start’ to begin the group backup. You can set the Group resource to either ‘enabled’ or ‘disabled’ and still be started.

Restarting a Group backup

You can also initiate a restart from the same right-click menu as the on-demand group backup. If a group is restarted, any save sets that had not completed in the last run of the group will be started and any save sets that had completed from beginning to end

2010 EMC Proven Professional Knowledge Sharing 5 will not be started again. The inclusion or exclusion of data from the restarted group is based on the individual save sets. You could include an earlier failed save set of a client in the restart, but different successful save sets of that same client will not need to be started again. For a restart to behave in this manner, it will need to be started within 12 hours of the last time the group finished in order for it to include only failed save sets. Every save set in the group will be started again if a group is restarted out of this 12 hour restart window. This “Restart window:” is an attribute setting in the Group resource that you can customize and change from the default “12:00” hour window to any value that is less than the interval setting of the group.

Savegrp

You can also start a Group from the command line using the command “savegrp” on the NetWorker server. This command runs only on the NetWorker server and can be called directly from the command line as savegrp with appropriate options, or included in a .bat or script file run from the command line. To automate the start of the backup using the command, the savegrp command can be put into a .bat or script file and called on the NetWorker server through an external scheduler such as the Windows Task Scheduler or UNIX/Linux cron.

Different switches can be applied to the end of the savegrp command to allow for different ways of starting the backup. If the switches overlap or conflict with options already set in the GUI, the options used at the end of the command will over-ride or take precedence over the GUI options. For example, there is a lower-case “L” switch (-l) that specifies the level of backup. If this is included as “-l Full” then regardless of what is scheduled in the GUI this backup will be of level Full. There are also switches to perform a preview backup with verbose options (-pvvv) that is useful to verify configuration or troubleshooting, back up specific clients (-c client_name), or to force a particular level of backup.

2010 EMC Proven Professional Knowledge Sharing 6 Here are some examples of the savegrp command: savegrp command Description of backup started Savegrp Start the Default group savegrp NightlyBackup Start group named “NightlyBackup” savegrp –pvvv NightlyBackup Start “NightlyBackup” group with preview and verbose savegrp –c client1 NightlyBackup Start only client with hostname “client1” in the “NightlyBackup” group savegrp –l Full NightlyBackup Start NightlyBackup group and back up all savesets at level full regardless of client schedules

Another useful form of the savegrp command is the upper-case “O” option that will back up Only the client file indexes of the clients in the group being started. It will ignore the save set attribute in the client resources. If the backup server is part of the group, the NetWorker server client file index and bootstrap will also be backed up. This is useful when you need to back up all the indexes and bootstrap at one time, such as just prior to performing an upgrade of the NetWorker server.

For daily operations, include the command in a .bat or script file and scheduled to run with an external scheduler to have all the indexes back up at the same time instead of only when the groups end at different times throughout the evening. Start a Group resources with all clients belonging to it to ensure that all client file indexes and the bootstrap are captured in the backup. You can create a special group with all clients belonging to it, but leaving the Autostart: attribute set to Disabled. This group can be used strictly for performing client file index and bootstrap backups with the savegrp –O option. All the clients will belong to more than one group – the normal scheduled group and the one that is created for the savegrp –O command.

2010 EMC Proven Professional Knowledge Sharing 7 Savefs and save

You can also start a backup from the command line on the client system; this is a client- initiated or manual backup. When a group backup starts, the NetWorker server contacts the clients that are part of the group and requests that a savefs command be started on each of the clients to: • collect information • build a work list of what needs to be backed up • call the save command(s) that will be needed for copying and sending the data to an appropriate storage node.

You can also call the savefs command or the save command directly on the NetWorker clients from a command line, or put in a .bat or script file and call through an external scheduler. Include a process to back up the client file indexes of these clients when choosing to start backups only from the client. Normally, NetWorker will automatically back up the client file indexes at the end of a group backup of the client, but will not when an administrator initiates the backup from the client itself.

You do not need to run the save commands separately if using the savefs command since part of the savefs’ job is to call necessary save commands. Save commands can be used directly to back up specific save sets as necessary if you are not using savefs. When you run save from the client, the backup level will be recorded as ‘manual’. You can force a level using a lower-case “L” option. For example, using the following command syntax to record the backup as level Full “save –s NetWorker-server- hostname –l full /etc” where “NetWorker-server-hostname” is the hostname of the NetWorker server and “/etc” is the directory being backed up.

Use the Level attribute or a pool if you need to sort the client-initiated backups to specific media pools in NetWorker. Sorting data to media pools using Group names is a common practice when configuring media pools in NetWorker. They can’t be sorted by Group name for the client-initiated backups since they don’t start as part of a group. Rather, sort them based on other criteria. “Manual” is one of the Level attribute values that you can use for a pool configuration to sort the client-initiated backups away from the Default pool and other backup data.

2010 EMC Proven Professional Knowledge Sharing 8 Windows Task Scheduler or UNIX/Linux cron

The Windows has a scheduler referred to as the Windows Task Scheduler (@) or Scheduled Tasks option in the . This scheduler allows an administrator to schedule .bat or script files to run at different times. There is some flexibility in regard to weekly, monthly, or other re-occurring events that are not available with NetWorker Groups. Commands like “savegrp” with appropriate options can be included in a .bat or script and called from the Windows Task Scheduler.

Likewise, on UNIX and Linux, the included cron scheduler can be used to call scripts that include the savegrp command. This may allow for more flexibility than relying strictly on the Group options in the NetWorker GUI.

One disadvantage of scheduling jobs outside of NetWorker is that they need to be monitored and updated separately. The flexibility achieved by using an outside scheduler can be configured with an appropriate combination of Groups, Schedules, and Directives in NetWorker to have everything in one interface. This is described later in this article.

External scheduling applications

There are other job scheduling (or batch scheduling) software products that you can use with NetWorker by calling commands directly or through .bat or script files. They may be a good option if there is already job scheduling software being used in the environment. Examples of job scheduling software products are BMC Control-M, CA Autosys, IBM Tivoli, NetIQ, etc.

Using Schedules effectively

As mentioned earlier, you sometimes schedule events once a week, once a month, or even once a year but can’t accomplish that with the NetWorker Group resource alone. NetWorker Groups can be configured in conjunction with Schedules to handle re- occurring events that are longer than 24 hours without the help of an external job scheduling tool. A NetWorker Schedule resource is used to indicate what level of backup should take place on a given day such as level Full, Incremental, or Differential (1-9) backups. You can include a ‘skip’ that schedules NetWorker to not back up on a

2010 EMC Proven Professional Knowledge Sharing 9 particular day. With a ‘skip’ scheduled, a Group resource may be set to “Autostart:” and still run at a given time of day, but once the ‘skip’ level is encountered in the Schedule the clients this schedule applies to will be skipped and no data will be backed up for them. A Schedule resource that has a Full backup on Sunday but set to skip for the other six days of the week will only back up once a week. A schedule could similarly be set up with a skip on all days of the month except one to do a once a month backup and skip all days but one in a year to accomplish a once a year backup.

Re-occurring events in the Schedule

Week or Month

Many EMC customers need to perform a NetWorker backup on a once a month schedule, but it needs to happen on a specific day of the week – such as a Friday or a weekend. Typically, in the Schedule resource of NetWorker, the interval for a backup can only be set to a Week or Month. When using “Week,” a change to the schedule is made only for a specific day of the week (for example, an incremental backup on Monday) and all similar days of the week (incremental on every Monday of every week) will be updated with that same level. With the Month period of time, a specific numerical day of the Month (for example, the 1st of the month) will be updated to the selected level for every month throughout every year. That numerical day of the month will usually fall on different days of the week. So even though it may be necessary to get a particular backup done at the beginning of a month, if the 1st is on a Wednesday that may not be convenient since Wednesday may be a day in the middle of the week and there may not be a large enough backup window. It might be better to perform the backup on the first Saturday of the month when there is an appropriate backup window.

Schedule Overrides

NetWorker Schedules have ‘overrides’ to help set levels outside of the normal week and month. You can easily set the overrides in the NetWorker GUI one day at a time wherever needed. When an override is used, it normally will change a very specific day

2010 EMC Proven Professional Knowledge Sharing 10 (to set a level skip on December 25th, 2010 for example) displaying an asterisk beside the set level and not having it replicate anywhere else in the schedule.

If there are many overrides to set – such as the first Saturday of every month – it is not convenient to select a single day in each month of each year. Override using text key words. Right-click on any one of the Schedules resources in the GUI to unselect the option called “Show Schedules as Calendars.” This allows the schedules to be shown in text mode – or similar to the way they are stored in NetWorker’s Resource database.

Opening the properties of a schedule allows an administrator to directly update the Override: attribute with one or more custom text overrides. This could mean entering in a level and specific date in a MM/DD/YYY such as “skip 12/25/2010.” Likewise, a specific day of a month for every year could be entered by leaving off the year value and including only MM/DD, with the syntax such as “skip 12/25.” Leaving off the year will cause the level to repeat on the same day every year. The general format for the text overrides include the level first and then the date or frequency that needs to be applied. There can be multiple overrides on the same schedule and they will be applied in the order listed in the attribute.

Some examples of acceptable syntax for the text overrides:

Description of needed policy Attribute setting Perform Full backup on first day of the year full 1/1 Perform a Full backup last Friday every month full last Friday every month Perform a level 5 on second Tuesday every month 5 second Tuesday every month Perform Full backup first Saturday every quarter full first Saturday every quarter Perform full once a year on last Friday full last Friday every year

You can use the nsradmin command to update these overrides in Override: attribute. Use the nsradmin command to create or modify resources in the NetWorker server resource database.

2010 EMC Proven Professional Knowledge Sharing 11 Backing up at intervals longer than a single day

Here are three examples of when a backup interval of longer than one day is useful: • when there is a need to perform an archive type backup such as a full once a week for onsite backups • full once a month to be sent off site • full once a year to be archived for longer period of time

Typically, a backup is a secondary of primary data that is kept in case of failed equipment, human error, or a site disaster that causes the primary copy of data to be lost as opposed to being archived. Saving a “once a month backup” for an extended period of time may seem like a good strategy to archive data, but there could be many files created and deleted within that month that may not be captured in the backup. Consider an archive option with NetWorker, or other archive software such as DiskXtender, if it is important to keep certain data for an extended period of time off the production servers for archive purposes.

There is a difference between archiving data and archiving a backup of data. Sometimes, you want to keep a certain backup for a longer period of time than the normal rotation. For example, a common business requirement might be to keep the first full backup of the month for 7 years, while keeping regular day to day backups for a shorter period of time (i.e., 3 months). You can implement this strategy with NetWorker, but be careful since there may be gaps in continuity between the archived backups.

Do not use backup to keep an archive of a particular file that has had its primary copy intentionally removed from a system. Backup data is subject to retentions; the copy will eventually expire from the backup environment and be overwritten. Archive is used to copy or data off primary storage to a secondary location and provide a method for easily accessing that file if it is needed – without subjecting the copy to expiration. There may be certain files or data that are not captured in an archived backup when a full backup is archived for a longer period of time than the normal backups.

For example, if you are considering a strategy for a file server for a busy office where thousands of users are creating, updating, and removing documents on a regular basis,

2010 EMC Proven Professional Knowledge Sharing 12 it is unlikely we could predict whether a critical file will exist on the system during a monthly full backup. For example, if we have a file created on Tuesday of the 2nd week of the month, and it is deleted the following week, it may not exist on the day of one of the monthly backups.

However, there are occasions where we must keep or store data for different lengths of time depending on when the backups took place or their frequency. A common requirement is to perform a Daily backup and keep one retention period, perform a Weekly backup and keep for a second retention, perform a Monthly backup and keep for a different retention, and sometimes perform a Yearly backup for a completely different retention. One strategy that you can use within NetWorker to accomplish this backup configuration is to create four parallel settings of Groups, Schedules, Retentions, and Pools for Daily, Weekly, Monthly, and Yearly requirements. Here is what a generic setup may look like in NetWorker to accomplish this kind of requirement:

Daily Group => Daily Schedule => Daily pool => Daily Clone pool Weekly Group => Weekly Schedule => Weekly pool => Weekly Clone pool Monthly Group => Monthly Schedule => Monthly pool => Monthly Clone pool Yearly Group => Yearly Schedule => Yearly pool => Yearly Clone pool

All the groups are set to run every day at a specified time, but the schedules are used to determine if an actual backup will take place. For example, the Daily groups will have a schedule that shows regular incremental backups during the weekdays with a ‘skip’ configured for one weekend day, and the days Monthly and Yearly backups are to run. The Weekly backups would be configured as the weekend Full backups with ‘skip’ overrides to be configured the days the Daily, Monthly, Yearly backups will run. The Monthly Schedule will have a ‘skip’ scheduled for all the days any Daily or Yearly backups are set to run. The Yearly schedule will be set with a skip 364 days of the year except for the one day the Yearly backup is set to run.

The idea is to have a complete picture but uses four different schedules. The chart displays a 31 day month with a Monthly Full level backup on the first Saturday of the month and a Yearly level Full on the last Saturday of the month that could be sent

2010 EMC Proven Professional Knowledge Sharing 13 offsite. Following the Monthly and Yearly backups are Weekly Full backups the next day that can be kept onsite for immediate recovery requests.

SUNDAY MONDAY TUESDAY WEDNESDAY THURSDAY FRIDAY SATURDAY 29 – Daily 30 – Daily 31 - Daily 1 - Daily 2 – Daily 3 - Daily 4 – Monthly 5 – Weekly 6 - Daily 7 - Daily 8 - Daily 9 – Daily 10 - Daily 11 - Weekly 12 - Daily 13 - Daily 14 - Daily 15 - Daily 16 – Daily 17 - Daily 18 - Weekly 19 - Daily 20 - Daily 21 - Daily 22 - Daily 23 – Daily 24 - Daily 25 – Yearly 26 - Weekly 27 - Daily 28 - Daily 29 - Daily 30 - Daily 31 - Daily

An advantage of configuring the schedules in this manner is that everything is configured and managed directly within the NetWorker GUI. Fewer configured resources may be needed in NetWorker if an external scheduler was used to accomplish the same schedule. However, once these resources are initially configured, it will be easier to meet ongoing changes in requirements for the backup environment and much easier to hand the environment off from one administrator to another since everything is configured in one place.

Planning for dependent save sets

Consider dependent save sets when a full backup is archived or stored out of sequence from the other backup data. When performing a Level Full backup once at the beginning of a week, the remaining incremental backups are dependent on the full. The incremental backups are only changes that occurred on the save sets since the previous backup was performed. You need to restore the full backup first, and restore the following incremental backups afterwards to bring the data back as it looked on a particular day of the week to restore an entire save set. If the full backup is not available, it will be impossible to restore the entire save set back to a point later in the week since any of the unchanged files were only kept in the full level backup.

If a full backup has been flagged as the monthly full offsite copy to maintain once it is sent offsite, you would need to recall files from that day and also from any of the following dependent backups as well. That is why in the previous chart example the Monthly and Yearly full backups show a Weekly full on the following day.

2010 EMC Proven Professional Knowledge Sharing 14 Retention Policy

Organizations often need to keep different types of backup data for different lengths of time. In NetWorker, a save set expires and is marked as recyclable when the Retention policy expires. The expiration date is assigned to the save set during the backup and comes from the Retention setting in the client resource. The expiration date is recorded in the media database as the ssretent value.

It is a good practice to keep data with different retentions sorted to different pools. This makes sense especially when backing up to tape or virtual tape because it is more likely a number of save sets on the same tape will expire around the same time. The entire tape can be expired and re-used by NetWorker once all the save sets expire. When backing up to different media types, such as disk backups, virtual tape, or physical tape, the pools can also be used to help direct a backup to the correct media type. The Device attribute in the Pool resource can be used to restrict a pool to using certain devices or you can select a chosen volume preference type.

How to use Directives effectively

Directives are used to direct the NetWorker save command that is used for file system backups to perform special functions such as encrypting data, compressing data, excluding files during a backup, etc. A common reason to use a directive is to exclude files from a file system backup.

Directives can be applied in two ways, either in a file on the client or through a directive resource in the NetWorker GUI. If applying a directive on the client, you will need to place the directive statements in a .nsr file for UNIX/Linux clients and in a nsr. file for Windows clients. However, creating a directive resource and applying it to a client resource is a better way to manage the directives since they can all be viewed and managed in one place – the NetWorker Management Console GUI. With the local directive, it is easy to lose track of which systems have them and where they were placed. With the directive resource, they are listed in one place in the GUI so they are much easier to track and manage.

2010 EMC Proven Professional Knowledge Sharing 15 Excluding files from a backup

You may need to exclude one or more file types from a backup. Simply list only the file systems, directories or files that you want to back up in the client resource save set attribute. However, use the “All” keyword for the save set attribute when you want to perform a backup of most everything from a client. Using “All” as the save set attribute lets NetWorker discover the save sets on the client at the time of the backup. This is helpful when you are performing backups of servers managed by other administrators since any new file systems that are added to a client will be captured during the backup without any additional configuration. However, there are situations when it is not necessary to back up everything on a client. Directives help bridge the gap between these two methods by using “skip” or “null” statements to exclude data from the backup. Use a save set of “All” to back up all save sets and a directive applied to the client to keep from backing up what is not wanted.

Use Null or use Skip?

Whether you use Null or Skip in a directive to exclude data from backup can make a difference, even though they both will effectively exclude your data. The primary difference has to do with the way the directories and files are recorded in the client file indexes. When you decide to use skip in the directive it effectively records a kind of ‘blank’ or whitespace in the client file index in place of the directory or file being skipped for that point in time. This means you will not see any record where that file or directory would have existed.

When you use null it will not record anything in that place in the client file index. If a particular file or directory was backed up previously, you would still see those previous record entries, not from the current backup but from previous backups.

Perform backup and recovery testing to be sure you are getting the result you need from either skip or null. Testing will help to understand the difference so that you can make an informed decision about which is needed.

2010 EMC Proven Professional Knowledge Sharing 16 Using Groups, Schedules, and Directives for Application Backups

Creating client configurations for backing up application data is one very useful situation; we can use many of these configuration strategies with Groups, Schedules, Directives, and Pools. When performing a file system backup with NetWorker, the server will request a client to run the savefs and save commands to perform the backup. The save command is used to capture a copy of data and send it to a storage node. However, the save command does not understand how to successfully capture data from a file system that has data on it that may be locked or is changing during a backup – such as application or database data. This is where the base NetWorker product will need help from special commands that are installed and included with the NetWorker Modules.

The NetWorker Modules are installed on the clients and used to back up the applications using special commands, shared application library files, and scripts. This will ensure that NetWorker backs up the applications in a consistent state that can be successfully restored. The NetWorker Modules are designed to ensure that each application or database vendor recommendations are followed and any needed backup utilities or provided by the vendors are used. Although the NetWorker Modules do allow for level Full and other granular levels of backups (for example, transaction logs, archived redo logs, block-level incremental, etc.) they do not capture all the data from the systems. There is data such as the operating system, configuration files, and other file data that is not part of the application that may need to be backed up on these application servers.

You must create at least two client resources in NetWorker using the client hostname to make sure all data is captured from a client. One client is used to back up the NetWorker Module specific save sets and the other to perform file system backups of the client. There may need to be even more client resources created if some backups need to include special commands or schedules for incremental or transaction log type backups. Needing multiple retentions for different types of data may be another reason to have more than two client resources created for the same client since the retention policy is configured in each client resource.

2010 EMC Proven Professional Knowledge Sharing 17 When creating the client to handle file system backups, use a save set value of “All” to capture all local (and SAN) file systems for a client. However, using “All” alone will mean that NetWorker will also capture any application data on the client. This may include getting errors when NetWorker tries to open locked files on Windows servers or capture application data that is active and changing on UNIX and Linux servers. To avoid this, create and apply a Directive resource for the file system client that is used to exclude the known locations of the application files that will be captured by the NetWorker Module backup. Additionally, if backing up a Windows-based server, use a “save operations” setting to exclude Volume Shadow Copy Services (VSS) from backing up applications such as MS-SQL and MS-Exchange if they are being backed up with a NetWorker Module instead. The save operations attribute in the client resource can be used to tell NetWorker to turn off the capability to request backups using VSS on Windows 2003, 2008, and Vista systems. Only one method should be used for backing up the applications when backing up MS-SQL and MS-Exchange so that all transaction logs are found in the same place when performing a restore.

Cloning and staging

Overview of Cloning

Cloning makes additional copies of data that has already been previously backed up. Cloning is a valuable function in NetWorker to help ensure data is protected on multiple pieces of media. It is gaining even more importance to help with integrating backup to disk and backup to tape technologies. Cloning helps to implement different tiers of disk storage, virtual tape libraries, or physical tape and keeps backup copies on each of these different media for different lengths of time. The introduction of specific clone retention policies (clretent in media database) and the ability to set or extend the browse policy of clones save sets has helped with managing tiered backup storage.

2010 EMC Proven Professional Knowledge Sharing 18 Backing up and Cloning with Disk

Disks are part of a backup strategy and in some instances they are the only media used for storing backup data. There are different ways that disk storage can be configured for NetWorker. For example, if using a SAN storage array, the disk is presented to the NetWorker server or storage node as one or more file systems and then configured as an advanced file type device (adv_file). Disk can also be presented as a CIFS or NFS share to the NetWorker server or storage node and used as an advanced file type device. This will allow network attached storage (NAS) devices or disk libraries with NAS features (such as the Data Domain® deduplication appliances) to be a destination for storing backup data. Another method for using disk with NetWorker is to use a virtual tape library such as an EMC Disk Library (EDL) to be presented as if it were a tape library. The Data Domain deduplication appliances can be presented as a tape library to NetWorker with a virtual tape library option as well.

With all of these ways to use disk, the backup speeds will usually be quicker than what can be achieved through traditional tape. Seek and for recovery with disk storage is definitely quicker than traditional linear tape due to the random-access nature of disk storage. It is beneficial to have the original backup data reside on the disk for some period of time in case a recovery is required. Typically, recovery requests are needed shortly after the backup so having the backup data on disk allows a quick recovery. However, you may have to keep some or all of the backup data recoverable for a longer period of time than is practical or cost-efficient for keeping it on backup disk. Copying or moving the data from the disk media to another media allows you to extend the length of time the data is available for recovery. Cloning and staging operations in NetWorker can be used to copy or move the primary backup data from disk to another media type like physical tape.

Another reason for copying the data off of the original disk media is to have or maintain a copy in case the original back copy is lost. The original copy could be lost due to mechanical failure, data corruption, human error, backup software mis-configuration, site disaster, etc. It is important to have the backup data stored on more than one piece of

2010 EMC Proven Professional Knowledge Sharing 19 media. NetWorker cloning is ideal for helping to protect against these possible failures soon after the backup, while still allowing the original backup copy to be available in case of any recovery requests.

Copy the original backup data to physical tape and then transport it if sending the data offsite. If a method other than physically transporting tape is needed, NetWorker can also clone the data across a LAN-WAN connection to a destination NetWorker storage node at a remote location. Cloning is more ideal than staging in these situations since it will allow the original data to be available onsite in the event a recovery is needed.

To free up space and make room for new backups, different copies of the save sets can have different retentions. The backup copy on disk can have a retention (ssretent) to expire the data sooner than the clone copies (clretent) that can have a longer retention. For example, the original backup copies on disk can have a shorter retention, such as 7 days, while the clone copies on tape can have a longer retention, such as 1 year.

Scheduling cloning jobs

A cloning operation can be started in three ways: 1. through a setting in the Group resource 2. by running nsrclone from the command line 3. by manually selecting volumes or save sets in the NetWorker Management Console GUI

A cloning operation reads an original backup of a save set and then write out the data that is read to a destination device. Two devices are used for every clone operation that is started; one for reading the original backup and one for creating the clone copy. The number of devices available will affect your decision about when to begin cloning. For example, if there are many devices available for backup and even more devices can be added and dedicated for cloning, it might be alright to begin cloning data while a backup is occurring. This scenario is possible when using Advanced File type devices or virtual tape drives in a virtual tape library.

2010 EMC Proven Professional Knowledge Sharing 20 Cloning as part of a backup Group

There is an attribute setting in the Group resource to enable clones and choose a destination clone pool to begin a clone job automatically through the NetWorker Administration GUI. When this option is enabled (set to “Yes”), after a Group completes, all save sets that were backed up in that group will be cloned to the destination clone pool. If there are Groups that are scheduled to start afterwards, they will be fighting for device resources with the clone operation for the group. If we have many Groups starting and ending in the same backup window, it can be difficult for NetWorker to find enough devices to complete all operations in a reasonable time with Groups and Cloning running at the same time. When a clone operation starts, it will use two devices that could be used for backups and further delay the time the overall backup takes to complete. It may make sense to let the groups start the cloning operations if there are dedicated devices for cloning and the backup window is not impacted.

Start the clone operations after all the backups have ended if there are a limited number of devices. Wait until all the Groups have ended before starting any cloning when using physical tape devices for backup and cloning operations. Consider throughput and storage node resources such as Fibre, iSCSI, CPU, memory, etc. and ensure that they are sufficient to handle backups and cloning at the same time even if there are plenty of devices to perform backup and cloning simultaneously.

Cloning using nsrclone

Start the GUI manually or using the nsrclone command from the command line to start the cloning operation after all backups have completed. Using nsrclone in a .bat file on Windows or a shell script on Unix or Linux is a good option since it allows the Windows Task Scheduler or cron to start the clone scripts automatically.

The basic form of the nsrclone command is to specify the unique ssid of a save set that needs to be cloned such as: # nsrclone –S ssid Where ssid is the save set identification number of the save set that needs to be cloned.

2010 EMC Proven Professional Knowledge Sharing 21 However, there are other variations of the command, and some possible switches. A description of each follows.

Valid nsrclone option Description of option -b pool_name Clone data to destination pool “pool_name”. When cloning data, the destination copy needs to be sent to a clone type pool. -B pool_name Clone data from source pool “pool_name.” -C number Clone only save sets with copies less than “number.” -f file_name Clone only those save sets or volumes listed in the file “file_name”. Each entry for save sets to be the ssid listed each on separate lines. - y retention_date Set clone retention (clretent) to the date specified as “retention_date” to be in acceptable nsr_getdate format. -w browse_date Set clone retention (ssbrowse) to the date specified as “browse_date” to be in acceptable nsr_getdate format. -n Do not actually execute the clone operation but instead list out all save sets that will be cloned. -t start_query_time Set a start time for a query window from the media database where “start_query_time” is the beginning of a window of time to select save sets from for cloning specified in nsr_getdate format. -e end_query_time Set an end time for a query window from the media database where “end_query_time” is the ending of a window of time to select save sets for cloning specified in nsr_getdate format. -c client_name Clone only those save sets belonging to client with name “client_name”. - g group_name Clone only those save sets that were backed up in the group of name “group_name”. -F If included, will tell NetWorker to skip cloning invalid save sets.

2010 EMC Proven Professional Knowledge Sharing 22 Some of the options mentioned in the chart reference nsr_getdate(1m) format. The term nsr_getdate is a reference man page on UNIX and can be found in the NetWorker Command Reference Guide. The commands that use date formats in NetWorker support the described date formats in nsr_getdate(1m). The acceptable date formats include hours, days, weeks, months , years as well as MM/DD/YYY or “MM/DD/YYYY HH:MM:SS”. When a period of time in the past needs to be specified, the word “ago” can be attached to the end such as “1 week ago.”

Some examples and their descriptions: nsr_getdate format Period of time description 3 days Three days into the future 3 months Three months into the future 1 quarter Three months into the future 1 year One year into the future 7 days ago 7 days or 168 hours into the past 9 months ago 9 months into the past now The current time today Midnight of the current day yesterday Midnight of the previous day 7yesterday 7 times yesterday or 7 days into the past

Examples using nsrclone

This following section shows some full nsrclone command examples with switches, sample pool names, and date formats. The browse policy would never be set to less than the clone retention or the save set retention.

This first example shows the command to clone all save sets that were backed up in the last week to a pool named “Default clone.” Keep the copies for one year and exclude save sets that were already cloned. nsrclone –b “Default clone” –y “1 year” –C 2 –t “7 days ago” –e “now”

2010 EMC Proven Professional Knowledge Sharing 23

This second example shows how to clone all save sets backed up in the last day from the Daily Backup pool to the Daily Clone pool, that have not yet been cloned. It also shows how to set a browse policy of three months and retention policy of two years. nsrclone –B “Daily Backup” –b “Daily Clone” –w “3 months” –y “2 years” –C 2 –t “24 hours ago”

This third example shows how to clone all save sets backed up in the last two days from the Nightly Group backup to the Daily Clone pool, that have not yet been cloned. It also shows how to set a browse policy of one month and retention policy of one month. nsrclone –g “Nightly Group” –b “Daily Clone” –b “1 month” –y “1 month” –C 2 –t “48 hours ago” –e “18 hours ago”

This last example shows how to clone all save sets that have not yet been cloned and belong to client with hostname “database-client” to a destination pool named “Tape pool.” nsrclone –c “database-client” –b “Tape pool” –C 2

Retentions on Cloned Save Sets

You can specify retention for a clone copy of a save set in a command option of nsrclone or in the Pool resource for a clone pool. When you specify retention in the pool resource for a clone pool, it will automatically be applied to all save sets copied to that pool.

Browse and retention times for clone save sets can also be shortened or extended using the nsrmm command with the –w option to set the browse expiration time and the –e option to set the retention. Values of time can be specified in nsr_getdate format. If a specific clone instance of a save set is to be modified, after the –S option at the end of the nsrmm command, the save set id and clone id will both need to be specified.

2010 EMC Proven Professional Knowledge Sharing 24 For example, nsrmm –w “3 months” –e “1 year” –S 12345678/98765432 where 12345678 is the save set id number of the save set to be modified and the 98765432 is the clone id of that instance of the save set to be modified. This example command would update the browse policy to three months and the retention one year from the current date and time.

Using nsrclone with mminfo

You can also use the nsrclone command in conjunction with mminfo. Mminfo is a command used to query the media database and report back information. The long form of the command uses a “-q” option for the query constraint and the “-r” option to report information to the display where the command is being run.

Use the mminfo command on any NetWorker server to generate a list of save sets as ssid numbers and have the output redirected to a file. Then, use the nsrclone command with the “-f” option to read back the file mminfo created to clone all save sets listed in the file. Various options can be added or removed from the mminfo queries to get specific save set lists in different files, and have multiple nsrclone commands started at one time to make use of multiple pairs of devices for cloning. The overall cloning operations will complete faster by having multiple cloning operations running at once.

You can combine the mminfo and nsrclone commands in a .bat or script file with some wrapper commands and an external scheduler like Windows Task Scheduler or cron to automate the cloning operation. An example mminfo and nsrclone command combination is shown here. mminfo -q “savetime>=”date/time”,copies<2,!incomplete' -r -ssid > mminfo.txt nsrclone -b Default -S -f mminfo.txt

The example mminfo command will build a list of save sets that were backed up after “date/time” (date/time to be replaced by a valid time specified in nsr_getdate format), have not yet been cloned (copies<2), and are completed save sets (for example, exclude save sets that are aborted or not yet complete) and write this list to the

2010 EMC Proven Professional Knowledge Sharing 25 mminfo.txt file. The nsrclone command will read the list of save set ids from the mminfo.txt file and begin cloning them sequentially.

Cloning by Volume

You can start clone operations by selecting volumes in the GUI or volume names from the command line. When starting a clone by volume, NetWorker will look at a volume and build a list of save sets that begin on that volume or are completely contained on that volume and clone all of those save sets. Since it is building a list of save sets to clone anyways, it is usually better to focus on one of the methods that allow you to select save sets on criteria related to the importance of the data rather than the pieces of media. There are more variations of switches that you can use with the mminfo and nsrclone commands to include or exclude save sets that make it more convenient to focus on selecting data to a clone based on save sets rather than volume names.

Recovering from a Cloned Save Set

If you would like NetWorker to recover from a clone copy of a save set and the original still exists (it hasn’t expired or been overwritten), the original save set or volume can be marked as ‘suspect.’ That will force NetWorker to skip over it when performing the recovery and look for a clone copy of that same save set. You can mark volumes ‘suspect’ easily in the Volumes list in the NetWorker GUI Media section. Both volumes and save sets can be changed to a status of ‘suspect’ by using the nsrmm command on the NetWorker server and using the lower-case O option (nsrmm –o).

Overview of Staging

Staging moves backup data from one media type to another. It can be used in conjunction with adv_file media type (local disk, SAN disk, or CIFS/NFS). Move the backup data to other media once certain criteria are met. When a staging process is started, NetWorker uses cloning to copy the save set to a new location and then the original copy is removed and moved to the new location. Like cloning, staging will require at least two devices. One is used to read the original backup and the second to create a copy at the new location. With staging, NetWorker can move the backup data to any type of pool, unlike cloning that requires a clone pool as a destination.

2010 EMC Proven Professional Knowledge Sharing 26

Staging data can be accomplished by using two different approaches, either through a Staging resource in the NetWorker GUI or through the nsrstage command from the command line.

Staging Resource

Confiugre a Staging resource in the NetWorker GUI if you are planning to stage from a file type device (file) or advanced file type device (adv_file). The resource is used to monitor the directory where the file or adv_file device exists and to automatically move data to another destination. You can configure the staging resource to use criteria such as amount of data written or length of time data has been stored to decide when to move data to the new destination as configured through the “Destination pool“ attribute. There are some attribute settings in the Staging resource labeled as “High water mark” and “Low water mark” that are used to decide when to begin a staging operation based on data storage. If the disk is more than the percentage specified in the “High water mark” value when a “File system check” is performed, NetWorker will kick off the staging operation to begin moving save sets to free up space for new backup data. This operation will run until the “Low water mark” setting is reached. If using an advanced file type device, the staging operation can occur for an existing save set even if a backup of a newer save set is still running to the device.

The “Max storage period” attribute in the staging resource is used to specify when to move data off the device based on a length of time. Once a save set has been on the file device longer than the “Max storage period,” NetWorker will move it to the selected “Destination pool.” This will free up space for new backups and can be started automatically once a save set has passed the Max storage period and the “Recover space interval” check has been performed. Backup data that has an expired retention policy will be removed when the “Recover space interval” check runs and this will also free up space for new backups.

Using nsrstage command

You will start the staging operation from the command line or script using the nsrstage command if you are using staging to move data from virtual tape to physical tape or from

2010 EMC Proven Professional Knowledge Sharing 27 physical tape to physical tape. Staging can also be useful if you need to move one or more save sets that may have been written to an incorrect piece of media.

Other ways to schedule scripts in NetWorker

There are some other unique settings in NetWorker that can be used to call scripts for scheduling. The “Backup command” attribute in a client resource is one. Any script that is called from here must have its name begin with nsr or save (for example, nsr_custom_script) and will be run instead of the save command.

The savpnpc command can be used to call a pre-script before the backup and a post- script after the backup. The script name to be called is configured in the groupname.res file on the client in the /nsr/res directory, wherever it is located.

The owner notification in the client resource and any of the Notification resources are other places that NetWorker calls commands or scripts that can be customized. The owner notification command will be called after a client is backed up. It is typically used to send an e- alert of what was backed up on a client. You can also customize the Notification resources to call needed commands such as sending e-mail alerts or other custom commands or scripts.

Conclusion

This article provides some insight on different ways to schedule operations in NetWorker including strategies for configuration in the NetWorker Management Console GUI as well as different ways to use commands and call commands that are in scripts. The goal of this article was to document methods that are not as clearly documented in the typical NetWorker references, and provide some insight for NetWorker administrators who may be new to the product. It is also useful for experienced NetWorker administrators who may be rethinking their backup strategy.

2010 EMC Proven Professional Knowledge Sharing 28