VSS (Volume Shadow Copy Service) Troubleshooting

By Raja Lakshmanan R

Table of Contents

Volume Shadow Copy Service ...... 2 What is VSS?...... 3 What is Shadow Copy?...... 3 What is Provider? ...... 3 What is Requester? ...... 4 What is Writer? ...... 4 VSS Admin Tools ...... 4 Vssadmin ...... 4 Disk Shadow ...... 5 Microsoft Hotfixes for VSS ...... 6 Microsoft Documents which resolve specific VSS problems ...... 6 General Troubleshooting for VSS Problem ...... 6 Troubleshooting VSS related Problems seen in ARCserve D2D ...... 7 VSS related logs and debug settings in ARCserve D2D ...... 9 VSS Diagnostics Tools ...... 10 VSS Tracing Tools ...... 10 VSS related Knowledge Documents ...... 10

VSS (Volume Shadow Copy Service) Troubleshooting

Volume Shadow Copy Service

What is VSS?

The Volume Shadow Copy Service (VSS) is a set of COM APIs that implements a framework to allow volume backups to be performed while applications on a system continue to write to the volumes.

VSS provides a consistent interface that allows coordination between user applications that update data on disk (writers) and those that back up applications (requesters).

The VSS service starts on demand; therefore, for VSS operations to be successful, this service must be enabled.

Refer the links below for details about:-

 Volume Shadow Copy Service  How Volume Shadow Copy Service Works  How a Shadow Copy is created http://technet.microsoft.com/en-us/library/ee923636.aspx http://msdn.microsoft.com/en-us/library/windows/desktop/aa384649%28v=vs.85%29.aspx

What is Shadow Copy? A shadow copy is a snapshot of a volume that duplicates all of the data that is held on that volume at one well-defined instant in time. VSS identifies each shadow copy by a GUID.

A shadow copy set is a collection of shadow copies of various volumes all taken at the same time. VSS identifies each shadow copy set by a GUID.

What is Provider?

Providers manage running volumes and create the shadow copies of them on demand.

 Software Providers  System Provider  Hardware Providers

The system provider (MS Software Shadow Copy Provider) is supplied as a default part of a Windows.

Refer the following link for more details about VSS Providers, http://msdn.microsoft.com/en-us/library/windows/desktop/aa384594(v=vs.85).aspx

VSS (Volume Shadow Copy Service) Troubleshooting

What is Requester?

A requester is any application that uses the VSS API to create and manage shadow copies/shadow copy sets

 CA ARCserve Backup  CA ARCserve D2D  CA ARCserve RHA

A requester can:

 Initiate shadow copies.  Communicates with writers to gather information on the system.  Signal writers to prepare their data for backup.

What is Writer?

An application that coordinates its I/O operations with VSS shadow copy and shadow copy related operations (such as backups and restores) so that their data contained on the shadow copied volume is in a consistent state.

Writers are applications or services:

 Store persistent information in files on disk.  Provide the names and locations of these files to requesters by using the shadow copy interface.

During backup operations:

 Writers ensure their data is quiescent and stable.  Writers collaborate with restores by unlocking files when possible and indicating alternate locations when necessary.

VSS Admin Tools

Vssadmin

VssAdmin is used to create, delete, and list information about shadow copies. It can also be used to resize the shadow copy storage area ("diff area").

VssAdmin includes commands such as the following:

 create shadow: Creates a new shadow copy

 delete shadows: Deletes shadow copies

 Vssadmin list providers: Lists all registered VSS providers

 Vssadmin list writers: Lists all subscribed VSS writers

VSS (Volume Shadow Copy Service) Troubleshooting

 Vssadmin list shadows: Lists existing volume shadow copies

 Vssadmin list shadowstorage: Lists volume shadow copy storage associations

 Vssadmin add shadowstorage: Adds a volume shadow copy storage association.

 Vssadmin delete shadowstorage: Deletes volume shadow copy storage associations.

 Vssadmin resize shadowstorage: Changes the maximum size of the shadow copy storage area

VssAdmin can only be used to administer shadow copies that are created by the system software provider.

To view the command syntax for any of the commands, please refer the following technet article, http://technet.microsoft.com/en-us/library/cc754968.aspx

Disk Shadow

This tool applies to Windows Server 2008.

DiskShadow.exe is a tool that exposes the functionality offered by the Volume Shadow Copy Service (VSS).

This tool would be useful while troubleshooting VSS backup failures to isolate whether the problem is in CA ARCserve D2D or with VSS.

This applies to Windows Server 2008.

Refer the following technet article for details about DiskShadow.exe and its commands, http://technet.microsoft.com/en-us/library/cc772172%28WS.10%29.aspx

VShadow

This tool applies to Windows Server 2003 and Windows XP.

If the backup application fails to create a shadow copy, use the VShadow tool and see if a VSS shadow copy can be created.

This tool would be useful while troubleshooting VSS backup failures to isolate whether the problem is in CA ARCserve D2D or with VSS. VShadow tool download link for Windows 2003 Server and Windows XP, http://www.microsoft.com/en-us/download/details.aspx?id=23490

After installing the package, please go to the command prompt and then go to folder “C:\Program Files\Microsoft\VSSSDK72\TestApps\vshadow\bin\debug-server”, type “vshadow c: > c:\vshadowoutput.txt” and run it.

Refer the following Microsoft article for more Details about the VShadow tool and various command line options, http://msdn.microsoft.com/en-us/library/windows/desktop/bb530725(v=vs.85).aspx

VSS (Volume Shadow Copy Service) Troubleshooting

Microsoft Hotfixes for VSS

1) Install the VSS Rollup Package for Windows Server 2003 if it is not installed already. Volume Shadow Copy Service (VSS) update rollup package for Windows Server 2003 to resolve some VSS snapshot issues http://support.microsoft.com/default.aspx?scid=kb;EN-US;940349

2) This hotfix is for a VSS snapshot Data Corruption issue. Symptoms, Causes and Resolution are explained in detail in the Microsoft KB. Please read through them. http://support.microsoft.com/kb/976329

3) If calls to VSS management interface APIs are made while protected snapshot replications are occurring, the APIs encounter performance issues in Windows Server 2008 and in Windows Vista SP1. http://support.microsoft.com/default.aspx?scid=kb;en-us;956136&sd=rss&spid=11734

Microsoft Documents which resolve specific VSS problems

1) Error message when you run the "vssadmin list writers" command on a Windows Server 2003-based computer: "Error: 0x8000FFFF". http://support.microsoft.com/kb/940184

General Troubleshooting for VSS Problem

1) Reboot the server. Most of the times, a reboot corrects writer related problems.

2) Check if the Volume Shadow Copy service (VSS service) and Microsoft Software Shadow Copy Provider services are enabled. If they are disabled, then enable them.

3) Check the status of the VSS Writers and VSS Providers.

VSS Writers From the command prompt run the command, "vssadmin list writers". This should display the list of writers and their state. The writers should show a stable state.

VSS Providers From the command prompt run the command, "vssadmin list providers" to list the currently registered shadow copy providers. Make sure the "Microsoft Software Shadow Copy provider 1.0" System Provider is displayed.

a. If the "vssadmin list writers" command show any of the writers in a failed state, restart the service related to that writer; and then run the "vssadmin list writers" command to determine whether the writer returns to a stable state. b. Check with the Administrator before restarting the service related to the Application, as the Application might be in production and users accessing it. c. Also check for any errors or warnings related to the writer, search Microsoft Knowledge base or the Application Vendor Knowledge base. If the problem related to a specific writer continues, then contact Microsoft or the vendor of the writer for information about

VSS (Volume Shadow Copy Service) Troubleshooting

how to fix the errors.

4) If there is no sufficient space to create the shadow copy, the backup job can fail with the error message "Failed to create snapshot for selected volumes". This can also happen when the shadow copy could not grow due to insufficient space.

If the backup job fails, you can perform either of the following task: a. Free up some disk space on the volumes being backed up or increase the limit for the Shadow copies. b. Reconfigure the Volume Shadow Copy settings to save shadow copy to a volume with sufficient free disk space. Note: - When you change the location of the shadow copy storage area, all previous shadow copies are deleted.

5) If it is Windows 2003 Operating System,

a. Then install the VSS Rollup Package for Windows Server 2003 if it is not installed already. Volume Shadow Copy Service (VSS) update rollup package for Windows Server 2003 to resolve some VSS snapshot issues. Reboot the server. http://support.microsoft.com/default.aspx?scid=kb;EN-US;940349 b. If the “Vssadmin list writers” still has some writers which do not show a stable state or if the shadow copy could still not be taken, then re-register the VSS dll’ s as mentioned in the following Microsoft article. http://support.microsoft.com/kb/940032

6) If it is Windows 2003 Server or Windows XP, Use the VShadow tool and see if a VSS shadow copy can be created. Refer the following Microsoft article for more Details about the VShadow tool and various command line options, http://msdn.microsoft.com/en-us/library/windows/desktop/bb530725(v=vs.85).aspx

7) If it is Windows 2008 Server, Use the Disk Shadow tool and see if a VSS shadow copy can be created. Refer the following technet article for details about DiskShadow.exe and its commands, http://technet.microsoft.com/en-us/library/cc772172%28WS.10%29.aspx

8) If the problem continues with VShadow or Disk Shadow tool fails to create shadow copies, then engage Microsoft support. Microsoft might use VSS Diagnostic Tools to diagnose VSS problems and they might also enable VSS Tracing to collect debug traces for the VSS problem.

Troubleshooting VSS related Problems seen in ARCserve D2D

Problem 1:-

The following knowledge document helps in troubleshooting VSS related problems in Windows 2003 Server and Windows XP. http://arcserve-knowledgebase.com/index.php?View=entry&EntryID=2637

Details about the Problem: CA ARCserve D2D Backups would fail with VSS Snapshot related errors.

 Some VSS related errors seen in the CA ARCserve D2D Activity Log are:-

VSS (Volume Shadow Copy Service) Troubleshooting

a) The provider was unable to perform the request at this time. This can be a transient problem. Retry after 600 seconds. b) Error during the last asynchronous operation, please refer to Windows Event log for more details. (EC=[0x8000ffff], MSG=[E_UNEXPECTED]) c) Failed to create snapshot for selected volumes.  Vssadmin list writers command would show that some or all of the writers are not in a stable state.  The Windows Application and System event logs would have error events created by the Volsnap and VSS sources at the time of failure.

Problem 2:-

The following knowledge document helps in troubleshooting VSS related problems where CA ARCserve D2D backups fail which trying to take a snapshot or while performing the backup. http://arcserve-knowledgebase.com/index.php?View=entry&EntryID=4104

Details about the Problem:

 CA ARCserve D2D backups fail while trying to take a snapshot. The following error messages can be seen in the ARCserve D2D Activity Log, "The provider returned an unexpected error code. This can be a transient problem. Retry after 600 seconds." "Failed to create snapshot for selected volumes."

 ARCserve D2D backups would fail with the following error while performing the backup, Failed to read shadow copy of volume C:, the system error is [The system cannot find the file specified. (2)]. When the above error is seen in ARCserve D2D Activity Log, the following errors are seen in the Windows Event Logs,

Windows Application Event Log shows the following errors,

EventID ::: 8193 Event Log Type ::: Error Event Source ::: VSS Time ::: 11/12/2012 12:55:16 PM Volume Shadow Copy Service error: Unexpected error calling routine Cannot find anymore diff area candidates for volume \\?\Volume{985d7460-1ec5-11e2-a1fd-806e6f6e6963}\ [0]. hr = 0x8000ffff, Catastrophic failure

EventID ::: 12293 Event Log Type ::: Error Event Source ::: VSS Time ::: 11/12/2012 12:55:16 PM Volume Shadow Copy Service error: Error calling a routine on a Shadow Copy Provider {b5946137-7b9f-4925-af80-51abd60b20d5}. Routine details EndPrepareSnapshots({75bba9b0- cc67-4d87-91f4-ba662df0ec77}) [hr =0x80042302, A Volume Shadow Copy Service component encountered an unexpected error. Check the Application event log for more information.].

Problem 3:-

Backups fail with the following error message, Failed to read shadow copy of volume C:, the system error is [The system cannot find the file specified. (2)]. Failed to read shadow copy of volume D:, the system

VSS (Volume Shadow Copy Service) Troubleshooting error is [The device is not ready. (21)]. http://arcserve-knowledgebase.com/index.php?View=entry&EntryID=2751

Problem 4:-

Error Code 1450 "Insufficient system resources".

If there are errors with Error Code 1450 "Insufficient system resources" found in the ARCserve D2D Activity Log, Backup Log or AFStorxxxx.Log, then it could be due to high utilization of non-paged pool memory. This could happen if there are lots of VSS snapshots in the machine, the driver attaches to the snapshots even when they were created by other application. Driver attaches to the snapshots to monitor the changes happening on the snapshots; if the changes to the snapshots are not monitored it will result in a corrupted backup.

Please refer the following knowledge document for solution, http://arcserve-knowledgebase.com/index.php?View=entry&EntryID=3612

Please refer the following Microsoft article for more details about Non-Paged pool memory limitation, http://technet.microsoft.com/en-us/library/cc755419(v=ws.10).aspx

VSS related logs and debug settings in ARCserve D2D

 Following logs would be required by support to analyze VSS related problems,

1) Copy the X:\Program Files\CA\ARCserve D2D\Logs folder to another location and zip it. (Replace X:\ with the Drive Letter where CA ARCserve D2D is installed.) 2) Windows Application Event Log (Saved in Text format). 3) Windows System Event Log (Saved in Text format). 4) Windows System Information File (MSINFO32). Type "msinfo32" under Start->Run. This would launch the "System Information". Click on File- >Save, provide the name of the server as the "File Name" and save it as an ".NFO" file. 5) Output of the following VSS Commands. Run the following commands from the command prompt. (Replace x:\ with the drive letter where you want to create the output files) vssadmin list writers > x:\temp\outputwriters.txt vssadmin list providers > x:\temp\outputproviders.txt vssadmin list shadows > x:\temp\outputshadows.txt vssadmin list shadowstorage > x:\temp\outputshadowstorage.txt Collect the output files, after executing the above mentioned commands.

 Which are the ARCserve D2D logs that would have reference to the VSS Related errors?

X:\Program Files\CA\ARCserve D2D\Logs -Activity.Log -Backup-xxxxxxxx-xxxxxx-xxx.log - Debug log for backup. -ARCserve D2D VSS related log is VSSWrapperDll.Log (D2D VSS related binary is VSSWrapperDll.dll)

Log level for the VSSWrapperDll can be controlled by the below registry value [HKLM\SOFTWARE\CA\CA ARCserve D2D\Debug\VSSWrapperDll:DWORD] Set log level to 10 for D2D VSS troubleshooting.

VSS (Volume Shadow Copy Service) Troubleshooting

VSS Diagnostics Tools Vsdiagview and Vssagent are tools that can be used to troubleshoot VSS applications.

Refer the following link for more details about VSS Diagnostic Tools, http://msdn.microsoft.com/en-us/library/windows/desktop/aa384631(v=vs.85).aspx

VSS Tracing Tools

To collect tracing information for the VSS infrastructure, you can use the VssTrace tool, the Logman tool, or the Tracelog tool. VssTrace is available in the Microsoft Windows Software Development Kit (SDK) and can be used to trace VSS applications on Windows 7 and later versions of the Windows operating system.

Tracing Tools, to collect tracing information for the VSS infrastructure.

 VssTrace  Logman  Tracelog

Refer the following link for more details about VSS Tracing Tools, http://msdn.microsoft.com/en-us/library/windows/desktop/dd765233(v=vs.85).aspx#vsstrace_command- line_options http://support.microsoft.com/kb/887013?wa=wsignin1.0

VSS related Knowledge Documents

Unable to view the Microsoft Snapshot versions of the network shares. http://arcserve-knowledgebase.com/index.php?View=entry&EntryID=2012

Event ID 29, source volsnap, error logged into system event log. This happens while attempting to perform a backup on an Exchange server (with Exchange Granular Recovery). http://arcserve-knowledgebase.com/index.php?View=entry&EntryID=3988

D2D backup may fail on windows 2008 with VSS errors 8220/8193 when taking snapshot. http://arcserve-knowledgebase.com/index.php?View=entry&EntryID=2440

Microsoft Exchange databases are not backed up from a Windows 2003 SBS Server. The Microsoft Exchange VSS writer is not seen under CA ARCserve D2D recovery points. http://arcserve-knowledgebase.com/index.php?View=entry&EntryID=2024