#1 Global Leader in Data Resilience
#1 Global Leader in Data Resilience
BACK TO KB LIST

VSS Timeout When Backing up Exchange VM

KB ID: 1680
Product: Veeam Backup & Replication | 6.1 | 6.5 | 7.0 | 8.0 | 9.0 | 9.5 | 10 | 11 | 12 | 12.1
Published: 2012-09-28
Last Modified: 2024-07-10
mailbox
Get weekly article updates
By subscribing, you are agreeing to have your personal information managed in accordance with the terms of Veeam's Privacy Notice.

Cheers for trusting us with the spot in your mailbox!

Now you’re less likely to miss what’s been brewing in our knowledge base with this weekly digest

error icon

Oops! Something went wrong.

Please, try again later.

Article Applicability

This article was initially created for Veeam Backup & Replication 6.1.
While the article remains technically accurate regarding the behavior of Microsoft VSS and Veeam's interactions with VSS, the issue discussed rarely occurs thanks to the improvements made in Veeam Backup & Replication version 8.

Veeam Backup & Replication 8 introduced Persistent VSS Snapshots, which significantly diminished the occurrences of VSS Timeouts during VSS preparation of Exchange servers.

However, some customer may still be affected by these timeouts if their Exchange server does not meet the requirements for Persistent VSS Snapshots to be utilized. Review Limitations for Persistent VSS Snapshot Technology for more information.

Challenge

The backup of an Exchange server VM fails with:

Unfreeze error:[Backup job failed]
Cannot create shadow copy of the volumes containing writer’s data
A VSS critical writer has failed. Writer name: [Microsoft Exchange Writer]. Class ID: [{76fe1ac4-15f7-4bcd-987e-8e1acb462fb7}]. Instance ID: [{0db23250-4d1e-42c1-8d14-2be32f448184}]. Writer's state: [VSS_WS_FAILED_AT_FREEZE]. Error code: [0x800423f2].]

If you run the command ‘vssadmin list writers’ on the Exchange server after the job fails, typically you will see an Exchange Writer has failed because of a timeout error (error code 9).

Cause 

VSSControl: Failed to freeze guest, wait timeout

This "wait timeout" refers to the limit imposed by Microsoft VSS on the maximum duration VSS writers may be frozen. This timeout is not configurable. Veeam Backup & Replication (VBR) uses VSS to freeze the VSS writers immediately before creating the VMware snapshot and then sends the thaw command as soon as snapshot creation is complete. Microsoft VSS will only hold a freeze on the writers for up to 60 seconds (20 for Exchange) and will automatically unfreeze (thaw) the writers once that timeout is met. This means that the following steps must all be able to complete within that timeframe:

  1. Microsoft VSS completes the requested VSS Writer freeze task and notifies VBR. (Freeze Timeout timer starts)
  2. VBR verifies the writers are frozen.1
  3. VBR uses VIM API to send a snapshot creation request to vSphere.2
  4. The ESXi host creates a snapshot on the VM.
  5. vSphere notifies VBR that the snapshot has been created.2
  6. VBR sends the Thaw request to Microsoft VSS within the Guest OS.1
  7. Microsoft VSS processes the Thaw request, resuming VSS writers’ I/O.

1 If a network connection to the guest OS is unavailable, VIX API will be used, introducing additional latency.
2 These steps should usually be near-instantaneous, but the delay may be significant if the vCenter is heavily loaded or has a high latency to the ESXi hosts.

Solution

Persistent VSS Snapshots

When using Veeam Backup & Replication version 8 or higher, the software will automatically switch to using Persistent VSS Snapshots with Exchange servers, which mitigate issues associated with the 20-second freeze limitation.

Review the Limitations for Persistent VSS Snapshot Technology to ensure the Persistent VSS Snapshots feature can function.

If Persistent VSS Snapshots cannot be used, you will need to determine why the tasks that must be performed within the Microsoft VSS freeze timeout window could not occur in a timely manner. This issue is an infrastructure issue that can be difficult to narrow down. The following is a comprehensive list of solutions that customers have used to resolve the issue:

  • First, please make sure you can create a Windows backup of the VM using VSS. This will prove that the issue isn’t explicitly VSS-related but is instead a combination of VSS and VMware snapshot technology.
  • Ensure that you have no other backup vendor agents on the server you are backing up, and if you do, uninstall them. If you need to do VSS operations on a guest OS, you should do this with only one backup product. Note that Veeam Backup & Replication uses Microsoft VSS, while other backup solutions may use their own VSS providers/writers; therefore, the successful capability of another backup solution does not necessarily indicate that the issue is with Veeam Backup & Replication.
  • Reboot the Exchange Server.
  • ESX(i) host not having enough resources.
  • VMware snapshot creation takes longer than 20 seconds (hardcoded Exchange VSS Writer timeout)
  • The Exchange freeze is too I/O intensive on the back-end storage, which may necessitate that the backup time and/or the datastore on which the Exchange server is located may need to be modified.
  • COM+ Event System Service may need to be restarted. Root cause unknown. In some cases, customers have scripted this service to restart before backup.
  • The latency between VC and Hosts can cause backing up through the host directly to produce successful VSS backups, whereas going through the VC causes freeze issues.
  • If Veeam Backup & Replication does not have direct network communication to Exchange, as a test, build a VM that is able to directly communicate with both Veeam Backup & Replication and Exchange and assign that machine as a Guest Interaction Proxy.
  • Ensure there is no snapshot on the Exchange VM before the backup starts, as that could cause additional storage I/O.
  • The exchange server may need additional resources (CPU/RAM) if taxed during the unfreeze.
  • One thing that is extremely important if you are attempting to use "connectionless mode" for VSS (i.e. if there is a firewall and thus we rely on the VIX API to communicate) is that you must make sure that the account being used for Application-Aware Processing is either the "built-in" local administrator or the "built-in" domain administrator (i.e. it must have a "well-known" SID ending in 500), other local or domain administrator accounts will not work. (See: KB1788)

 

To submit feedback regarding this article, please click this link: Send Article Feedback
To report a typo on this page, highlight the typo with your mouse and press CTRL + Enter.

Spelling error in text

This site is protected by hCaptcha and its Privacy Policy and Terms of Service apply except as noted in our Privacy Policy.
Thank you!

Thank you!

Your feedback has been received and will be reviewed.

Oops! Something went wrong.

Please, try again later.

You have selected too large block!

Please try select less.

KB Feedback/Suggestion

This form is only for KB Feedback/Suggestions, if you need help with the software open a support case
By submitting, you are agreeing to have your personal information managed in accordance with the terms of Veeam's Privacy Notice.
This site is protected by hCaptcha and its Privacy Policy and Terms of Service apply except as noted in our Privacy Policy.
Verify your email to continue your product download
We've sent a verification code to:
  • Incorrect verification code. Please try again.
An email with a verification code was just sent to
Didn't receive the code? Click to resend in sec
Didn't receive the code? Click to resend
Start using Veeam:
Download the product
&
Activate the license key
Thank you!

Thank you!

Your feedback has been received and will be reviewed.

error icon

Oops! Something went wrong.

Please, try again later.