- Veeam Support Knowledge Base
- VSS Timeout when backing up Exchange VM
VSS Timeout When Backing up Exchange VM
Cheers for trusting us with the spot in your mailbox!
Now you’re less likely to miss what’s been brewing in our knowledge base with this weekly digest
Oops! Something went wrong.
Please, try again later.
This article was initially created for Veeam Backup & Replication 6.1.
While the article remains technically accurate regarding the behavior of Microsoft VSS and Veeam's interactions with VSS, the issue discussed rarely occurs thanks to the improvements made in Veeam Backup & Replication version 8.
Veeam Backup & Replication 8 introduced Persistent VSS Snapshots, which significantly diminished the occurrences of VSS Timeouts during VSS preparation of Exchange servers.
However, some customer may still be affected by these timeouts if their Exchange server does not meet the requirements for Persistent VSS Snapshots to be utilized. Review Limitations for Persistent VSS Snapshot Technology for more information.
Challenge
The backup of an Exchange server VM fails with:
Unfreeze error:[Backup job failed]
Cannot create shadow copy of the volumes containing writer’s data
A VSS critical writer has failed. Writer name: [Microsoft Exchange Writer]. Class ID: [{76fe1ac4-15f7-4bcd-987e-8e1acb462fb7}]. Instance ID: [{0db23250-4d1e-42c1-8d14-2be32f448184}]. Writer's state: [VSS_WS_FAILED_AT_FREEZE]. Error code: [0x800423f2].]
If you run the command ‘vssadmin list writers’ on the Exchange server after the job fails, typically you will see an Exchange Writer has failed because of a timeout error (error code 9).
Cause
VSSControl: Failed to freeze guest, wait timeout
This "wait timeout" refers to the limit imposed by Microsoft VSS on the maximum duration VSS writers may be frozen. This timeout is not configurable. Veeam Backup & Replication (VBR) uses VSS to freeze the VSS writers immediately before creating the VMware snapshot and then sends the thaw command as soon as snapshot creation is complete. Microsoft VSS will only hold a freeze on the writers for up to 60 seconds (20 for Exchange) and will automatically unfreeze (thaw) the writers once that timeout is met. This means that the following steps must all be able to complete within that timeframe:
- Microsoft VSS completes the requested VSS Writer freeze task and notifies VBR. (Freeze Timeout timer starts)
- VBR verifies the writers are frozen.1
- VBR uses VIM API to send a snapshot creation request to vSphere.2
- The ESXi host creates a snapshot on the VM.
- vSphere notifies VBR that the snapshot has been created.2
- VBR sends the Thaw request to Microsoft VSS within the Guest OS.1
- Microsoft VSS processes the Thaw request, resuming VSS writers’ I/O.
1 If a network connection to the guest OS is unavailable, VIX API will be used, introducing additional latency.
2 These steps should usually be near-instantaneous, but the delay may be significant if the vCenter is heavily loaded or has a high latency to the ESXi hosts.
Solution
When using Veeam Backup & Replication version 8 or higher, the software will automatically switch to using Persistent VSS Snapshots with Exchange servers, which mitigate issues associated with the 20-second freeze limitation.
Review the Limitations for Persistent VSS Snapshot Technology to ensure the Persistent VSS Snapshots feature can function.
If Persistent VSS Snapshots cannot be used, you will need to determine why the tasks that must be performed within the Microsoft VSS freeze timeout window could not occur in a timely manner. This issue is an infrastructure issue that can be difficult to narrow down. The following is a comprehensive list of solutions that customers have used to resolve the issue:
- First, please make sure you can create a Windows backup of the VM using VSS. This will prove that the issue isn’t explicitly VSS-related but is instead a combination of VSS and VMware snapshot technology.
- Ensure that you have no other backup vendor agents on the server you are backing up, and if you do, uninstall them. If you need to do VSS operations on a guest OS, you should do this with only one backup product. Note that Veeam Backup & Replication uses Microsoft VSS, while other backup solutions may use their own VSS providers/writers; therefore, the successful capability of another backup solution does not necessarily indicate that the issue is with Veeam Backup & Replication.
- Reboot the Exchange Server.
- ESX(i) host not having enough resources.
- VMware snapshot creation takes longer than 20 seconds (hardcoded Exchange VSS Writer timeout)
- The Exchange freeze is too I/O intensive on the back-end storage, which may necessitate that the backup time and/or the datastore on which the Exchange server is located may need to be modified.
- COM+ Event System Service may need to be restarted. Root cause unknown. In some cases, customers have scripted this service to restart before backup.
- The latency between VC and Hosts can cause backing up through the host directly to produce successful VSS backups, whereas going through the VC causes freeze issues.
- If Veeam Backup & Replication does not have direct network communication to Exchange, as a test, build a VM that is able to directly communicate with both Veeam Backup & Replication and Exchange and assign that machine as a Guest Interaction Proxy.
- Ensure there is no snapshot on the Exchange VM before the backup starts, as that could cause additional storage I/O.
- The exchange server may need additional resources (CPU/RAM) if taxed during the unfreeze.
- One thing that is extremely important if you are attempting to use "connectionless mode" for VSS (i.e. if there is a firewall and thus we rely on the VIX API to communicate) is that you must make sure that the account being used for Application-Aware Processing is either the "built-in" local administrator or the "built-in" domain administrator (i.e. it must have a "well-known" SID ending in 500), other local or domain administrator accounts will not work. (See: KB1788)
To report a typo on this page, highlight the typo with your mouse and press CTRL + Enter.
Spelling error in text
Thank you!
Your feedback has been received and will be reviewed.
Oops! Something went wrong.
Please, try again later.
You have selected too large block!
Please try select less.
KB Feedback/Suggestion
This form is only for KB Feedback/Suggestions, if you need help with the software open a support case
Thank you!
Your feedback has been received and will be reviewed.
Oops! Something went wrong.
Please, try again later.