Service Provider guide on troubleshooting slow merge in Cloud Connect jobs

During synthetic operations, data processing is handled solely by a Cloud repository. This article provides troubleshooting recommendations which may help to correctly identify the cause of slowness, increase performance and shorten backup window.

1) Storage performance

The first thing to determine is a type of the backup chain. Below is the correlation between the backup method and IOPS:

Method	IOPS/Block
Active full	1 write I/O
Forward incremental	1 write I/O
Forever forward incremental	2 I/O (1 I/O read + 1 I/O write)
Synthetic full	2 I/O (1 I/O read + 1 I/O write)
Reverse incremental	3 I/O (1 I/O read + 2 I/O write)
Synthetic full with rollbacks	4 I/O (2 I/O read + 2 I/O write)

Synthetic operations cause a significant disk load; thus, it is recommended to monitor IOPS, latency and throughput when processing is performed. For Windows-based repository servers, storage performance can be tested independently with DiskSpd (KB2014).

Synthetic operations can be avoided:

• in Backup job by configuring it as a Forward Incremental with periodic Active Full backups
• in Backup Copy job by enabling the option Read the entire restore point from source backup instead of synthesizing it from increments

Note: Active Full strategy imposes a larger amount of data to be transferred between sites.
 

2) Cloud repository server load (RAM/CPU)

Merge puts a significant load on repository server, please make sure that it doesn’t run out of RAM/CPU during backup window.

It is recommended that the task limitation settings are defined using the following rule: 1 concurrently running task = 1 CPU core. For the RAM requirement, if the backup repository server is running multiple Veeam roles, please add up memory requirements of each individual role.

If overload is discovered, consider increasing available resources.
 

3) Max concurrent tasks for Tenant

If Cloud repository is configured as per-VM, the main setting which defines how many VMs in the job can be merged in parallel is the Max concurrent tasks (Tenant Settings > Bandwidth tab). Adding more tasks to a tenant increases the load on the Cloud repository, however, it allows merging more VMs simultaneously, which eventually shortens backup window.
 

4) Antivirus exclusions must be present on all Managed Servers

File-scanning and HIPS activities of antivirus software may significantly slow down or even terminate merge operation. Please consider adding exclusions on Cloud repository servers (KB1999).
 

5) Deduplicating Storage Appliances

As Deduplicating Storage Appliance devices are unable to dedupe encrypted data blocks, synthesized backup file size might be significantly bigger in size. It slows down merge performance and increases backup window size.

More information on limitations can be found here.
 

6) CIFS as a Cloud repository

Under heavy load, with lots of concurrent tasks running towards it, CIFS (SMB) generally offers the poorest merge performance. It may lead to various issues described in KB1735.

The machine which acts as a repository gateway server must be located as close to the backup repository as possible.

If NAS device is used, consider reconfiguring it as an iSCSI target or an NFS mount on a separate Linux server, depending on which type is supported by the device. iSCSI usually offers better performance and stability.

If Dell EMC Data Domain is presented as a CIFS share to Veeam and added as a Cloud Repository, consider switching it to DDBoost to avoid the rehydration penalty during synthetic operations (KB1956).
 

ReFS considerations

For Windows Server 2016 (or later) repositories, formatting volumes into ReFS, will allow utilizing Fast Clone technology, which drastically improves performance of the synthetic operations.

With a ReFS repository in place, mind the following:

• Latest Windows updates are installed on the repository server.
• Volumes are formatted into 64K.
• System requirements for Backup repository servers are met.

 If ReFS performance is still below expectations, please contact support for assistance.