Troubleshooting Signature-Based Firewalls

Data transfer is failing across a connection that is protected by an advanced firewall. The firewall uses signature-based detection. Relevant features may have names like antivirus, anti-spyware, intrusion prevention, or application control.

This may affect any make or model of a firewall with similar features, but support cases have been opened for:

• Check Point
• Cisco
• SonicWALL
• Fortinet appliances
• Palo Alto
• Sophos
• CrowdStrike
• Kaspersky

Any type of data transfer may be affected. This problem may manifest in a wide variety of error messages and failure patterns. It may appear random or consistent.

For the Veeam Backup & Replication product, this can affect all job types. A common failure pattern is for the transfer of specific VM's disks to fail at or around the same percentage of completion repeatedly.

Common error messages include:

An existing connection was forcibly closed by the remote host.

Unstable connection: unable to transmit data.

Depending on the job type and the version of Veeam Backup & Replication, connection failures may cause the job to fail immediately, or the connection may be retried several times. While the connection is retried, the job may appear to be frozen because it is unable to transmit data. For more information, see:
Resume on Disconnect
Backup Copy: Automatic Job Retries
WAN Acceleration: Data Transport on WAN Disconnect

 

With Veeam Backup for Microsoft 365 this issue may appear with an error message:

A blocking operation was interrupted by a call to WSACancelBlockingCall

Data transferred by Veeam software products can contain a potentially unlimited variety of data blocks. Because the traffic is compressed (and in most cases encrypted), data blocks analyzed by a firewall will be different from data as it exists in production. Over the long term, this approximates feeding random data into the signature-based threat detector: false positives are inevitable.

Data transfer is not actually random: a particular data block will always have the same signature after compression and encryption. If the source data does not change, the same blocks will be re-sent on every reconnect attempt and every retry of the job. In this case, the firewall will close the network connection every time a Veeam product attempts to transfer that data block because the firewall incorrectly detects a pattern of data within that block as matching the signature of a known threat.

It is advisable to begin by reviewing any logging available from the firewall to identify interactions and false detections that interfere with Veeam software data transmissions. Then, create exclusions for Veeam data traffic.

• In most cases for Veeam Backup & Replication, Veeam Agent for Microsoft Windows, and Veeam Agent for Linux, the relevant traffic will be between servers when data is transferred over the transport ports of 2500-3300 (TCP). This range can be configured for each managed server in the backup infrastructure settings.
  
• Cloud Connect Service Providers should create exclusions for data traffic sent to Cloud Gateways on port 6180 (TCP and UDP).
• For Veeam Backup for Microsoft 365, data transferred over port 443 (TCP) can be affected by this issue.
• For more information on port ranges, find the relevant product or component user guide in the Help Center and consult the Used Ports page.
• For information on how to configure exclusions on a specific firewall appliance, please contact the firewall vendor.

To isolate the firewall, temporarily disable all signature-based features in the firewall’s configuration. For best results, do this while data transfer appears frozen – traffic should resume in no more than a few minutes. In some cases, the firewall may allow you to disable specific sites or zones selectively; this can be useful as a solution, but it is not a good isolation step because such features are easily misconfigured.

Reset packets generated by firewall appliances can usually be distinguished from normal traffic by their IP time to live. For example, if most packets in a TCP stream have a TTL of 128, the reset packet that closes the stream has a TTL of 64, the connection was closed by a firewall.

Firewall features that block encrypted key exchange will block most WAN connections used by Veeam Backup & Replication.