#1 Global Leader in Data Resilience

Unable to collect files information from datastore in a timely manner due to high vCenter server load.

KB ID: 1653
Product: Veeam ONE | 6.0 | 6.5 | 7.0 | 8.0 | 9.0 | 9.5 | 10 | 11 | 12 | 12.1 | 12.2 | 12.3
Published: 2012-07-06
Last Modified: 2025-02-04
mailbox
Get weekly article updates
By subscribing, you are agreeing to have your personal information managed in accordance with the terms of Veeam's Privacy Notice.
This site is protected by hCaptcha and its Privacy Policy and Terms of Service apply except as noted in our Privacy Policy.

Cheers for trusting us with the spot in your mailbox!

Now you’re less likely to miss what’s been brewing in our knowledge base with this weekly digest

error icon

Oops! Something went wrong.

Please, try again later.

Challenge

Veeam ONE Reporter collects data from datastores using the SearchDatastoreSubFolders_Task tasks, which are pre-defined by vSphere. During that process, if a datastore becomes unavailable or experiences performance issues, Veeam ONE Reporter may be unable to complete data collection tasks against that datastore.

The collection job in Veeam ONE Reporter will finish with a "Warning" status. Additionally, the session log may contain the following entry:

There were errors while collecting VM files: Failed in retrieve some VM files because of the timeout is exceeded.

 

When this occurs, the following error messages appear in the log file:
C:\ProgramData\Veeam\Veeam ONE Reporter\Logs\Tasks\{hostname}.local.log

Failed to collect VM files: Datastore [{datastore_name}] timeout is exceeded
Unable to collect files information from datastore [{datastore_name}] in a timely manner due to high vCenter server load.

 

Cause

Veeam ONE Reporter uses a 900-second timeout to wait for 10 datastore search tasks (default number) to complete.

Any performance or access issues affect the entire collection process, and the collection job finishes with a "Warning" status.
 
The error messages above mean that some issues have occurred on datastores. Due to a lack of response, the datastore scan tasks have been marked as failed, and the data cannot be collected from some of the datastores.

The problem lies either with the datastore or the vSphere environment.

However, there are ways to work around these environmental performance issues by making the collection timeout within Veeam ONE Reporter high enough to allow all datastore tasks to be completed.

Note: This workaround does not fix the underlying environmental issue; it simply changes the SearchDatastoreSubFolders_Task task timeout to ensure adequate time for the datastore collection process to complete.

Solution

To successfully tune the product to complete your datastore search tasks, you must first identify a suitable timeout value for completing all the “SearchDatastoreSubFolders_Task” tasks, then apply registry changes and set the new timeout value.

Part 1: Identify Timeout Value Based on Environment

Using the PowerCLI script below to simulate the “SearchDatastoreSubFolders_Task” tasks and determine how long it takes to complete all mentioned datastore tasks.

Download the VMware vSphere PowerCLI package for free by the following link: https://developer.broadcom.com/powercli
Or install the module using the following command:

Install-Module -Name VMware.PowerCLI
Script Instructions
  1. Open PowerCLI and connect to the vCenter server using the following command:
    Replace <vcenter_name> with the vCenter's hostname, FQDN, or IP.
Connect-VIServer -Server <vcenter_name> -Credential (Get-Credential)
  1. With the connection to the vCenter server now established, navigate to the directory where you saved the script.
  2. Run the script and wait until it's finished, noting when it started and when it completes.

Note: This script does not directly reproduce the API calls that Veeam ONE Reporter uses but is a close approximation of the actions it takes.

Expand to View PowerCLI Script
# Display the current time and a message to indicate that datastores are being retrieved
Write-Host (Get-Date).TimeOfDay "Getting datastores..."

# Retrieve the list of all datastores
$datastores = Get-Datastore
# Display the current time and the count of datastores found
Write-Host (Get-Date).TimeOfDay "$($datastores.Count) datastores found."

# Create a FileQueryFlags object to specify which file details to retrieve
$fileQueryFlags = New-Object VMware.Vim.FileQueryFlags
$fileQueryFlags.FileSize = $true
$fileQueryFlags.FileType = $true
$fileQueryFlags.Modification = $true
$fileQueryFlags.FileOwner = $true

# Create a FileQuery object for searching files
$fileQuery1 = New-Object VMware.Vim.FileQuery

# Define a search specification for general file search
$searchSpec1 = New-Object VMware.Vim.HostDatastoreBrowserSearchSpec
$searchSpec1.details = $fileQueryFlags
$searchSpec1.sortFoldersFirst = $true
$searchSpec1.MatchPattern += "*.*" # Match all files

# Define a search specification for finding VMDK-related files
$searchSpec2 = New-Object VMware.Vim.HostDatastoreBrowserSearchSpec
$searchSpec2.details = $fileQueryFlags
$searchSpec2.sortFoldersFirst = $true
$searchSpec2.MatchPattern += "*delta.vmdk" # Match delta VMDK files
$searchSpec2.MatchPattern += "*flat.vmdk" # Match flat VMDK files

# Assign the file query to the general search specification
$searchSpec1.Query = $fileQuery1

# Iterate over each datastore
foreach ($datastore in $datastores) {
# Retrieve the view object of the datastore for detailed access
$datastoreView = $datastore | Get-View

# Check if the datastore is accessible
if ($datastore.Accessible -ne $true) {
# If not accessible, skip file collection and display a message
Write-Host (Get-Date).TimeOfDay "Skipping file collection because $($datastoreView.Summary.Name) ($($datastoreView.Summary.Datastore.Value)) is not accessible."
} else {
# Retrieve the view object for the datastore browser to perform searches
$datastoreBrowser = Get-View $datastoreView.Browser
# Construct the root path for searching
$rootPath = "[" + $datastoreView.Summary.Name + "]"

# Attempt to collect VMDK-related files
Write-Host (Get-Date).TimeOfDay "Trying to collect *.vmdk and *.flat files from $($datastoreView.Summary.Name) ($($datastoreView.Summary.Datastore.Value))."
$searchResult2 = $datastoreBrowser.SearchDatastoreSubFolders($rootPath, $searchSpec2)
# Display completion message with the count of files found
Write-Host (Get-Date).TimeOfDay "Completed. $($searchResult2.Count) files found on $($datastoreView.Summary.Name) ($($datastoreView.Summary.Datastore.Value))."

# Attempt to collect other files
Write-Host (Get-Date).TimeOfDay "Trying to collect other files from $($datastoreView.Summary.Name) ($($datastoreView.Summary.Datastore.Value))."
$searchResult1 = $datastoreBrowser.SearchDatastoreSubFolders($rootPath, $searchSpec1)
# Display completion message with the count of files found
Write-Host (Get-Date).TimeOfDay "Completed. $($searchResult1.Count) files found on $($datastoreView.Summary.Name) ($($datastoreView.Summary.Datastore.Value))."
}
}
If errors occur for particular datastores, it may indicate an underlying problem. Still, based on the output, you should see how long it takes to scan all available datastores.

Part 2: Set the Timeout for Veeam ONE Reporter

Assuming the script took longer than 900 seconds (15 minutes) to complete, adjust the GetVmSnapshotTimeout value to increase the timeout Veeam ONE Reporter uses to complete the entire datastore search process:

  1. Connect to the machine where Veeam ONE Reporter installed.
  2. Using Registry Editor (regedit) or the PowerShell script below, set the following registry value:

    Key Location: HKLM\SOFTWARE\Veeam\Reporter Enterprise
    Value Name: GetVmSnapshotTimeout
    Value Type: DWORD (32-Bit) Value
    Value Data(Dec): #

    Set # to be equal to or greater than time, in seconds, it takes to complete scanning all datastores based on the script's outputted start and end timestamps.

    PowerShell:
# Set value in seconds to be used for the GetVmSnapshotTimeout value. Shown is the default.
$time = 900
# Create or update the registry value
New-ItemProperty -Path 'HKLM:\SOFTWARE\Veeam\Reporter Enterprise\' -Name 'GetVmSnapshotTimeout' -Value $time -PropertyType DWORD -Force
To submit feedback regarding this article, please click this link: Send Article Feedback
To report a typo on this page, highlight the typo with your mouse and press CTRL + Enter.

Spelling error in text

This site is protected by hCaptcha and its Privacy Policy and Terms of Service apply except as noted in our Privacy Policy.
Thank you!

Thank you!

Your feedback has been received and will be reviewed.

Oops! Something went wrong.

Please, try again later.

You have selected too large block!

Please try select less.

KB Feedback/Suggestion

This form is only for KB Feedback/Suggestions, if you need help with the software open a support case

By submitting, you are agreeing to have your personal information managed in accordance with the terms of Veeam's Privacy Notice.
This site is protected by hCaptcha and its Privacy Policy and Terms of Service apply except as noted in our Privacy Policy.
Verify your email to continue your product download
We've sent a verification code to:
  • Incorrect verification code. Please try again.
An email with a verification code was just sent to
Didn't receive the code? Click to resend in sec
Didn't receive the code? Click to resend
Thank you!

Thank you!

Your feedback has been received and will be reviewed.

error icon

Oops! Something went wrong.

Please, try again later.