In the sage words of pop culture philosopher Camila Cabello, “I have questions.” When it comes to the behavior of backup operations, there are sometimes more questions than answers. A big question and pain point I have observed out in the field were intermittent large incremental backup points that seemed to have no reason. Here, we will work to uncover the data changes at the file-level behind incremental backup runs with Veeam Backup & Replication using a Windows Utility I built and am sharing with you.
Identifying file-level changes between restore points
Over time, daily incremental backup points are fairly consistent and reasonably linear in terms of storage consumption. However, outliers inevitably appear, which makes one wonder what exactly changed. What is different about this backup cycle and most importantly, is there any cause for concern? For instance, when an increment appears that is 5x larger than usual for a critical machine.
One place we can start to identify problems is with Veeam ONE. Veeam ONE tracks the underlying backup change rates that generate “suspicious incremental backup size” alerts. This alert has been triggered below in my lab. For more information on how to configure this alarm check out this [Help Center]( https://helpcenter.veeam.com/docs/one/alarms/backup_alarms_events.html?ver=110) document.
Even without Veeam ONE, a close investigation of repository file sizes will reveal outsized incremental point(s). In the case below my domain controller, where I could see the larger incremental point on disk. Obviously, changes in the OS filesystem have produced an unusually large set of block-level changes than expected, but how do you go about determining if this is a normal or abnormal condition? Let’s see if we can find out if the underlying filesystem changes can be identified to determine whether this is a “normal” change, or if in fact something nefarious has occurred.
Restore point compare utility
What I ultimately wanted to accomplish in these cases was to automatically compare the outlier backup increment against a “normal” point. To solve this problem, I developed a C# application which leverages the Veeam Data Integration API. This application mounts the points in question and then programmatically compares the mounted filesystems against each other to changes.
In addition to comparing backup points, the application can also compare a live workload filesystem state against a designated backup restore point in order to easily identify what percentage of the filesystem folders have changed. Each folder change is totaled and displayed under its respective parent directory. For files common to both restore points that have been changed in some way, the storage difference is calculated as point “A” file size minus point “B” file size.
This solution (and source code if you’re so inclined) is now available on VeeamHub. The installer can be downloaded from VeeamHub/veeam-restore-point-utility (setup.MSI). The application operates as follows.
1) After launching the application as administrator, a workload is selected from an active backup job:
2) Two restore points are then selected for comparison (along with mount credentials):
3) Once the backup mount operations have completed, a selective comparison of the full machine volume(s) or volume subfolders may be specified:
4) Finally, selecting “compare” will check the mounts both for filesystem level changes e.g., file size, encryption status, etc. and for files that are unique to the respective restore points.
To identify the filesystem folders with the highest change rates more easily, each difference is summed and rolled up to its respective parent directory. For files common to both restore points that have been changed in some way, the storage difference is calculated as point “A” file size minus point “B” file size.
In this example, I observed that the 3/22/2022 restore point referenced from repository contained 4.1GB of data not found in the prior point. This is primarily accounted for by the 4GB of shadow copy space added to the system volume information folder and Windows Defender activity which added an additional 278MB of net new storage to this restore point, which we can surmise accounted for the larger than expected incremental backup.
Until now, this has been an entirely manual proposition accomplished by procedures such as initiating a file-level restore (FLR) on the restore point in question. An FLR operation mounts the backup to the Veeam mount server and exposes it for filesystem-level inspection. Alternately, the Veeam Data Integration (PowerShell) API could be similarly employed to mount a backup for data reuse. While certainly workable, these approaches aren’t exactly user friendly, but there’s good news! Now there is a simple, automated solution to answer the question “what changed?” between restore points.
For a full description of the application’s system requirements and operating scenarios refer to the readme document at – https://github.com/VeeamHub/veeam-restore-point-utility. We hope you find this useful in your endeavors to fully understand how your backups operate and set your mind at ease when you too encounter an unusual incremental point in a backup chain.