How to Manually Repair a VMware Replica created by Veeam

Before Beginning

1. Stop all replication jobs to the target location of the replica in question.
2. Manually check each target side proxy for stuck replica hotadded disks (VMDK disks belonging to the replica that remain attached to the target proxy after the job is completed). See KB1775 for details.
   ^{Consider switching the target proxies to use Network Transport mode to prevent stuck hotadded disks if it becomes a recurring issue.}

Gather Information

The following actions are performed with the replica VM using a vSphere Client.

3. Edit the Replica in vSphere Client.
4. Note which disk files correlate to each SCSI ID.
   Example:

   [Datastore1] DC01_replica\DC01-00000023.vmdk on SCSI0:0
   [Datastore1] DC01_replica\DC01_1-00000023.vmdk on SCSI0:1
   [Datastore2] DC01_replica\DC01-00000023.vmdk on SCSI0:2
   

Prepare the Replica

The following actions are performed with the replica VM using a vSphere Client.

5. Open the Snapshot Manager, and starting with the oldest snapshot, delete the snapshots one at a time. The goal is to get as much new information into the base disks as possible. At some point, though, there will be a snapshot that cannot be removed.
6. If any snapshots are left in the snapshot manager, try using the Delete All option in the snapshot manager.
7. Use the consolidate function to consolidate any orphaned snapshots.
   Note: It is expected that these steps to fail at some point. When you receive a failure, move on to the next step.

Preparing Veeam Backup & Replication

8. Within the Veeam console, under Replicas, find the replica that will be repaired and right-click it.
9. From the context menu, choose “Remove from replicas…” ("Remove from configuration")
   
   Note: After you use the “Remove from replicas…” ("Remove from configuration") function, it will remove the VM from the Replication job. You must re-add the VM back to the replication job manually.

Detach Snapshot Disks and Attach Base Disks

The following actions are performed with the replica VM using a vSphere Client.

10. Edit the replica, select each of the disks, and click remove. It will put a strikethrough the drive and show the word (removing).
11. After selecting all the disks for removal, press OK.
    
    When using the vSphere Web Client, if you run into a disk that displays “0” as the disk size, it won’t let you remove that disk from the VM. To remove this disk, you need to add a size to the disk. The number that you input here does not matter. We want to make sure the size of the disk no longer displays “0”. At this point, it will allow you to remove that disk.
12. Edit the replica again and reattach the base disks to the replica.
    • Choose to add an existing disk.
    • Navigate to the location of the base disks for the replica.
    • Attach them to the same SCSI nodes that were noted earlier.
      

Datastore Cleanup

13. Using the datastore browser, go to the folder of the replica.
14. Most likely, there will be many files; keep in mind that the only files that are required are:
    • VMX — Virtual machine configuration file
    • VMXF — Additional virtual machine configuration files
    • NVRAM — Virtual machine BIOS or EFI configuration
    • VMDK — Virtual machine data disk (one base VMDK per disk)
      _{Note: Technically, the .vmdk file is a descriptor file, and the actual data contents are stored in a -flat.vmdk (base disk) or -delta.vmdk (snapshot) file. These flat and delta files remain hidden as long as there is a matching .vmdk file in the same folder. Snapshot .vmdk files can typically be identified by their names, which include a hyphen and six digits before the .vmdk extension (e.g., DC01-000001.vmdk).}

    So, for example, here is a folder pre-cleanup post repair.
    
    
    We can remove the following files:
    
    
    Leaving the VMX, VMXF, NVRAM, and the VMDK for each disk. Removing the associated snapshot files that are no longer needed.

     

Test the replica

15. Create a snapshot on the replica.
16. Remove the snapshot.
17. If no error occurs, re-add the VM to the replication job and map the replica.
18. See if the job runs successfully.