Restored Linux Machine Fails to Boot

Part 1: Prerequisites

• You must have a Linux LiveCD matching the distribution and release version of the restored Linux system that is compatible with the target system hardware. It can be either a distribution-supplied live/rescue CD or Custom Veeam Recovery Media generated on the machine prior to restore.
• You must be aware of the target system's disk and volume layout and the devices corresponding to the root filesystem and boot partitions. If this information is not readily available, you may use File-level Restore (with Veeam Recovery Media, Veeam Agent for Linux, or Veeam Backup&Replication) to mount the backup contents and inspect the /etc/fstab file.

Part 2: Boot from LiveCD

1. Boot from the distro's LiveCD or Custom Veeam Recovery Media.
2. Switch to the bash shell (elevate to root user if necessary).
3. Locate the volumes/partitions hosting the Linux VM's root and boot filesystems. You can use commands like lsblk (list hierarchy of block devices), lvdisplay (list logical volumes), etc.

Part 3: Mount Guest OS Filesystem and Chroot

1. Create a directory where the system root will be mounted.

Example:

2. Mount the device hosting the guest OS root filesystem inside the directory (/targetroot).

Example for sda3 partition:

3. Mount the guest OS's boot partition inside /targetroot/boot.

Example for sda2 boot partition:

4. If UEFI is in use: The guest OS's EFI system partition (/boot/efi) needs to be mounted to /targetroot/boot/efi.

Assuming EFI system partition on sda1:

5. Bind mount the pseudo filesystems inside target root - /dev, /proc, /sys, and /run:

6. Chroot into the mounted system:

Part 4: Rebuilding initramfs

Identifying and Correlating Kernel Versions to Initial Ramdisk Images

The initial ramdisk image (initramfs/initrd) is an image of an initial root filesystem used as a part of the Linux boot process. Initial ramdisk images are stored as files under /boot, named initramfs-<kernel version>.img or initrd-<kernel version>.

Example for Red Hat Enterprise Linux 8:

Initial ramdisk image files correspond to GRUB/GRUB2 boot menu entries - when a specific entry is selected in the boot menu, a corresponding initramfs image is loaded. The association between GRUB menu entries and initial ramdisk image files can be determined via grub.cfg file (generally located under /boot/grub or /boot/grub2) - look for menuentry records which denote GRUB boot menu entries.

Example from SUSE Linux Enterprise Server 15 SP1:

### BEGIN /etc/grub.d/10_linux ###
menuentry 'SLES 15-SP1'  --class sles --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-simple-bf547c2f-3d66-4be6-98c3-8efe845a940d' {
<...>
        echo    'Loading Linux 4.12.14-197.15-default ...'
        linuxefi /boot/vmlinuz-4.12.14-197.15-default root=UUID=bf547c2f-3d66-4be6-98c3-8efe845a940d  ${extra_cmdline} splash=silent resume=/dev/disk/by-path/pci-0000:03:00.0-scsi-0:0:0:0-part3 mitigations=auto quiet crashkernel=175M,high
        echo    'Loading initial ramdisk ...'
       initrdefi /boot/initrd-4.12.14-197.15-default
}

Distro Specific Initramfs Rebuild Instructions

Now that you have identified the initramfs that needs to be rebuilt, expand the section below matching the distro for information about rebuilding it.

Part 5: Unmount and Reboot

With the initramfs having been rebuilt, exit the chroot environment:

Outcome 1: Boot Fails to Emergency Shell as Root

The system fails to boot, and because a root account was enabled, it dumps the user into an emergency shell.

Example:

1. Enter the root user password to access the Emergency console.
2. Remove the /etc/lvm/devices/system.devices file.

3. Reboot.

Outcome 2: Boot Fails without Emergency Shell

The system fails to boot, but because the root account was disabled, it displays the message:

Cannot open access to console, the root account is locked.

Example:

1. Boot from the Veeam Recovery Media or any preferred recovery ISO.
2. Activate all volume groups with the command:

3. Review the available block devices using:

4. Make a directory where the root filesystem will be mounted:

5. Mount the root filesystem to the folder.
   ^{The root filesystem path may be different than the example shown here. Review the results from lsblk.}

6. Remove the /etc/lvm/devices/system.devices file.
   

7. Unmount the root filesystem.

8. Reboot the machine.