Installing DISA STIG Hardened Red Hat Enterprise Linux for Veeam Hardened Repository

Since I wrote my blog post about how to install and harden Ubuntu Linux for Veeam Hardened Repository about a year ago, I get questions on how to do the same with Red Hat Enterprise Linux (RHEL). The reason why this blog post took so long to write is that Veeam Backup & Replication fully supports DISA STIG hardened RHEL systems only with the most recent update (version 12.1.2). Before this update, we had to do manual workarounds (see KB4250).

Before we start, let’s recap the requirements:

• A RAID 1 for the operating system on SSDs with at least 100 GB disk space.
• A RAID 6/60 for backup data.
• Static IP address, gateway, and DNS settings.
• Recommended: Redundant network connection.
• Switchport: Untagged.
• UEFI Secure Boot is enabled.

RHEL Installation

With all environmental preparations done, we can now install Red Hat Enterprise Linux 9. In this example, I show RHEL 9.4, but it’s very similar to other 9.x versions (and even 8.x).

The first step is to select your language. I recommend choosing English since it makes troubleshooting easier.

Now we are at the main configuration screen. Here we can select all installation options before starting the installation process. In the localization settings, you can add additional languages, keyboard layouts, and change your time zone if needed.

Configure Networking

The first thing we need to do is configure the network to be able register with Red Hat later.

Click on “Network & Host Name”

First up, create a “bond” by clicking 1 and 2 as shown below. The wizard also has “team” as an option but go for “bond” since “team” is deprecated in RHEL 9.

Give the bond a “connection name” and an “interface” name and press “add” in the “bonded connections” section.

The connection type is “Ethernet”. If the options are greyed out, move it with the mouse to make the options clickable. Then add your device in the dropdown menu and repeat for the other network card.

Once all ports are added, the mode needs to be set. There are different options depending on your needs and the capabilities of your network switch. The default selection “Round robin”/“balance-rr” requires an EtherChannel (or similar, depending on the switch vendor) without LACP. If your network team prefers LACP, then choose 802.3ad. For situations where the switch cannot be configured to EtherChannel with/without LACP, an active-backup policy can be used. This ensures high availability on the network side without load balancing between the network links. The result should look like this:

Before you click “Save”, switch to the IPv4 Settings (or IPv6 if used) and configure your desired IP address and DNS options.

I recommend static IP addressing to avoid challenges in case the DHCP service is unavailable. A few seconds after you saved the settings above, the IP should be reachable. I recommend testing the connection with a simple ping.

Before pressing “done” (3) at the top left of the wizard, you can set the hostname (1) at the bottom left and apply with (2).

Register with Red Hat

In the next step, we need to connect to Red Hat. For individuals that don’t need full support from Red Hat, the No-Cost Red Hat Enterprise Linux Individual Developer Subscription can be used. These free developer licenses are not for production though.

Disk Partitioning

Next, your disk needs to be partitioned. DISA STIGs require a specific partitioning scheme, and this scheme needs to be followed (sizes can be adjusted). Otherwise, the next step (i.e., selection of the DISA STIG security profile) will fail.

My server has two volumes (i.e., disk devices) where the small one is used for the OS and the larger one for your backup data. In Linux, there are only file system paths. Everything is mounted to mountpoints inside the root (“/”) file system and there are no drive letters as in Windows (C:\, D:\ etc.). That means you cannot run “out of drive letters” after 26 drives, but it also means the partitioning scheme will look a bit different than on Windows.

Select the disks (all disks in most cases, except if your server has some SD cards or other devices attached), select the “custom” storage configuration, and click “done”.

This opens a new wizard where we’ll switch the dropdown menu to “Standard Partition” and add new partitions by clicking the “+” (1) button. Type in the “mount point” and size (2 and 3).

We select “Standard Partition” because LVM (like Dynamic Disks or Storage Spaces in Windows) is too complex for this article. LVM would be more flexible, but in most cases, this flexibility is not needed.

Next, we need to ensure that the partition lands on the right physical disk. For that, click the “Modify” button in the “Device(s)” section and ensure it goes to the smaller disk.

With the first partition done, continue with the others in the table below. For a 100 GB disk, the partitioning scheme in the table below can be used. For larger disks, I recommend increasing /home, /var/log and /tmp since these are locations where data can grow to a certain point.

Now add the data disk. The mount point is /mnt/backup. If you have multiple disks, then you can add numbers or better names for each disk. Technically the mount point path is irrelevant, but /mnt/ is a common path to mount disks.

The final result should look like this:

Press the “Done” button at the top left and confirm that everything gets deleted/formatted.

Harden System by Applying DISA STIG Security Profile

Hardening according to DISA STIG standard is easy with RHEL. Just go to “Security Profile” and select the DISA STIG profile. If you get any errors here, then the disk partitioning from the previous section is wrong and needs to be fixed.

Select Software

Next, we’ll select the software that needs to be installed by clicking the “Software Selection” and going for Minimal Install:

Create User Account

Finally, we’ll configure a user that will be used to install the Hardened Repository role. Make the user an administrator, since the root account will stay disabled. 

After all options are set, the installation routine can do its job (just click “begin installation” on the bottom right of the main installation screen). After everything is finished, reboot the machine. After reboot, this screen will welcome you:

For simplicity, I recommend connecting remotely with SSH. Veeam Backup & Replication includes the Putty SSH client (main menu/burger menu -> console -> Putty) or you can use any SSH client. Technically you can also log directly into the console. However, once you type these passwords in manually, you will notice that copy and pasting into Putty or any other SSH client makes life much easier! 😊

Next step: Log into the machine with the “veeamrepo” user. Follow the instructions to change your password if needed. Passwords need to be complex according to DISA STIG requirements.

Set Permissions for your Repository Path

During installation, we decided that our backups will be stored in /mnt/backup. Right now, that path is owned by “root” and if we were to connect Veeam Backup & Replication, then we would not have write permissions to store backups. We also want to prevent everyone except “veeamrepo” from being able to read the backups (even if they may be encrypted from Veeam’s side).

To achieve that, run the following commands:

sudo chown veeamrepo:veeamrepo /mnt/backup/
sudo chmod 700 /mnt/backup/

With that, you can now use the machine as a Hardened Repository.

Add Hardened Repository Role

Adding the Hardened Repository role is the same as in other scenarios. There’s nothing special here, so I’ll refer to the user guide for a step-by-step description.

Configure User Accounts

The “veeamrepo” user account that was created in the installation process has full root (i.e., administrative) permissions since it is member of the wheel group. sudo is the command that allows you to run a command as root (i.e., administrator on Linux). Every member of the wheel group can run every command with sudo in RHEL.

During Hardened Repository installation, “sudo” or “su” permissions are required. After installation, “sudo” or “su” is not required anymore. “su” (substitute user) in this context means that a user can enter the root password to become root. We kept the root account disabled during installation, which means, you cannot use “su” to switch to the root user.

Here comes the quirks: When removing the “veeamrepo” user from the wheel group, there is no usable administrative/root user available anymore — not even a simple shutdown or reboot command would be possible.
There are different options to deal with this:

1. Set a password for the root user. With the default settings, the root cannot log in with password via SSH. This option adds some risk for the local console if an attacker knows that the root has a password. The attacker still must know (or brute force) the root password, but protection of the root password is required regardless.
2. After installing the Hardened Repository, there is no need for sudo with “veeamrepo”. You can remove the “veeamrepo” user from the wheel group and move on without any root permissions. If a root login is required for troubleshooting, then you can boot into single user mode (with a local console login required) and perform troubleshooting. With only minimal installation, this type of troubleshooting should never be required. Bigger issues can be solved by re-installing the whole operating system exactly as before once again. Since the backup data is located on a second RAID-set/volume, the backup data will stay where it is and only the OS RAID-set/volume will get re-formatted.
3. Option 2 can be enhanced and some sudo commands can be allowed (e.g., reboot or shutdown).
4. An additional administrative user can be created. In the end, that user should have the same permissions as root. The only advantage would be that an attacker could guess the username and password here. Compared to option 1, this is a bit of “security through obscurity”, which is considered a bad practice.
5. One can configure multi-factor authentication for scenario 1 or 4 as well. This adds complexity however and will not be covered in this blog post.

For a good balance of security and complexity, Veeam recommends going with option 3: Adding a few convenience commands to the /etc/sudoers file and removing the “veeamrepo” user from the wheel group after installing the Hardened Repository role.

“visudo” is the command to edit the /etc/sudoers file. “visudo” ensures that the syntax is correct. With broken syntax, the sudo command immediately stops working. So be careful and only edit the /etc/sudoers file with “visudo”. Log in as “veeamrepo” user and execute the command:

sudo visudo

After entering your password, the vi editor will open. If you never used vi before, then this can be a bit of a challenge. In case anything goes wrong, exit without saving by typing “:q!”

Go to the last line (scroll down or press “shift+g”) and go with the cursor or end key to the end of the line and press the “a” button (for “append”). The vi editor now indicates that it is in “INSERT” mode (1 in the screenshot below). Press “enter” and add the following lines to allow reboot and shutdown:

veeamrepo ALL = (root) NOEXEC: /usr/sbin/reboot
veeamrepo ALL = (root) NOEXEC: /usr/sbin/shutdown

The result should look like this:

Exit the vi editor by pressing “ESC” (to exist the “insert” mode) and “:wq” (write and quit)

Alternatively, you can also edit the file directly by copying and pasting the commands below. Note that the username “veeamrepo” needs to be adjusted if you chose another username.

sudo bash -c "echo 'veeamrepo ALL = (root) NOEXEC: /usr/sbin/reboot' >> /etc/sudoers"
sudo bash -c "echo 'veeamrepo ALL = (root) NOEXEC: /usr/sbin/shutdown' >> /etc/sudoers"

Configure Automatic Updates

To ensure your security updates are installed in a timely fashion, we enabled automatic updates. First, the “dnf-automatic” tool needs to be installed with the following command:

sudo dnf install dnf-automatic -y

Then, we can edit the configuration file with the following command to only install security updates:

sudo vi /etc/dnf/automatic.conf

The following changes need to be done:

• 5th line: and change the “upgrade_type” to “security”
• 22nd line: change apply_updates = yes

Press the “i” (insert) or “a” (append) key to switch vi to “insert mode” and edit accordingly. Save and exit by pressing “ESC” and then “:wq” again.

Enable the download timers with the following commands:

sudo systemctl enable dnf-automatic-download.timer
sudo systemctl start dnf-automatic-download.timer
sudo systemctl enable dnf-automatic-install.timer
sudo systemctl start dnf-automatic-install.timer

That’s it! Security updates will now be installed automatically.

Improving Network Time Protocol (NTP) Security

In this blog post I described possible attack vectors for the network time protocol. To prevent a scenario where time servers are manipulated/a man-in-the-middle attack happens in a way that affects Hardened Repository reboots, we set the “-R” option in /etc/sysconfig/chronyd (sudo vi /etc/sysconfig/chronyd).

The result should look like this:

# Command-line options for chronyd
OPTIONS="-R -F 2"

Then restart the chronyd:

sudo systemctl restart chronyd

Locking it Down

After the Hardened Repository role is installed, you can lock down the system by disabling SSH and removing the “veeamrepo” user from the wheel group. This way, even if somebody finds out the password for the “veeamrepo” user and gets access to the local console (or SSH, if enabled), they still cannot do any harm. This is because they cannot elevate to root via a “su” or “sudo” command.

To get access to the system after this kind of lockdown, either boot a live Linux system or boot into single-user mode. If you don’t want to lockdown the system that hard, then can create an additional administrative user and remove “veeamrepo” from the wheel group. This would be option 4 from the list above and is described in the “Create Admin Account” section below.

The following three commands disable and stop the SSH service, remove the “veeamrepo” user from the wheel group, and performs a final reboot. If you are logged in with SSH, then this connection will stay active until you disconnect/reboot.

sudo systemctl disable sshd
sudo systemctl stop sshd
sudo gpasswd -d veeamrepo wheel
sudo reboot

Create Admin Account

Please ignore this section if you want to go with option three, as described above. Creating an additional admin user is done with the following commands:

sudo useradd -G "wheel" admin
sudo passwd admin

After the Hardened Repository role is installed and the admin account is created, the “veeamrepo” user can be removed from the wheel group.

sudo gpasswd -d veeamrepo wheel

Summary

You have successfully set up Red Hat Enterprise Linux 9 to act as a Veeam Hardened Repository with applied DISA STIG security profile. If you applied the “Locking it Down” section, then even you won’t have root access anymore. That said, with console access, you can gain full permissions again of course – it’s your hardware 😊. Only non-privileged Veeam services can accept incoming network connections.