Securing Veeam Hardened Repository

After selecting the right hardware for the Veeam Backup & Replication Hardened Repository and installing the Ubuntu Linux operating system, the next step is secure the operating system according to the DISA STIG (Defense Information Systems Agency Security Technical Implementation Guides) guidelines.

This blog post describes the hardening process for Ubuntu 20.04 free edition. For Ubuntu Pro users, there is an easier way. To keep it as simple as possible, Veeam provides a script that automates the setting of security options. The requirement for the script to work is such that Ubuntu is set up as described in the installing the Ubuntu Linux blog post.

Prerequisites

Before starting the hardening process, let’s recap the current state and adjust settings if needed.

• Hardware monitoring is enabled in the out-of-band management of the Hardened Repository server. Vendor-specific hardware monitoring tools are not covered in this blog post
• Minimal Ubuntu installation on an UEFI Secure boot-enabled physical system (virtual machines are unsupported/only for test installations)
• There is only one user account with full sudo permissions, and the root account is locked (no password)
• UDP port 123 outgoing is open on the firewall to the following servers (servers can be changed if needed. More details on NTP are available in this blog post and will be covered later in this article)
  • tick.usno.navy.mil
  • tock.usno.navy.mil
  • ntp2.usno.navy.mil

That’s it! Let’s start securing Veeam Hardened Repository.

Getting started

After installing Ubuntu as described in this blog post, there are two ways to connect or manage the Veeam Hardened Repository.

1. Direct login via the console (usually via a remote management console web interface)
2. SSH (secure shell)

The easiest way is using SSH, because it supports copy and paste. One popular SSH client is Putty, which is included in the Veeam Backup & Replication installation.

Log in to the Hardened Repository server with your credentials (SSH or console). Then download the script with:

wget https://vee.am/harddisa

Before running the script, it’s recommended that you understand what it does and what it does not do. The script applies DISA STIG security settings where applicable. That means the script implements things like password policies, audit rules, permissions or TCP security settings. The STIGs also have requirements for systems with graphical user interface or user management via a directory service. As the Hardened Repository is a “minimal installation” standalone machine, things like that don’t apply and are “not applicable”. A full list of settings is available in the Appendix.

This guide assumes that customers reading this guide are new to Linux or have only basic Linux knowledge. The hardening script skips settings that have notifications where someone should do something manually on the Hardened Repository server as root. That is because the final and manual hardening step after the Veeam hardening script will remove root and sudo permissions and disable SSH remote management. This is a hard step, but it addresses concerns around “I can log in as root and delete files.”

For advanced Linux users: Feel free to use the script to save time. If you are using automation with Ansible, there is material available for that as well.

Apply DISA STIG Requirements

You can also apply each hardening step of the DISA STIGs manually. On the DISA document library, search for Ubuntu, download the guidelines and open them with DISA STIG Viewer. Most rules have a code example to apply the guideline.

The script in this article handles the rules of the guidelines to make your life easier. Before running the hardening script, one needs to make sure that NTP traffic is allowed to the servers listed in the script (tick.usno.navy.mil, tock.usno.navy.mil and ntp2.usno.navy.mil). Alternatively, one can change the script to other time sources, or configure time manually and delete the time configuration section in the script. Apply the guidelines by running the script with the following command:

sudo bash harddisa

The output will look like this. Explanations on all the steps are provided later in this document.

After the hardening script finished, the Veeam Hardened Repository role can be installed.

Install Hardened Repository Role

Technically, installing the Hardened Repository role could also be done before doing the hardening. Before adding the Hardened Repository role, you need to configure the filesystem path where the backup data will be stored in future. In the “Installing Ubuntu Linux for Veeam Hardened Repository” blog, we defined /mnt/backup as the path for backup data storage. This path needs to be owned by the Hardened Repository user and we will limit permissions only to that user (veeamrepo in our case):

sudo chown –R veeamrepo:veeamrepo /mnt/backup
sudo chmod 700 /mnt/backup

The user guide describes how to install the Veeam Hardened Repository role. Please note that the ‘use “su” if “sudo” fails’ option cannot be used because the root account has no password/is locked.

After installation of the Hardened Repository, there is still one user with full “sudo” privileges. Let’s close this.

Locking It Down

There are a few steps where we decided to keep it a manual process instead of automating it with the hardening script. One of them (others are described in the appendix table) is UBTU-20-010009, which states: “Ubuntu operating systems when booted must require authentication upon booting into single-user and maintenance modes.”Single-user mode is a way to get into the system after locking down Hardened Repository as described in the next section.

Protecting single-user mode against unauthorized access is described in a separate blog post.

These following steps are outside the official DISA STIG hardening guides. These steps are the answer to customer concerns about the security of Hardened Repository and the criticism that one can “simply” log in as root and delete everything.

First, we will disable SSH remote management. The following command disables the SSH service, so that it does not start automatically during boot time.

sudo systemctl disable ssh

You can continue working with SSH until the next reboot. You can also stop the SSH service directly and the current SSH session will continue to work. But you can no longer reconnect after stopping the SSH service. The following command stops the SSH service:

sudo systemctl stop ssh 

The last commands above used “sudo” to give you root permissions. Root permissions are no longer needed to run Veeam Hardened Repository. We will only allow two commands: reboot and shutdown.

The overall concept of the following steps is also described in the Installing Ubuntu Linux for Veeam Hardened Repository blog post.

Replace the username with the username “veeamrepo” you are currently using:

sudo bash -c "echo 'veeamrepo ALL = (root) NOEXEC: /usr/sbin/reboot' >> /etc/sudoers"
sudo bash -c "echo 'veeamrepo ALL = (root) NOEXEC: /usr/sbin/shutdown' >> /etc/sudoers"

Now remove yourself from the sudo group. After executing the command below and next login, you lose your sudo permissions. To get full root/sudo permissions again, you must boot into “single-user mode”. Run this command only as last step!

sudo deluser veeamrepo sudo

With that, the configuration of the Hardened Repository itself is finished. But there are some more settings to become compliant with DISA STIGs.

Required Configurations on the Veeam Backup & Replication Side

There are DISA STIG requirements around encryption (see table below) that can be configured on the Hardened Repository directly or in Veeam Backup & Replication. The operating system data itself does not require protection. Only the backup data needs to be protected. One can protect the backup data also via disk encryption, but that increases complexity. It’s easier to manage encryption on the Veeam Backup & Replication side in the job settings.

Summary

As a “software only” vendor, Veeam offers customers flexibility regarding which hardware or operating system to use. On the other hand, customers ask for security best practices. Note the following blog series:

1. Selecting Hardware and Setting Up Environment for Veeam Hardened Repository
2. Installing Ubuntu Linux for Veeam Hardened Repository
3. Securing Veeam Hardened Repository (this blog post)

With this blog series, customers are prepared to achieve trusted immutability by applying globally recognized DISA STIG cybersecurity standards on top of Veeam best practices. If you are looking for pre-configured immutable hardware, where you don’t need to manage the hardware and operating system yourself, then there are several options:

• Object storage with immutability support. There are around 30 partner solutions available today.
• As secondary target (remember the 3-2-1 Rule) you can use the HPE StoreOnce deduplication appliance with immutability.
• If tape is an option, then WORM tape media can ensure that data can never be deleted again (while with immutability, data can be deleted after the immutability retention is over).
• “Veeam Hardened Repository” as a service is offered by regional service providers (availability varies depending on location).

Appendix

The table below shows DISA STIG requirements that must be applied manually on the server hardware or in Veeam Backup & Replication.

The table below shows the DISA STIG requirements that were implemented automatically by the script or are default settings.

The table below shows the DISA STIG requirements that are not applicable. Many of them are out of scope because there is only one (locked down) user account, no graphical user interface and this guide does not require having a mail server set up for notifications. FIPS compliance can also only be achieved by using Ubuntu Pro, while this guide refers to the free version.