Veeam sees strong demand for immutable backups, and the Veeam Backup & Replication Hardened Repository is well positioned to provide that. Sometimes people believe that remote time attacks against Veeam Hardened Repository via network time protocol (NTP) are easy. That’s wrong! Even with the default settings of modern NTP clients, it’s complicated. The default settings either don’t allow large changes during runtime (NTPD) or they only allow changes to a running system in small steps (chrony). To attack Hardened Repository immutability retentions, the attacker needs to change time by days, weeks, or even months or years. With these small, incremental time changes enabled per default settings, such attacks would take months or years. However, if an attacker can send wrong time information during operating system boot or NTP client restart, there is a risk with the default settings.
Of course, the default settings can be improved to reduce attack vectors. The easiest way to prevent remote time attacks is using local time. Local time of physical server clocks are “good enough” for immutability (I’m only talking about immutability here). Immutability times are set in days, weeks, months or even longer. Remember that GFS restore points (weekly, monthly or yearly backups) are protected for the entire retention period. That means requirements on precisely correct time are low.
For those who are interested in setting up network time in a secure way, keep on reading. This article covers how to secure NTP clients “as good as possible”, while keeping in mind that NTP, per se, is a protocol that is prone to man-in-the-middle attacks. Network time security (NTS) is a relatively new protocol and solves man-in-the-middle problems. This article also describes how to set up an NTS infrastructure with chrony as an NTS client and server on Ubuntu Linux.
The Quick Fix: Use Server Hardware Clock Time
As mentioned in the introduction, it’s okay to use the local clock of a server as time source for Veeam Hardened Repository. The time will err slightly over time — but this is irrelevant, as the minimum immutable time is seven days, and clocks of “production-grade” servers are “good enough”. I’m not talking about 10-year-old servers with weak batteries that lose time on every boot!
Network Time Protocol
Of course, local time is not “good enough” for every customer. Security and compliance regulations can require synchronization with defined time sources. A customer that wants to run Veeam Hardened Repository with SEC 17a-4(f), FINRA 4511(c) and CFTC 1.31(c)-(d) compliance must use a secure time source (no protocol defined). DISA STIG for Ubuntu 20.04, for example, describes:
The Ubuntu operating system must, for networked systems, compare internal information system clocks at least every 24 hours with a server which is synchronized to one of the redundant United States Naval Observatory (USNO) time servers, or a time server designated for the appropriate DoD network (NIPRNet/SIPRNet), and/or the Global Positioning System (GPS).
The list of servers provided in the STIG details are all NTP servers. In some environments, a Veeam administrator wants to use internal NTP servers, as it’s the only available internal time protocol and opening outgoing ports on corporate firewalls involves a lot of paperwork. NTP is still the standard protocol for many customers. There’s also a huge list of public time servers on www.ntppool.org for different regions of the world, or one can simply use pool.ntp.org as source. For corporate networks, the NTP service running on Windows domain controllers is a popular time source.
As with many protocols from last century, there are security challenges associated with NTP. Let’s imagine a scenario where an attacker became administrator/root on all NTP servers that are configured, or the attacker can run man-in-the-middle attacks and is able to manipulate all NTP traffic. NTPD, with default settings, simply stops with a “time error” if the time difference is too large when compared with local time. This scenario is not a problem with chrony’s default settings (a modern NTP implementation and default on Red Hat and Red Hat derivates). Chrony runs the clock only slightly faster, which means it will take very long time before one can see a significant time change.
Let’s imagine the attacker has time to wait until the Hardened Repository server reboots (or the chrony time service gets restarted). The default settings will not help here, because, per default, chrony sets the clock “correctly” during startup. The next section describes how to configure the chrony NTP client against this type of attack.
Improve NTP Security With Chrony as Client on Ubuntu
Chrony is a popular NTP client and server application. It’s the default network time client on Red Hat derivates since version 8.0. On Ubuntu, the installation is done with one command. I also install gnutls-bin for checking the TLS connection later.
apt install chrony gnutls-bin
The installation of chrony removes systemd-timesyncd (the default NTP client on Ubuntu). The chrony website has many details regarding security-related options. There are two main settings; of which, one must be changed from its default state.
Edit /etc/defaults/chrony and add the “-R” option. The result looks like this:
DAEMON_OPTS="-R -F -1"
The “-R” option prevents chrony from doing large time steps during a restart of the service or a server-wide reboot. The second option is the “makestep” directive, which is set to a good value per default. The “makestep” directive defines how and how often the largest step the clock progresses. If an attacker, for example, wants to change the clock to weeks in the future, it will take months or years before the clock will change to that wrong time.
makestep 1 3
An attack against a Hardened Repository with the configuration above ends with the time being out of sync between NTP server and Hardened Repository. With “chronyc tracking”, one can see the time difference between NTP server and client.
As well, messages about that problem exist in /var/log/syslog:
chronyd[28782]: System clock wrong by 85466.423485 seconds
Now, let’s solve the man-in-the-middle attack problem by using the NTS protocol. Remember that this does not protect against situations where the attacker owns all time servers. The “-R” setting is also valid for NTS.
Network Time Security Client With Chrony on Ubuntu and Red Hat
NTS is a secure version of NTP and uses TLS. It’s a relatively new protocol, described in RFC 8915. There are relatively few public NTS servers available today. Cloudflare is likely the largest provider; there are other, additional sources in Germany, The Netherlands, Sweden and United States (list).
Chrony supports NTS beginning from version 4.0. That means Ubuntu 22.04 needs to be used for the following section. (CAUTION: Ubuntu 22.04 is only supported as Hardened Repository as of Veeam Backup & Replication v12.)
NTS requires TCP port 4460 outgoing, additionally to UDP port 123 for NTP. The configuration in /etc/chrony/chrony.conf needs to be changed to a server that supports NTS, and the NTS keyword must be added:
server nts.justadomain.win iburst nts
Restart chrony with “systemctl restart chrony”. Check with “chronyc -N authdata”, whether the mode is NTS and KeyID, Type and KLen have non-zero values. If that’s not the case, /var/log/syslog has more information.
If you see certificate errors in the logs, gnutls-cli can help to get a view independent from chrony. The following command must return “Handshake was completed”:
gnutls-cli -p 4460 --alpn=ntske/1 --logfile /dev/stderr your.server.tld < /dev/null > /dev/null
The configuration on Red Hat Enterprise Linux (RHEL) or other derivates is similar to Ubuntu. The configuration file is /etc/chrony.conf and “systemctl restart chronyd” is restarting chrony. Remember, RHEL 9 is only supported with Veeam Backup & Replication v12 as Hardened Repository.
If you want to host your own NTS service, the next section explains how to do that with chrony as time server on Ubuntu 22.04.
Network Time Security Server With Chrony on Ubuntu
Before we begin the chrony server configuration, we need TLS certificates. In the example below, I use Let’s Encrypt certificates that are managed by certbot. The installation of certbot is again one command:
apt install certbot
Then a certificate gets requested and the challenge solved via DNS TXT record. You must have permissions on the public DNS server configuration.
certbot --manual --preferred-challenges dns certonly -d nts.justadomain.win --staple-ocsp -m certbot@example.com --agree-tos
Requesting a certificate for nts.justadomain.win
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Please deploy a DNS TXT record under the name:
_acme-challenge.nts.justadomain.win.
with the following value:
ZI8HODh6B4n8naQGX4nRsHIqNu13KWjuBHIR5o6Ix7E
Certificate is saved at: /etc/letsencrypt/live/nts.justadomain.win/fullchain.pem
Key is saved at: /etc/letsencrypt/live/nts.justadomain.win/privkey.pem
Copy the certificate and private key file to a place where the “_chrony” user has access. I put them in /etc/chrony/certs/ and the permissions look like this:
root@ntpserver2204:/etc/chrony/certs# ls -la
-r-------- 1 _chrony root 5603 Oct 5 17:23 fullchain.pem
-r-------- 1 _chrony root 1704 Oct 5 17:26 privkey.pem
Finally, chrony needs to be configured to use the certificate and act as server. The configuration itself is done again in /etc/chrony/chony.conf:
ntsservercert /etc/chrony/certs/fullchain.pem
ntsserverkey /etc/chrony/certs/privkey.pem
allow 192.168.0.0/16 #define your client networks here
Restart chrony with “systemct restart chrony”. If clients connect with NTS, that information is visible with the “chronyc serverstats” command.
If the firewall is enabled on the NTS server, then incoming ports UDP 123 (NTP) and TCP 4460 (NTS-KE) need to be opened.
Summary
It’s easy to prevent network time attacks against Veeam Hardened Repository. The default settings of common NTP clients prevent attacks against an up-and-running Hardened Repository server. To protect against possible attacks during operating system startup or restart of the time service, one can protect by starting chrony with the “-R” flag and setting the makestep directive properly (use defaults or remove completely). If one wants to use a more secure network time protocol, then NTS is easy to set up.