When exporting snapshots to an external location such as an S3 Bucket or NFS file store, you might be concerned about the security of your backups in that location. If the S3 bucket or file storage were to be compromised, would the attacker be able to access sensitive data? The answer to this is easy: no.
When Kasten K10 by Veeam stores application data, it is always protected using AES-256-GCM symmetric encryption. This might be the case for snapshots that are exported to an external location, but also application metadata, configs and secrets that are stored in the Kasten K10 catalog.
Local snapshots are not stored (and therefore encrypted) by Kasten K10 itself. For example, if you are using the Amazon Elastic Kubernetes Service (EKS), your workload’s Persistent Volume Claims (PVC) will be handled by AWS EC2 Volumes. When Kasten K10 creates snapshots of these volumes, the content isn’t copied to another volume; instead, an EBS snapshot is created in AWS natively. You can find more information about EBS encryption here. For other platforms, you have to research how the storage class you are using is handling encryption.
But how does encryption in Kasten K10 work, how does it affect operations, and how can you configure it to your needs? These are the topics we’ll explore in this blog post.
How Kasten Encryption Works
To understand how encryption in Kasten K10 works, some concepts need to be introduced:
- Master Key: When you first install Kasten K10, the crypto service ( crypto-svc ) will generate the master key. The master key is the essential key for encrypting and decrypting backups. Even though it’s not used directly to encrypt backups, it can be used to derive the encryption keys. To protect the master key, Kasten K10 only stores it in encrypted form. The Passkey is used to encrypt and decrypt it. Then the encrypted master key can be written to the Kasten K10 catalog.
- Encryption Key: Kasten K10 uses symmetric encryption keys to encrypt your backups. Each policy has its own encryption key that is derived using the master key and the name of the policy. Kasten K10 never stores the encryption keys; instead, they’re derived on-demand.
- Passkey: When retrieving the master key (for example after a restart), the crypto service used the Passkey to decrypt the encrypted master key stored in the catalog. The same happens in reverse when storing a newly generated master key, then the Passkey is used to encrypt the master key.
However, the Passkey is not really a key itself. It’s just a Kubernetes API object representing the method used for decrypting/encrypting the master key.
By default, Kasten K10 creates a Passkey that should look something like this:
vault.kio.kasten.io/v1alpha1
kind: Passkey
metadata:
name: k10MasterKey
spec:
secret:
name: k10-cluster-passphrase
namespace: kasten-io
status:
inuse: true
state: Added to Catalog You can list the Kasten K10 Passkeys by running kubectl get passkeys . To get a specific Passkey, run kubectl get passkey k10MasterKey -o yaml . The property in use tells you that it’s the active Passkey.
The default Passkey is using a passphrase, but that’s not the only option. Right now Kasten K10 supports three different types of Passkeys:
-
- Passphrase: When using a passphrase, the Passkey references a secret in the Kubernetes cluster. Based on this secret, a cryptographic key is derived, and that key is used to encrypt and decrypt the master key. When installing Kasten Kq10 for the first time, a random passphrase will be generated; however, it is also possible to provide your own passphrase upfront or update the passphrase later.
- AWS KMS: If you are running on AWS, you can use its native Key Management Service (KMS) to protect the master key with your own Customer Managed Key (CMK). The master key is still stored in the Kasten K10 catalog, but it will be sent to KMS to encrypt or decrypt it. Backups are still encrypted locally within Kasten K10 using the encryption keys.
- HashiCorp Vault: Anyone using Vault can use the Transit Secrets Engine to protect the master key in a similar manner to KMS. The master key will also still be stored in the Kasten catalog but sent to your Vault installation for encryption and decryption.
Migration Key
For application mobility, for example, when migrating an application from one cluster to another, Kasten K10 uses the encryption key of the application as the migration key. The migration key is automatically included in the import data that you need to copy and paste when exporting and importing an application. Because the encryption key is unique for each policy and cryptographically derived from the master key, it only gives access to the backups created by that policy.
Including the migration key in the import data means you can easily move applications between clusters without needing to sync up the configuration of the underlying encryption mechanisms. You could easily move an application from a cluster with Kasten K10 configured to use Vault to another configured to use a passphrase.
Disaster Recovery Passphrase
The Kasten K10 Disaster Recovery (DR) feature allows recovering the Kasten K10 installation, even in case of complete cluster destruction.
For this use case, a separate mechanism is used: When enabling DR, a passphrase needs to be specified. The DR passphrase is then stored in the k10-dr-secret Kubernetes secret and used to encrypt the data before it’s stored in an external location. This data also includes the master key (now encrypted twice).
How Encryption Affects Operations
Since all snapshots are encrypted, you might be wondering what steps you need to take during operation to ensure backups can always be decrypted.
Disaster Recovery (DR)
Disaster recovery has to go through two levels of encryption:
- The DR passphrase helps decrypt the data stored in the external storage location. A list of artifacts gets downloaded, decrypted and stored in Kasten K10’s catalog. One of the artifacts is the still decrypted master key.
- The Passkey is then used to decrypt the master key. This means both the DR Passphrase (k10-dr-secret) and Passkey are important in a Kasten K10 DR scenario.
Depending on how you’ve configured Kasten K10, the secret for the Passkey may or may not be in the backup.
Particularly when configuring a custom Passkey passphrase with a secret that is outside of the kasten-io namespace, the Passkey will not be backed up by the DR feature. In any case, it makes sense to **store the Passkey passphrase in a secure location outside the Kubernetes cluster.
If you are using the KMS or Vault integration, you will still have the ability to generate a new access token on-demand.
Setup
When installing Kasten K10, you can either let it automatically generate a new random passphrase, or you can use the k10-cluster-passphrase secret to configure the Passkey upfront. Either way, Kasten K10 will generate a new master key.
Restarts
When the Kasten K10 crypto service restarts, it has to decrypt the master key again. To do so, it reads the ciphertext of the master key from the Kasten K10 catalog, then either decrypts it using the passphrase or sends it to your configured key management solution (KMS or Vault) for decryption.
Key Rotation
You can update the passphrase or keys used in KMS or Vault anytime. To do so, simply create a new Passkey with the usenow flag. Kasten then re-encrypts the master key using the new Passkey and stores the updated ciphertext. The old Passkey can now be deleted.
Since the master key is used to derive the encryption keys and the encrypted snapshots are immutable, the master key itself is never rotated.
Possible Configurations
Passkey Types
You can protect the master key by using a passphrase, a CMK in AWS KMS, or the HashiCorp Vault Transit Secrets Engine. See the section on passkeys above and my upcoming two separate posts which provide details about how to set up Kasten with Vault and Kasten with KMS.
Multiple Passkeys
In special cases, it’s also possible to have multiple passkeys that all can be used to decrypt the master key. If you require this, contact the Kasten K10 support.