Since version 4.0.13 (Release Date: 2021-09-28), Kasten by Veeam has supported using the AWS Key Management Service (KMS) to protect Kasten K10 encryption keys. This provides an alternative to the passphrase Passkey, which is used by default to protect the master key and encryption key for encrypting snapshots exported to external locations such as S3 or NFS.
When using the KMS integration, Kasten K10 sends the ciphertext of the master key to KMS and retrieves the decrypted master key. Encryption keys are derived for each policy based on the master key.
For more details on how encryption works in Kasten K10, see my previous post, “Deep Dive: Encryption in Kasten.”
When To Use AWS KMS
If you are running on AWS and have a requirement to manage your own cryptographic keys, using KMS and a Customer Managed Key is a great option.
Additionally, using KMS frees you from the need to store the passphrase and reduces the number of secrets used in the system.
Walkthrough: Adding KMS to an Existing Kasten Setup
Create AWS KMS Key
First, we need to set up KMS in AWS. How you do this will depend on how you manage your AWS resources (e.g. whether you use the Console, CLI, Terraform, CloudFront, etc.). For the purpose of this blog post, I’ll be using the AWS CLI, because it’s accessible and transparently shows what resources are created. If you want to use something else, it should be fairly straightforward to translate the steps. If you are using Terraform, you can also take a look at the KMS branch of my Terraform + Kasten demo repository.
In KMS, you have different options for creating Customer Managed Keys (CMK). First, it’s important to note that the key needs to be symmetric. For the key material, you can choose either of the options that KMS offers:
- KMS – Lets AWS generate a key for you
- External – Upload your own key into KMS
- CloudHMS – Gives you direct control of the hardware security modules (HSMs)
The complexity of the options varies. We will stick with the basic KMS generated keys for now.
Let’s go ahead and generate the key:
# Create key
aws kms create-key
# --> copy "KeyId" and "Arn" we will need this a couple more times
# Using the KeyId from before, assign the key alias for usability
aws kms create-alias \
--alias-name alias/kasten \
--target-key-id <kms_key_id>
Grant Kasten Access to KMS
Of course, the KMS key is sensitive and not everyone should be able to access it. So let’s create a fine-grained access policy. How you do this depends on how you installed Kasten K10. There are two ways:
- Use an IAM Role: If you set the value awsIamRole in your helm release, Kasten K10 this role for authorization in AWS. You can archive this by enabling OIDC in EKS and creating a special IAM linked service account, or you can use access tokens and an IAM user that is allowed to assume the role.
- Use an IAM User: If you didn’t set awsIamRole in your helm release but have an awsAccessKeyId configured, Kasten K10 will simply be authorized using this user.
It’s important that you know which method you’ll use ahead of time and have the Amazon Resource Name (ARN) of the IAM role/user on hand.
Now, we can add the Kasten user/role to the KMS key policy and to the permissions of the IAM user/role. KMS also uses access policies that restrict who can use each key.
It’s important that we keep the default policy, because otherwise, we would deny ourselves access to the key. To do that we first retrieve the current default key policy:
# Using the KeyId from before...
aws kms get-key-policy --key-id <kms_key_id> --policy-name default | jq .Policy -r | jq
# --> This should log a JSON document
# If you do not have jq installed you can also do it without, but it makes formating way nicer
We can now copy the old policy and add this additional statement:
{
"Effect": "Allow",
"Principal": {
"AWS": "<kasten_iam_user_or_role_arn>"
},
"Action": "kms:*",
"Resource": "*"
}
The full policy should now look something like this and can be applied to the key:
# Using the KeyId and policy from before...
aws kms put-key-policy --key-id <kms_key_id> \
--policy-name default \
--policy '{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "Enable IAM User Permissions",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::123456789:root"
},
"Action": "kms:*",
"Resource": "*"
},
{
"Effect": "Allow",
"Principal": {
"AWS": "<kasten_iam_user_or_role>"
},
"Action": "kms:*",
"Resource": "*"
}
]
}'
Next, we need to make sure our Kasten IAM user/role is authorized to perform KMS actions. To do this we need to add a statement to the policy of the IAM user/role. Instead of the ARN, we use the kms:ResourceAliases attribute so that we could recreate the key without needing to update the policy.
You should already have a policy that grants permissions for ec2:CreateSnapshot, ec2:CreateVolume, etc., where you can simply add this statement:
{
"Effect": "Allow",
"Action": [
"kms:Decrypt",
"kms:Encrypt"
],
"Resource": "*",
"Condition": {
"ForAnyValue:StringEquals": {
"kms:ResourceAliases": "alias/kasten"
}
}
}
The final Kasten IAM policy should look something like this with the correct KMS key ARN:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"ec2:CopySnapshot",
"ec2:CreateSnapshot",
"ec2:CreateTags",
"ec2:CreateVolume",
"ec2:DeleteTags",
"ec2:DeleteVolume",
"ec2:DescribeSnapshotAttribute",
"ec2:ModifySnapshotAttribute",
"ec2:DescribeAvailabilityZones",
"ec2:DescribeSnapshots",
"ec2:DescribeTags",
"ec2:DescribeVolumeAttribute",
"ec2:DescribeVolumesModifications",
"ec2:DescribeVolumeStatus",
"ec2:DescribeVolumes",
"ec2:ResourceTag/*"
],
"Resource": "*"
},
{
"Effect": "Allow",
"Action": "ec2:DeleteSnapshot",
"Resource": "*",
"Condition": {
"StringLike": {
"ec2:ResourceTag/Name": "Kasten: Snapshot*"
}
}
},
{
"Effect": "Allow",
"Action": [
"kms:Decrypt",
"kms:Encrypt"
],
"Resource": "*",
"Condition": {
"ForAnyValue:StringEquals": {
"kms:ResourceAliases": "alias/kasten"
}
}
}
]
}
After updating the policy, Kasten K10 should have permission to use the KMS key.
Switching Encryption to KMS
Now that we’ve set up the key and configured the necessary policies, we can go ahead and create a new KMS Passkey. This will re-encrypt the master key using the KMS key and store the ciphertext. See the Kasten K10 docs on changing Passkeys for more details.
In the future, Kasten K10 will send the ciphertext of the master key to KMS whenever it needs to access it, for example, when Kasten K10 needs to derive an encryption key for protecting a snapshot that should be exported.
We will need to create a new Passkey to switch the default Passkey to a KMS Passkey. To do so, we will create passkey-kms.yaml :
apiVersion: vault.kio.kasten.io/v1alpha1
kind: Passkey
metadata:
name: kmsKey
spec:
awscmkkeyid: <kms_key_arn>
usenow: true
We can then create the Passkey using this command:
kubectl create -f passkey-kms.yamlThe new key should now be the only key with inuse:true :
kubectl get passkeys.vault.kio.kasten.io -o yamlShould the creation fail for whatever reason, we can look for more information in the logs of the crypto-svc service:
kubectl -n kasten-io logs -f -l run=crypto-svc -c crypto-svcKasten Kq10 will now use the new Passkey and AWS KMS for encryption. It’s now safe to delete the old Passkey.
Setting Up Kasten with KMS From Scratch
When setting up a fresh Kasten K10 installation, we can immediately connect Kasten K10 to KMS using a secret.
After you’ve created the KMS key with the necessary policies, we can create a special secret with the name k10-cluster-passphrase , and use that for the configuration:
kubectl create secret generic k10-cluster-passphrase \
--namespace kasten-io \
--from-literal awscmkkeyid=<kms_key_arn>
You will always have the option to either configure KMS directly in the Passkey or add a reference to a secret and add the configuration there. However, after you’ve installed Kasten K10, the Passkey API resource already exists and there is little value in going the extra step to create a secret that the Passkey points to.
Rotating the KMS Key
KMS has a feature called automatic key rotation. If enabled, AWS will generate new cryptographic material for the KMS key every year. For obvious reasons, the feature is only available for keys where the key material is generated by KMS (i.e. not external or CloudHMS keys). However, enabling or disabling this feature has little impact on the Kasten K10 setup.
When AWS generates new key material, the old key material doesn’t go away. When encrypting data, the latest key material is used. However, decrypting data encrypted using previous versions of the key still works. Because Kasten K10 isn’t aware that KMS rotated the key, it won’t re-encrypt the master key, and will keep using the old key material.
Kasten K10 can be forced to re-encrypt the master key by creating a new Passkey with a different name. You could create an automation that waits for the KMS key rotation event, for instance.
If you like to have more control and the ability to trigger a key rotation any time, you can perform a manual key rotation. A manual key rotation is a three-step process: create a new key, update all references to it, then remove the old key. I’ll demonstrate the manual key rotation process here:
1. Create the New Key
Let’s first get our key and the key policy:
aws kms describe-key --key-id alias/kasten
aws kms get-key-policy --key-id <old_kms_key_id> --policy-name default | jq .Policy -r | jq > policy.json
Now we can create a new key with the same policy:
aws kms create-key --policy "$(cat policy.json)"
# --> copy new "KeyId" and "Arn" we will need this a couple more times
Next, we update the alias:
aws kms update-alias --alias-name alias/kasten --target-key-id <new_kms_key_id>2. Update All the Ciphertext
To update the cyphertext , we need to force Kasten to re-encrypt the master key. We do this by creating a new Passkey. You can edit passkey-kms.yaml , but rename the key to kmsKey_v2 and put the new key ARN as awscmkkeyid .
kubectl create -f passkey-kms.yaml
Now we can check that the new key is now also the used key:
kubectl get passkeys.vault.kio.kasten.io kmsKey_v2 -o yaml
# -> Should be "inuse: true"
# Optionally, you can now delete the old key:
3. Schedule Deletion for the Old Key
We can now remove the old KMS key:
aws kms schedule-key-deletion --key-id <old_kms_key_id> --pending-window-in-days 7Automated Key Rotation
I’ve also created a script for automating the process outlined above. You can find it on github.com/MoritzKn/kasten-key-rotation.
Further Information
- Onkar Bhat, one of the engineers behind this feature held a talk about this at Cloud Field Day.
- The code that I created while researching this topic can be found in the aws-kms branch of my tf-k8s-kasten-demo
- Review the Kasten K10 documentation on Configuring K10 Encryption.