One of the cornerstones of Amazon Web Services (AWS) best practices is using multiple accounts to manage your workloads. Dedicating accounts to specific business units or even specific applications provides numerous benefits that increase agility, improve security and streamline operations. This practice also helps operators follow the principle of least privilege, whereby only a minimal set of privileges required to perform a task are assigned to responsible users and services, and ideally, only for the time required to complete the task.
For customers using Veeam Backup for AWS to protect their AWS workloads, it is recommended to dedicate an account to serve as the backup account. The backup account is where Veeam Backup for AWS would be deployed and where Amazon Simple Storage Service (Amazon S3) buckets would be created to store backup data. To access other accounts to perform backups and restores, Veeam Backup for AWS uses the AWS Security Token Service to temporarily assume cross-account Identity and Access Management (IAM) roles. This architecture allows Veeam to operate and store backup data in an account which is completely separate and isolated from accounts that hold your production data.
The main requirement with this architecture is to create IAM roles in each account you want to protect with Veeam. In a small environment with a handful of accounts, you can feasibly create the roles and add them to Veeam by hand. But for a large organization where there may be hundreds of accounts managed by multiple teams, the task becomes more daunting. Thankfully, AWS provides many tools and services for simplifying tasks like creating IAM roles in all your accounts, and these tools can be used with Veeam’s REST APIs to automate the process.
A solution for this scenario was recently published to VeeamHub, Veeam’s curated collection of projects on GitHub. The project is located in the veeam-aws-cloudformation repository in the veeam-backup-aws folder.
It uses two common AWS services to achieve our objective of creating IAM roles in organization member accounts and automatically adding them to Veeam Backup for AWS. Let’s review the AWS services briefly to understand what they are and how they are used.
AWS CloudFormation and Lambda-backed custom resources
The first is AWS CloudFormation. CloudFormation is AWS’s native infrastructure-as-code (IaC) service that helps you model and provision AWS resources. For example, you could write a template that defines an Amazon Elastic Compute Cloud (Amazon EC2) instance and all its properties, then use CloudFormation to create a new Amazon EC2 instance exactly as specified in your template. When CloudFormation deploys a template, the resources created by the deployment are managed in a single unit called a stack. A stack can also be deployed into multiple accounts within an organization using StackSets. We’ll look closer at StackSets in a bit.
The second is AWS Lambda, a computing service that lets you run code without operating a server. A key feature of Lambda is that functions run only when they are invoked by an event. This way you don’t need to manage an operating system to run your code, and you don’t need to leave a server running at all times to listen for events.
While CloudFormation lets you provision AWS resources like Amazon EC2 instances and Amazon S3 buckets, it does not natively support third-party resources. However, AWS has a method for CloudFormation to interact with third-party resources in the form of custom resources. In short, a custom resource is one in which you define your own provisioning logic to respond to an event. A custom resource specifies a service token that references a resource, such as an SNS topic or a Lambda function, which will handle the actual provisioning task.
This ability to invoke Lambda functions from CloudFormation custom resources is perfect for our use case:
- Lambda supports a variety of language runtimes that we can use to write a function to make REST API calls to Veeam Backup for AWS. (This solution’s functions are written in Python.)
- We can add custom resources to a template that defines our IAM roles so that when our roles are created, the Lambda function will be invoked to add the roles to Veeam.
- CloudFormation integrates with AWS Organizations, making it easy to create the IAM roles in organization member accounts using StackSets.
Now that we’ve reviewed the AWS services and features the solution uses, let’s take a look at how it works.
How it works – Lambda functions
There are two components to the solution:
- A stack in the backup account which has two Lambda functions and a secret stored in AWS Secrets Manager. We’ll call this the Lambdas stack.
- A stack set which creates stacks in organization member accounts that create IAM roles for Veeam. This stack also includes the custom resources that will invoke our Lambda functions. This is the Roles stack.
The Lambdas stack contains:
- A Lambda function named RandomExtIdLambda that creates random strings. The random strings are used to give each role a unique external ID in the role trust policy.
- A Lambda function named VeeamConfiguratorLambda that uses data provided by the custom resource to add an IAM role to Veeam Backup for AWS using the REST API.
- A secret containing admin-level user credentials for the Veeam Backup for AWS console.
Protecting secrets
Why the secret? For requests to be made to the Veeam Backup for AWS REST API, a client must authenticate with the API to authorize the request. Since we will be adding IAM roles to the Veeam console, the user account must have Portal Administrator rights. Storing such credentials in plain text anywhere presents a severe security risk and must be avoided.
Protecting sensitive secrets like user credentials is exactly what AWS Secrets Manager is designed for, and this is how the solution stores and accesses credentials. The Lambdas stack will generate a secret when the stack is deployed to keep the credentials from ever appearing in plain text. Once the stack is created, the user simply retrieves the secret from AWS Secrets Manager and uses the generated credentials to create an administrator account in the Veeam Backup for AWS console.
Additionally, the solution uses the recently released AWS Parameters and Secrets Lambda Extension to retrieve the credentials when the Veeam configurator function is invoked. This keeps the credentials secure and makes subsequent credential rotations easily achievable. For example, if you were to rotate the user credentials on the Veeam console, you would simply update the secret in AWS Secrets Manager with the new password, and the next time the function is invoked, it would have valid credentials to do its job — all without ever exposing them in plain text in a template or code.
Network considerations
Another consideration the solution accounts for is how the REST API will be accessed. If the Veeam console and REST API is accessible only from internal networks, as is common for a data protection solution, the Veeam configurator function must operate within that network to make the API calls. The function also needs access to AWS Secrets Manager to access the credentials and to Amazon S3 to send responses to CloudFormation.
Consequently, the Veeam configurator function is configured to connect to an Amazon Virtual Private Cloud (Amazon VPC), and the Lambdas stack creates a VPC endpoint for AWS Secrets Manager to give the function private access to the service. This allows the function to operate properly within a private network and satisfies security best practices.
How it works – CloudFormation StackSets
The second component of the solution is the Roles stack which contains:
- An IAM role named VeeamEc2Role and associated policies.
- An IAM role named VeeamWorkerRole, associated policies, and an instance profile for this role.
- A custom resource for each role of type “Custom::RandomExtIdGenerator” to generate external IDs.
- A custom resource for each role of type “Custom::VeeamAwsConfigurator” which invokes the VeeamConfiguratorLambda function to add the role to Veeam Backup for AWS.
As mentioned earlier, CloudFormation StackSets are used to create these resources in the organization member accounts. This makes the deployment process very straightforward and lets you take advantage of the integration between AWS organizations and CloudFormation.
For example, when you create a stack set, you define a target where the stacks will be created. The target can be either the entire organization or an organizational unit (OU), giving you control of where these roles will be created. You can also specify whether you want the stack to be deployed automatically or manually when an account is added to the deployment target.
Finally, should any changes ever need to be made to these roles, such as adding permissions to support new services in a future release, you can update the template with the added permissions and update the stack set to roll out the changes to all accounts with minimal effort.
Once you create your stack set and define your target and deployment options, the roles will be created in the accounts and will be added to the Veeam Backup for AWS console. You can now log in to Veeam Backup for AWS and start using these roles to create backup policies.
Conclusion
Using native AWS services with Veeam REST APIs creates significant opportunities for automation and orchestration. This solution is just one example of what you can accomplish when you manage your backups as code and bring the power of your platform together with the power of Veeam to make data protection secure and simple.
Lastly, please feel free to customize and adapt this solution to suit your needs by forking the repo on GitHub. If you add enhancements that you think would benefit the greater community, submit a pull request to get your changes reviewed and merged back into the main repo.