As a support engineer, I get all sorts of questions related to retention: from basic questions such as “How does it work?” and “What would happen if…?” to nightmare fuel like “I changed my retention settings, but why are my backups gone?” So, I can say with confidence that retention policy is something you’re just not allowed to make a mistake with because it directly affects the most important thing: the backup data. And as you’ve probably already guessed, its misconfiguration might lead to the data being removed earlier than expected or not being backed up in the first place.
In this article, I will explain the difference between retention types available in Veeam Backup for Microsoft Office 365 and demonstrate how it works at several different points of a backup job cycle.
Veeam Backup for Microsoft Office 365 retention types
There are two types of retention available.
Snapshot-based retention
This retention type is designed to provide a similar retention experience as in Veeam Backup & Replication and Veeam Agent products. Although there are some differences due to a file-level structure of backed up data in Veeam Backup & Replication, the idea is the same. The number of days (or years) that have been selected on the following page affects how long the whole restore point would be kept in the repository.
Every restore point would have a complete copy of all objects added to a backup job and this copy doesn’t get changed by retention over time.
This type of retention should be selected if you’re planning to create the exact copy of all the data stored in Office 365 and then manage it as a single, unchanged entity both in terms of backup and retention runs.
Item-level retention
This retention type works in a similar way to retention policies in Exchange online:
All the data in an Office 365 mailbox is stored for a specific number of days configured in the Exchange admin center. If the data has not been modified for that number of days, it would be either deleted or placed in the archive part of a mailbox depending on the settings configured.
For example, if a repository with item-level retention has seven days retention configured, then only the data that has been modified at least once for the last seven days would be backed up.
All the data with older information in the Date Modified field would be skipped. Moreover, this mechanism would not only be applied on the go while the job is creating a new restore point, but would also check the content of existing restore points and remove the files that are falling out of that seven-day Date Modified age as well!
This retention type is suitable, for example, if you’re planning to replicate not only your Office 365 data but its retention rules as well. Usually, it’s used to meet legal requirements that might state that all the data that gets older than a particular lifecycle must be deleted. Or it can simply help to lower the amount of disk space required for a full backup.
Although this option only works in Office 365 for Exchange Mailbox and optionally its Archive, in Veeam Backup for Microsoft Office 365, it can be applied to OneDrive, SharePoint and Teams.
Workflow examples
These images show how both retention types work in a span of a few days while encountering some common events and issues on the way.
Let’s make a simplification and imagine there is a user who has a mailbox with 3 messages and OneDrive for Business with 3 files in it:
Other conditions are:
- We use separate backup jobs for OneDrive and Exchange data of this user:
- OneDrive jobs runs daily at 01:00AM
- Exchange jobs runs daily at 02:00AM
- Retention policy for the Item-level repository is set to three days
- Retention policy for the snapshot-based repository is set to two days
- Retention policy is set to be applied daily at 00:01AM for both repositories
Full backup
Starting with the first session for any newly created job – a full backup.
22.08.2019 00:01 am: The retention policy is being applied, but since the repository is empty it doesn’t do anything.
Snapshot-based retention
The job would back up all items existing in production on every run.
22.08.2019 01:00 am: A OneDrive job starts and backs up every file.
22.08.2019 02:00 am: An Exchange job starts and backs up every file.
Item-level retention
This would only backup items with a date modified less than three days old.
22.08.2019 01:00 am: A OneDrive job starts and backs up ONLY the “4.log” item since the date modification for the other two is older than three days.
22.08.2019 02:00 am: An Exchange job starts and backs up the “2.msg” and “3.msg” items since both messages were received less than three days ago.
Unchanged file incremental backup
Now let’s see what happens on the next run considering that there’s been no changes in Office 365.
Snapshot-based retention
23.08.2019 01:00 am: A OneDrive job starts. It uses the Veeam change tracking feature to confirm that all items have been already backed up on a previous run and that there were no changes for it between job runs, so it creates a reference to the original items in the new restore point instead of re-downloading the same data from Office 365.
23.08.2019 02:00 am: An Exchange job starts. It uses Exchange Web Services change tracking feature to confirm that all items have been already backed up on a previous run and that there were no changes for them between job runs, so it creates a reference to the original items in the new restore point instead of re-downloading the same data from Office 365.
Item-level retention
23.08.2019 01:00 am: A OneDrive job starts. It uses the Veeam change tracking feature to confirm that “4.log” item has already been backed up on a previous run and that there were no changes for it between job runs, so it creates a reference to the original item in the new restore point instead of re-downloading the same data from Office 365.
23.08.2019 02:00 am: An Exchange job starts. It uses Exchange Web Services change tracking feature to confirm that the “2.msg” and “3.msg” items have already been backed up on a previous run and that there were no changes for them between job runs, so it creates a reference to the original items in the new restore point instead of re-downloading the same data from Office 365.
New or modified file incremental backup
In case of an existing file being modified or a new file being created, the job would correspondingly either update an existing record or create a new one for it.
So, for example, if someone sends a new message to our mailbox, there would be a new record created in the repository for it on the next job run.
Snapshot-based retention
23.08.2019 02:20 pm: Someone sent a new email to this user and it’s now stored with other items.
24.08.2019 01:00 am: A OneDrive job starts. It uses the Veeam change tracking feature to confirm that all items have been already backed up on a previous run and that there were no changes for it between job runs, so it creates a reference to the original items in the new restore point instead of re-downloading the same data from Office365.
24.08.2019 02:00 am: An Exchange job starts. It uses Exchange Web Services change tracking feature to confirm that all items have been already backed up on a previous run and that there were no changes for them between job runs, so it creates a reference to the original items in the new restore point instead of re-downloading the same data from Office365. It also creates a new record for 4.msg file and downloads it.
Item-level retention
23.08.2019 02:20 pm: Someone sent a new email to this user and it’s now stored with other items
23.08.2019 09:11 pm: The user made some modifications to 1.docx item and now its Date Modified field equals the current date.
24.08.2019 01:00 am: A OneDrive job starts. It uses the Veeam change tracking feature to confirm that all items have been already backed up on a previous run and that there were no changes for it between job runs, so it creates a reference to the original items in the new restore point instead of re-downloading the same data from Office365.
24.08.2019 02:00 am: An Exchange job starts. It uses Exchange Web Services change tracking feature to confirm that all items have been already backed up on a previous run and that there were no changes for them between job runs, so it creates a reference to the original items in the new restore point instead of re-downloading the same data from Office365. It also creates a new record for 4.msg file and downloads it.
Retention being applied
Now let’s observe both policies in action. Although there have already been several daily retention checks up to this point, I’m intentionally showing this particular one because all the previous runs did not delete any files.
Snapshot-based retention
25.08.2019 00:01 am: Retention policy is being applied. It removes all backups created earlier than 23.08.2019 00:01 am because their date and time of a backup is now more than 48 hours old.
Item-level retention
25.08.2019 00:01 am: Retention policy is being applied. It removes the “4.log”, “2.msg” and “3.msg” items from ALL existing restore points because their date modification fields are now older than three days.
What happens if a backup job fails to create a new restore point for an object?
In such a case, a job would postpone applying retention before it would be able to continue creating new restore points.
It works in the same way for both retention types, so let’s observe a snapshot-based scenario for it.
25.08.2019 01:00 am: OneDrive job starts and fails right away for all objects because of some technical issue. (for example, the server was disconnected from the Internet)
25.08.2019 02:00 am: Same happens for the Exchange job.
26.08.2019 00:01 am: Retention policy is being applied. It removes all backups created earlier than 24.08.2019 00:01 am because their date and time of a backup is now more than 48 hours old.
26.08.2019 01:00 am: OneDrive job starts and fails once again for the same reason
26.08.2019 02:00 am: Same happens for the Exchange job.
26.08.2019 11:00 am: The issue that was causing backups to fail has been acknowledged and fixed.
27.08.2019 00:01 am: Retention policy is being applied. It doesn’t remove any files from the repository although the date and time of a backup for them is no longer covered by its 48 hours period. This happens because the retention policy preserves the last successful backup during failed runs until the new one is created.
27.08.2019 01:00 am: A OneDrive job starts and successfully creates a new restore point. After that, it removes the restore point created on 24.08 because it no longer meets the retention policy and at the same time now we have a confirmation that a new restore point was successfully created.
27.08.2019 02:00 am: The same thing happens to the Exchange job.
It’s important to understand that this rule is only applicable for the job experiencing any issues. If a job was simply disabled in UI, retention would not be paused.
What happens on a storage?
If you’ve ever opened a folder of a local Veeam repository, you probably noticed that it doesn’t contain actual backed up emails or OneDrive documents as separate files. Instead, it has a structure like this:
Each of these folders stores a database container called repository.adb where we put all backed up data in. The container for any particular file is chosen based on its date modification field in Office 365.
Such an approach means that when retention removes backed up data from the repository, in most cases it would only mark a data block of that repository.adb container empty rather than clear any actual space on disk. And only if all the data in a container has been removed by retention does the container itself gets deleted. The good news is that those empty sectors would be reused for new backups.
All the mentioned storage specifics don’t affect object storage repositories due to a completely different architecture which, in my opinion, deserves its own post.
Concluding key points
- Both retention types are good for their own purpose, so prior to choosing one, I suggest determining what backup requirements you have.
- If not set to “Keep forever” both retention types would remove backed up data at some point and item-level retention would also define which data to back up.
- In case of a local repository, an actual disk space might not get freed up right after retention removes the data.
Are you ready to get hands-on with Veeam Backup for Microsoft Office 365? Download the 30-day free trial.
Related content:
- The Office 365 Shared Responsibility Model [blog post]
- Microsoft Teams backup and recovery in NEW Veeam Backup for Microsoft Office 365 v5 [blog post]
- 7 Reasons Why Microsoft 365 Backup is Critical [blog post]
- Cloud Data Retention [demo]
- VMware Snapshots vs Backups: What’s The Difference? [blog post]