#1 Global Leader in Data Resilience

Veeam Kasten GSB backups fail with the error "mkdir /tmp/kopia-log: read-only file system"

KB ID: 4652
Product: Veeam Kasten for Kubernetes | 7
Published: 2024-09-06
Last Modified: 2024-09-06
mailbox
Get weekly article updates
By subscribing, you are agreeing to have your personal information managed in accordance with the terms of Veeam's Privacy Notice.
This site is protected by hCaptcha and its Privacy Policy and Terms of Service apply except as noted in our Privacy Policy.

Cheers for trusting us with the spot in your mailbox!

Now you’re less likely to miss what’s been brewing in our knowledge base with this weekly digest

error icon

Oops! Something went wrong.

Please, try again later.

Challenge

The backup for the workloads that uses Generic Storage Backup (GSB) fails with the error:

mkdir /tmp/kopia-log: read-only file system\r\nUnable to create logs directory: mkdir /tmp/kopia-log: read-only file system

Cause

Veeam Kasten for Kubernetes's datamover needs write access to the /tmp directory for storing the config files and log files. If readOnlyRootFilesystem is set to true in the securityContext of the kanister-sidecar container, any writes to /tmp are restricted, causing the backup in Kasten to fail with the above error message.

Starting from Veeam Kasten for Kubernetes version 7.0.8, the automatic sidecar injector for GSB merges the securityContext of all the other containers in the workload and selects the most restrictive securityContext for the kanister-sidecar container.

Due to this change, there may be instances where customers need to manually adjust the kanister-sidecar to ensure the GSB functions properly. For example, when the usage of readOnlyRootFilesystem securityContext in any application containers in which the sidecar is injected. 

Solution

This issue can be mitigated by updating the kanister-sidecar container in the customers' workload by adding an emptyDir volumeMount for the /tmp so that the datamover can write its config/log files to that directory. 

Below is an example command to patch the kanister-sidecar container in a deployment with the emptyDir volume.

 

kubectl -n <namespace> patch deployment <deployment/workload name> -p \
'{"spec":{"template":{"spec":{"containers":[{"name":"kanister-sidecar","volumeMounts":[{"name":"tmp-volume","mountPath":"/tmp"}]}],"volumes":[{"name":"tmp-volume","emptyDir":{}}]}}}}'
This command can also be re-purposed to patch other workload types like statefulesets. 
Potential Service Interruption
Patching the workload causes the pods to be deleted and recreated with the changes. Please take caution while updating the sidecar and ensure to do it during a maintenance window for your application.

More Information

By default, data operations done by Kasten requires the Linux Capabilities 'CHOWN', 'DAC_OVERRIDE', and `FOWNER' to be allowed to run in the container.

Starting in Veeam Kasten for Kubernetes v7.0.8, any workload that is injected with a sidecar will automatically incorporate the necessary capabilities. However, existing workloads that have already been injected with sidecars will require a manual update to include these capabilities. 

To submit feedback regarding this article, please click this link: Send Article Feedback
To report a typo on this page, highlight the typo with your mouse and press CTRL + Enter.

Spelling error in text

This site is protected by hCaptcha and its Privacy Policy and Terms of Service apply except as noted in our Privacy Policy.
Thank you!

Thank you!

Your feedback has been received and will be reviewed.

Oops! Something went wrong.

Please, try again later.

You have selected too large block!

Please try select less.

KB Feedback/Suggestion

This form is only for KB Feedback/Suggestions, if you need help with the software open a support case

By submitting, you are agreeing to have your personal information managed in accordance with the terms of Veeam's Privacy Notice.
This site is protected by hCaptcha and its Privacy Policy and Terms of Service apply except as noted in our Privacy Policy.
Verify your email to continue your product download
We've sent a verification code to:
  • Incorrect verification code. Please try again.
An email with a verification code was just sent to
Didn't receive the code? Click to resend in sec
Didn't receive the code? Click to resend
Thank you!

Thank you!

Your feedback has been received and will be reviewed.

error icon

Oops! Something went wrong.

Please, try again later.