Mandatory S3:DeleteObject Permission in Velero Forces Risky Maintenance Operations

[andreasvikke]

Description

I am running into an issue where Velero’s integration with Amazon S3 requires s3:DeleteObject permissions, which I cannot disable without breaking the backup maintenance process.

Problem Description

• I use Kopia to back up data to S3, specifically backing up Kubernetes Persistent Volume Claims (PVCs) incrementally.
• Velero requires s3:DeleteObject permissions to perform repository maintenance.
• The maintenance job is mandatory and cannot be disabled or moved to a separate cloud environment.
• Because of this, I cannot restrict deletion permissions, meaning any entity with access to Velero’s credentials can delete backup data.

Expected Behavior

• Velero should allow users to configure maintenance without requiring delete permissions.
• There should be an option to separate the maintenance job from normal backup operations.

Steps to Reproduce

1. Set up Velero to back up PVCs to an S3 bucket.
2. Attempt to remove s3:DeleteObject permissions from Velero’s IAM role.
3. Observe that backups break due to the required maintenance job failing.

Impact

• If Velero’s credentials are compromised, an attacker could delete all backup data.
• Users cannot delegate maintenance operations to a separate cloud environment, reducing flexibility.

Potential Workaround

One option is to enable versioning on the S3 bucket and set up a cron job that removes delete markers only after a 7-day grace period. This would prevent immediate permanent deletion and provide time for recovery if an attacker deletes everything. However, this approach adds operational complexity.

A cleaner solution would be for Kopia to support repository maintenance and TTL expiration natively via the CLI, allowing backup pruning and maintenance to be handled separately from Velero’s delete requirements.

Request for Community Input

• Is there a workaround to prevent Velero from requiring s3:DeleteObject while still allowing proper repository maintenance?
• Can Velero introduce a feature to split maintenance operations from backup operations?

Looking forward to any feedback or suggestions from the community!

[sseago]

One note -- it's not just kopia maintenance that requires this. Backup and deletion are other operations that Velero does which require this permission. When the backup hits its expiration date, Velero automatically deletes the backup and associated metadata in the bucket (including restore metadata). When manually deleting a backup or restore, velero also deletes it.

[andreasvikke]

See reply in other thread, think that clarify my use case of Velero 🙂
#8724 (reply in thread)

[kaovilai]

There should be an option to separate the maintenance job from normal backup operations.

Seems doable but not probably not going to be any more secure.. just adds ability to separate credentials into two secrets instead of one.

[sseago]

@kaovilai Also, this doesn't remove the need for delete permission, since we need this for expiring and deleting backups and for deleting restores.

[andreasvikke]

But if you move the maintenance out from the cluster into the AWS account itself then the cluster will never know about the credentials capable of Deleting objects. This means that the maintenance can be disabled from the cluster all together and thereby keeping the security high from a cluster attack perspective.

The velero expiration can maybe also be disabled.
As these are not incremental, we can leverage the bucket lifecycle to clean up instead of using TTL.

[sseago]

If you used a bucket lifecycle approach, you'd probably need to prevent the bucket deleting things in kopia, although I don't know whether bucket lifecycle policies are flexible enough to only apply to certain directories in the bucket (i.e. expire/delete things in backups/restores but not under kopia). If the bucket deletes files in the kopia repo based on age, you run the risk of invalidating your repository metadata -- those objects should only ever be deleted by kopia maintenance actions. Also, you say "these are not incremental" -- but if you ever back up a volume more than once before the prior backups of that volume are deleted, then subsequent kopia snapshots will be incremental snapshots, meaning that you may still need the earlier data, even if the backup associated with that earlier snapshot has been deleted.

[sseago]

If you are able to configure the bucket lifecycle policy to ignore the kopia dir, then you might be able to work around this by:

1. setting your velero default backup TTL to be much longer than your bucket lifecycle TTL, relying on the bucket to delete things, which should (in theory) result in velero deleting the in-cluster CRs once they're orphaned
2. setting the maintenance frequency to a long time period (the default is 1 hour) and then just ignoring the fact that you get an occasional failed maintenance job -- and then run kopia maintenance manually on your own schedule outside of Velero.

I haven't tested this workaround, so I'm not 100% sure it would work. That being said, running kopia maintenance via cli outside of velero is not a supported or tested configuration, so we can't be certain that it won't cause problems in some way.

[andreasvikke]

What I meant with "these are not incremental" was the velero metadates under /backups.

Most S3 compatible buckets are capable of doing prefixed lifecycle so you only do lifecycle on the backups folder and not kopia.

I have tested the solution you suggest but got stuck at actually changing the maintenance frequency, do to unified repo settings... Maybe I should check out more of the source code to find a solution to that.

Also I noticed the maintenance job running after each backup was made even though the frequency stated 1 hour, and therefore the backup stated PartialFail. Maybe I'm missing something from the docs on that

[Lyndon-Li]

Velero is a backup/restore solution that works in various platforms, i.e., public clouds, private clouds, hybrid clouds and on-premise envs. So Velero should take the universal solutions so as to keep the compatibility to all platforms. For the original concern of bucket items being deleted, WORM storage(i.e., object lock) is already the universal solution, so Velero should take it instead of others.
Supporting storages without WORM is another topic, it will be very complicated (the suggestions mentioned here are not enough) and out of our plan.
Object lock is not yet supported by Velero, but it is already on our radar.

As for the repo maintenance, it is a critical operation to keep the repository healthy, so it should always run routinely. The repo itself already has lots of polices and margins to make sure the data won't be deleted too early or at the wrong time. So data security in this term is not an issue, or if there is any, it should be fixed by the repo, instead of making any workaround.