Feature Request: Store changed config keys as pending when they are immutable on running instances

[vosdev]

I often run into errors changing configuration that can only be changed when the instance is stopped.

I would like to request a feature that allows us to change these options regardless of the state of the instance.

Failed updating memory limit: Cannot increase memory size beyond boot time size when VM is running (Boot time size 8192MiB, new size 12288MiB)

After making a configuration change that is currently not allowed, the configuration can be temporarily stored in a volatile.pending.* key, to be automatically applied when the instance stops.

(In the case of my memory example, this would be volatile.pending.limits.memory: 12GiB)

Of course this is just a quick suggestion of a possible solution.

What does this fix?

The current way of work is to poweroff my instance, wait for it to power down, make the config change, power on instance. This is slow and gives you quite some downtime.

This strength of this feature would allow you to make a change to the instance and either reboot it for minimal downtime, or schedule a reboot at 03:00 in the morning so no one will notice the downtime.

[simondeziel]

@vosdev is this feature request mostly/only due to the limitation around live updating the limits.memory? If so, I think it would be worthwhile for us to instead make it changeable at runtime rather than defer it.

We have hot CPU plugging/unplugging so it would make sense to be a tad more flexible with RAM as well.

[vosdev]

lmits.memory was just an example, there are several other options not changable for running VMs.

I tried to find a list on the documentation but it's not specified what options are only limited to stopped instances.

hot swappable RAM would of course also be a very interesting feature! I didn't know if it would be possible!

[vosdev]

I just tried to set deletion protection on a VM and that also got rejected.

Key "security.protection.delete" cannot be updated when VM is running. I'm not familiar with this part of the code but it sounds like a the state should not matter for an option like this to be set.

[simondeziel]

I just tried to set deletion protection on a VM and that also got rejected.

Key "security.protection.delete" cannot be updated when VM is running. I'm not familiar with this part of the code but it sounds like a the state should not matter for an option like this to be set.

This is a recent change made in #14532. It's not clear to me why security.protection.start and security.protection.delete would not be live-updatable on VMs, @MggMuggins do you remember why by any chance? Thanks!

[MggMuggins]

I think this may be an unintended consequence of the way live-updateable keys are checked in the qemu driver (allowlist as opposed to a denylist). security.protection.start was only tested with containers IIRC.

Unfortunately, I introduced a dependency on this behavior in #14532; setting security.protection.start on a running instance would allow the instance's root disk to be shared with another VM while both instances are running without setting security.shared on the volume.

In general any check against the running state of an instance is prone to races because LXD queries the running state from qemu/lxc any time it is requested; a user could set security.protection.start when the instance is stopped and a parallel request starts the instance before the database change is committed. This might not be an issue in practice and we could probably work around the dependency in #14532. Hopefully this helps.

[simondeziel]

I think this may be an unintended consequence of the way live-updateable keys are checked in the qemu driver (allowlist as opposed to a denylist). security.protection.start was only tested with containers IIRC.

Is this something that could be fixed so those keys can be set live? Thanks