lxd: Add storage option to new projects

[boltmark]

This PR adds a --storage flag to project create, which ensures the default profile in the new project is set up with a root device. This allows for a more smooth project -> workload workflow, as users will be able to immediately use projects to launch instances.

[tomponline]

@boltmark @escabo which jira card is this for, id like to refresh myself on the history and context of this request.

[boltmark]

@boltmark @escabo which jira card is this for, id like to refresh myself on the history and context of this request.

Jira ticket here: https://warthogs.atlassian.net/browse/LXD-2316.

In discussion it seemed that the request was more that new projects shouldn't require extra setup steps from the user in order for simple actions such as launching an instance to succeed.

[tomponline]

@boltmark the jira card doesnt seem to have any discussion on it at all.

@roosterfish @mseralessandri do you have any additional context/links for this?

I am concerned this blanket change of behaviour will break existing users' workflows.

[tomponline]

@boltmark the jira card you linked to mentions server level project defaults, rather than changing the default profile config for all new profiles?

[boltmark]

@boltmark the jira card you linked to mentions server level project defaults, rather than changing the default profile config for all new profiles?

Apparently the requester is @escabo and another person, so maybe he can provide more info. I had discussed this with him a few weeks ago and it seemed that this was the desired change -- but I could have misunderstood and I do see your point that this could affect current users' workflows. It broke quite a few tests...

Perhaps this should instead provide the functionality to set default config such as a root device instead of doing so automatically for new projects as is done in this PR?

[escabo]

I would like that newly created projects allow launching workloads seamlessly i.e. without having to define defaults. If this change risk breaking existing workflows then a new parameter to control that behaviour makes sense.

[hamistao]

I would like that newly created projects allow launching workloads seamlessly i.e. without having to define defaults. If this change risk breaking existing workflows then a new parameter to control that behaviour makes sense.

Sorry to barge in here, it just came to me that -c features.profiles=false kind of accomplishes this, assuming that the default project's profile is configured in a usable maner. What is it exactly that we want to achieve here that we cannot already do with -c features.profiles=false?

[boltmark]

I would like that newly created projects allow launching workloads seamlessly i.e. without having to define defaults. If this change risk breaking existing workflows then a new parameter to control that behaviour makes sense.

Sorry to barge in here, it just came to me that -c features.profiles=false kind of accomplishes this, assuming that the default project's profile is configured in a usable maner. What is it exactly that we want to achieve here that we cannot already do with -c features.profiles=false?

Thanks @hamistao. The only issue I see with this is that we lose project functionality (no profiles in project) for the sole purpose of allowing immediate workload launch. @tomponline what do you think?

I also updated the description with another potential option.

[tomponline]

@boltmark @escabo

I've been thinking about this on and off for a few days (in between meetings) and whilst the request "Set up default root device for new projects" seems, at first glance, to be trivial and obvious the more I think about it the more complicated it seems, so I'm going to try and write down here some of the concerns I have:

So lets start with an easy one:

1. Does this apply new all profiles or just newly created default profiles in new projects?

The config key core.profiles_use_default_root can be set to a boolean value. If set to true, then all new profiles will be set up with the root device from the default profile in the default project.

Based on my discussions on this topic with @escabo and @edlerd the problem we are trying to solve is that when a new project is created (either via the UI or via lxc project create foo) the new project isn't immediately usable for new instances because its default profile (assuming the default features.profiles=true is set) does not contain root disk device.

So commands like lxc launch ubuntu:noble c1 --project foo will fail.

So, the thinking goes, why don't we make newly created default profiles in a project be auto-populated with the root disk device (if present) from the default project's default profile?

But, in my view, the quoted statement above has misunderstood that and has expanded the scope to now auto-populate "all new profiles will be set up with the root device from the default profile in the default project".

This is almost certainly not what we want, as it will hinder the use of new profiles in existing projects for their main use case of storing snippets of configuration that can be combined in different ways to result in an expanded instance config.

For example I might want to create a profile in an existing project (that already has a default profile with a root disk device) for connecting to a specific network2, e.g.

lxc profile create myfastpool
lxc profile device add root disk pool=myfastpool

lxc profile create mynetworks
lxc profile device add mynetwork1 eth1 nic nictype=macvlan parent=enp5s0
lxc profile device add mynetwork2 eth2 nic nictype=macvlan parent=enp6s0

lxc launch  ubuntu:noble c1 --profile myfastpool --profile mynetworks

This would combine 2 profiles, the myfastpool and the mynetworks one, resulting in an instance that uses the root disk from the myfastpool profile and 2 NICs from the mynetworks profile.

If we were to start adding a root disk device from the default profile to the newly created profiles, we'd first have to remove that device when adding the disk device to myfastpool and worse, if I forgot to remove it from mynetworks profile then because of the ordering of profiles, the root disk device in mynetworks would override the one I wanted in myfastpool.

So can we agree that the original intent for this work was not to apply a root disk device to all new profiles, but only that the default profile in newly created projects should have a default root disk device added?

[tomponline]

So can we agree that the original intent for this work was not to apply a root disk device to all new profiles, but only that the default profile in newly created projects should have a default root disk device added?

Assuming the answer is "yes" to this, lets move onto other concerns...

2. Should this be the new default behavior or should it be configurable?

The PR description describes a new global config key core.profiles_use_default_root to enable this new behavior.

This is, I believe, on the right track, as changing the existing behavior such that new projects get the default project's root disk device always is very likely to break existing workflows (because they may add their own root disk devices to newly created profiles and these calls would then break because the project already has one added by default).

The question then becomes: Do we want to make this the new default and accept some user workflow breakages, or do we want to make this opt-in and have MicroCloud configure LXD when its provisioning it to enable this setting by default instead?

[tomponline]

Should this be the new default behavior or should it be configurable?

Assuming the answer is "lets make this opt-in"...

3. Consider the behavior of limits.disk.pool.<DEFAULT_POOL_NAME>=0 in a newly created project

The https://documentation.ubuntu.com/lxd/en/latest/reference/projects/#project-limits:limits.disk.pool.POOL_NAME option controls how much disk space a project can consume on a specified pool, if the value is set to 0 then no space can be consumed.

So we should think about what the behavior should be if the default project's default profile's root disk device references a pool that has a zero quota in the newly created project.

Should we not add a root disk device to the newly created default profile, that we know isn't going to allow instance creation (which will also result in no instances being able to be created) or should we error out and reject creating a project at all?

On a related point, today we have restricted.networks.access config key that restricts access to specific networks.
It stands to reason we may want to add restricted.storage.pools.access (or similar) in the future, and if we do add that we would need to think about how this new setting interacts with that.

[tomponline]

So easy ones out of the way now :)

4. What about non-disk devices?

The original problem statement, as I understand it, is that a new project isn't ready to use for instances without additional manual configuration.

For example, the missing root disk device prevents instance creation.

But once we've fixed this issue, will we also face a similar problem with default network connectivity?

A new instance can be created, but it wont have a network device when created in a fresh project.

So should we also copy the eth0 NIC device from the default project's default profile too?

This would raise similar questions as above, wrt to restricted.networks.access specified on the new project.

This then begs the question, where do we stop with this?
And gets to the nature of what is a project for anyway if the user creating it doesn't need to think about how its configured?

Perhaps we should just copy the entire default profile from the default project into newly created projects?

Or perhaps we should have a global server setting that specifies which profile in the default project should be used as the template for the default profile in newly created projects?

Also, one idea I'm thinking of is that today we have a --storage and --network flags for lxc launch, perhaps we could add the same flags to lxc project create which add the associated devices to the new project's default profile.

We could then offer a similar experience in the UI, so that when creating a new project, the UI asks which storage pool and network to use.

This would avoid introducing new config keys and workflow breakages, but still make it easy to create a new project using the storage pool and network desired.

lxc project create foo --storage mypool --network mynetwork

We could go the opposite way (or do both) and have the global config key specifying which profile to copy into the new project, and then add a --empty flag to lxc project create to allow users to indicate that they don't want the profile copied into the project (just like we have lxc launch c1 --empty).

5. Other things to keep in mind

• Don't modify the default profile unless features.profiles is enabled.
• Don't add a NIC if features.networks is enabled, as it won't be reachable.

[edlerd]

Based on my discussions on this topic with @escabo and @edlerd the problem we are trying to solve is that when a new project is created (either via the UI or via lxc project create foo) the new project isn't immediately usable for new instances because its default profile (assuming the default features.profiles=true is set) does not contain root disk device.

When creating a new project in the UI, we ask the user what should be the storage pool in the default profile for the new project. We preselect the pool from the default projects default profile. Users can change it. This way we enforce that a new projects default profile is always created with a pool selected.

See screen below. So I think for the UI, the problem was resolved.

[edlerd]

Further details on the UI:

On the "normal" creation of a new profile, no storage pool will be added or selected by default. The default there is create the profile with empty root storage setting. I think that is intended and correct.

[tomponline]

We preselect the pool from the default projects default profile. Users can change it. This way we enforce that a new projects default profile is always created with a pool selected.

Cool thanks, we might want to do the same for NICs depending on the outcome of this discussion.

[tomponline]

On the "normal" creation of a new profile, no storage pool will be added or selected by default. The default there is create the profile with empty root storage setting. I think that is intended and correct.

Yeah i think this is good.

[boltmark]

So easy ones out of the way now :)

4. What about non-disk devices?

The original problem statement, as I understand it, is that a new project isn't ready to use for instances without additional manual configuration.

For example, the missing root disk device prevents instance creation.

But once we've fixed this issue, will we also face a similar problem with default network connectivity?

A new instance can be created, but it wont have a network device when created in a fresh project.

So should we also copy the eth0 NIC device from the default project's default profile too?

This would raise similar questions as above, wrt to restricted.networks.access specified on the new project.

This then begs the question, where do we stop with this? And gets to the nature of what is a project for anyway if the user creating it doesn't need to think about how its configured?

Perhaps we should just copy the entire default profile from the default project into newly created projects?

Or perhaps we should have a global server setting that specifies which profile in the default project should be used as the template for the default profile in newly created projects?

Also, one idea I'm thinking of is that today we have a --storage and --network flags for lxc launch, perhaps we could add the same flags to lxc project create which add the associated devices to the new project's default profile.

We could then offer a similar experience in the UI, so that when creating a new project, the UI asks which storage pool and network to use.

This would avoid introducing new config keys and workflow breakages, but still make it easy to create a new project using the storage pool and network desired.

lxc project create foo --storage mypool --network mynetwork

We could go the opposite way (or do both) and have the global config key specifying which profile to copy into the new project, and then add a --empty flag to lxc project create to allow users to indicate that they don't want the profile copied into the project (just like we have lxc launch c1 --empty).

5. Other things to keep in mind

* Don't modify the default profile unless `features.profiles` is enabled.

* Don't add a NIC if `features.networks` is enabled, as it won't be reachable.

Updated this PR with a PoC for adding the --storage flag to project create.