incorrect install command suggested in conda-lock.yml comments with 'dev' extra dependencies

[qmarcou]

Checklist

• I added a descriptive title
• I searched open reports and couldn't find a duplicate

What happened?

After building a lock file from a pyproject.toml with optional dependencies named 'dev':
conda-lock --extras dev -f pyproject.toml --mamba with pyproject.toml being:

[project]
name = "MNWE"
dynamic = ["version"]
dependencies = [
    "python",
]

[project.optional-dependencies]
dev = [
    "pytest",
]

[tool.conda-lock]
channels = [
    'conda-forge', 'defaults'
]
platforms = [
    'win-64'
]

the conda-lock.yml header displays incorrect install information:

# This lock contains optional development dependencies. Include them in the installed environment with:
#     conda-lock install --dev-dependencies -n YOURENV conda-lock.yml

which fails

conda-lock install --dev-dependencies -n YOURENV conda-lock.yml
Usage: conda-lock install [OPTIONS] [LOCK_FILE]
Try 'conda-lock install --help' for help.

Error: No such option: --dev-dependencies

Both :
conda-lock install --dev -n YOURENV conda-lock.yml and conda-lock install -e dev -n YOURENV conda-lock.yml work, any of these two commands should be suggested instead.

I guess these lines should be deleted (if the -e option is prefered) or updated:

conda-lock/conda_lock/lockfile/__init__.py

Lines 193 to 199 in fcb8d53

	if "dev" in categories:
	write_section(
	f"""
	This lock contains optional development dependencies. Include them in the installed environment with:
	conda-lock install --dev-dependencies -n YOURENV {path.name}
	"""
	)

When using any other name than dev (e.g foo) for extras the correct command is suggested : conda-lock install -e foo -n YOURENV conda-lock.yml

Conda Info

active environment : lock-env
    active env location : C:\Users\zzzzz\AppData\Local\miniconda3\envs\lock-env
            shell level : 2
       user config file : C:\Users\zzzzz\.condarc
 populated config files :
          conda version : 24.4.0
    conda-build version : not installed
         python version : 3.10.14.final.0
                 solver : libmamba (default)
       virtual packages : __archspec=1=skylake_avx512
                          __conda=24.4.0=0
                          __win=0=0
       base environment : C:\Users\zzzzz\AppData\Local\miniconda3  (writable)
      conda av data dir : C:\Users\zzzzz\AppData\Local\miniconda3\etc\conda
  conda av metadata url : None
           channel URLs : https://repo.anaconda.com/pkgs/main/win-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/win-64
                          https://repo.anaconda.com/pkgs/r/noarch
                          https://repo.anaconda.com/pkgs/msys2/win-64
                          https://repo.anaconda.com/pkgs/msys2/noarch
          package cache : C:\Users\zzzzz\AppData\Local\miniconda3\pkgs
                          C:\Users\zzzzz\.conda\pkgs
                          C:\Users\zzzzz\AppData\Local\conda\conda\pkgs
       envs directories : C:\Users\zzzzz\AppData\Local\miniconda3\envs
                          C:\Users\zzzzz\.conda\envs
                          C:\Users\zzzzz\AppData\Local\conda\conda\envs
               platform : win-64
             user-agent : conda/24.4.0 requests/2.28.1 CPython/3.10.14 Windows/10 Windows/10.0.19045 solver/libmamba conda-libmamba-solver/24.1.0 libmambapy/1.5.8
          administrator : False
             netrc file : None
           offline mode : False

Conda Config

# nothing

Conda list

No response

Additional Context

This is in between documentation and bug issue and I wasn't sure how to assign it. This is also somewhat related to #255 .

[maresb]

@mariusvniekerk, is there any reason not to deprecate all the dev related flags on conda-lock install and conda-lock render? It seems to me like --extras dev is equivalent, and much less confusing.

And I think we should readd --dev-dependencies with a deprecation error.

[qmarcou]

Hi,
Is this harmonization somewhere on the roadmap? I just got confused again by this --dev switch for categories called "dev" that I cannot select using --extras

[maresb]

Thanks @qmarcou for the reminder! I think this fix should be straightforward. Would you be interested in submitting a PR?

[qmarcou]

I could try to give it a go.
Just to make sure we are on the same page, here are the goals:

• remove all --(no-)dev(-dependencies) switches in favor of using the --extra ones that are more generic
• add a deprecation error upon receiving --(no-)dev(-dependencies) related switches

The scope of commands that would be altered is: install, lock, render and render-lock-spec (in a nutshell, all conda-lock subcommands)

[maresb]

Hi @qmarcou, your last message somehow disappeared from my inbox. 🤦‍

Yes, your description is correct. That would be amazing. And perfect for a breaking change to include in v3.

[qmarcou]

Hi @maresb ,
Just facing one behavior interrogation: according to the README --dev-dependencies are included by default when using the lock command, while for --extras they have to be explicitly specified.

Should I keep the current extras behavior or the dev one?
IMO for the lock command including extras by default (--dev-dependencies style) makes more sense given the more modern lock-file format that carries the category field information. This means that by default the lock-file is built such that anybody can obtain the same environment up to the extras at install time with the simplest lock command. A logic could be: by default all extras included during lock, if some extras given then only those extras included in the lock, and if --no-extras well... no extras.
Now this behavior makes less sense at render or install time, and the more sensible default would be no extras, but this means a less consistent behavior among the different commands...

So what should I aim for? Most sensible defaults or CLI consistency?

[maresb]

Oof, I hadn't thought of this. Thanks for explaining. Unfortunately I don't have any answers now. I hope we can discuss our way through this one. I originally envisioned this as something as simple as --dev-dependencies → --extras dev because I forgot that default behavior was to include dev.

From the user perspective, I think we should strive to make it as easy as possible to migrate from breaking changes. For example, I like the idea of deprecating an argument and failing with an error of how to switch to its direct replacement argument. If someone is maintaining some workflow, and we need to break it, then I'd like for them to have a 5-minute fix.

by default all extras included during lock

I'm concerned that with this approach, some users may potentially suddenly inherit a bunch of new extras after upgrading conda-lock, introducing weird constraints coming from the extra dependencies, possibly rendering the environment unsolvable.

To spell out the situation as I understand:

• The current default of the lock subcommand is to set extras to ["dev"] if dev exists, otherwise []. The other subcommands default extras to [].
• If we decide to preserve current behavior, then we need to introduce an alternative to --no-dev-dependencies, e.g. --no-extras.

I'm realizing now that this is tangent to the duplicated category vs extra terminology (also recently feature if you count pixi, and I'd like to move in the direction of that format). Should we also introduce --no-categories?

I think the default behavior of lock including dev but no other extras is pretty weird. So I'm wondering:

• Do we want to move away from this weird default behavior? If so, should we default to no extras or all extras?
• Instead of going all the way to the desired end state, would it make sense to break this up into multiple smaller breaking changes, like just rename --no-dev-dependencies but keep all the defaults for now?

[qmarcou]

I'll first draft something along the simple --dev-dependencies → --extras dev line.
Regarding making breaking changes explicit by throwing informative deprecation errors, we're on the same page.

Regarding the the extras default behavior, I think you have it all perfectly summed up.

Though I agree with the fact that adding all extras by default by may lead to unsolvable environments I think I have a different view on this. The way I see the conda-lock workflow is:
recipe (+possibly extra spices, the extrats) -> (lock) -> multi platform yml lock file (-> (render) -> explicit lock file) -> (install) -> installed conda environment

For me this workflow makes most sense when extra spices do not require to change the core of the recipe/environment. E.g. I may want to have a core computing environment to run my scientific code, have development extra with linters/formatters etc, an extra with security scanning (e.g with jake). Typically here, running the safety scanning extra only makes sense if I can scan the same core and development extra that the ones run without the security extras.
IMO unsolvable environments considering all extras should be an exception to which we can provide an exit hatch to the user: by specifying --no-extras or solving explicitly with only subsets of extras.

[maresb]

Thanks @qmarcou for the thoughtful response. I think I agree with you, but I'm not sure I've understood:

For me this workflow makes most sense when extra spices do not require to change the core of the recipe/environment. E.g. I may want to have a core computing environment to run my scientific code, have development extra with linters/formatters etc, an extra with security scanning (e.g with jake). Typically here, running the safety scanning extra only makes sense if I can scan the same core and development extra that the ones run without the security extras.

What exactly do you mean here by "change the core of the recipe/environment"? Even with your example I still don't understand what the "change" is.

I would like to adjust the defaults, and I think would make most sense to lock all the extras by default.

The upcoming release is way bigger than I wanted because i delayed it due to mamba v2. For that reason, for this particular release I want to postpone any breaking changes that may have weird side-effects (like changing the input dependencies to the lock command). I'm comfortable with including purely mechanical breaking changes like replacing argument names.

So if we change which dependencies are included by default, let's target that for conda-lock v4 instead of v3. As long as v3 goes smoothly, I don't see any reason to wait long to release v4.

[qmarcou]

Hi @maresb apologies for the late reply.
Sorry if my point is not clear, here is a minimal example to illustrate. Suppose I have these two environment files:

env_core.yml

channels:
  - conda-forge
dependencies:
  # Core dependencies
  - python

and
env_extra.yml

channels:
  - conda-forge
category: my_extras
dependencies:
  - python <=3.12
  - jake  # scan dependencies for known security issues

Solving using conda-lock lock --files env1.yml env2.yml or conda-lock lock --files env1.yml env2.yml --extras my_extras will produce two different environment, the first with latest python (e.g. 3.13.2 as of today) the latter with latest python 3.12 (e.g. 3.12.9 as of today). What I meant earlier by "core of the recipe/environment" here is contained in env_core.yml, meaning all packages required to install an environment without extras.

In my opinion the current default for extras encourages to lock twice, once with my_extras and once without, producing two locked environments with incompatible "core" components (here two different python versions). In this example change to the "core" locked environment is just in python's version, but in a more complex env_core.yml or env_extra.yml it may lead to quite different package sets due to chained dependencies and constraints.

On the other hand adding all extras by default makes everything much clearer in my opinion:

conda-lock lock --files env1.yml env2.yml 
conda-lock render --kind explicit --extras my_extras --filename-template conda-{platform}-withextras.lock
conda-lock render --kind explicit --filename-template conda-{platform}-core.lock # all packages from in this env are contained in the previous one

I can start with deprecating --dev related switches first and change defaults afterwards and let you judge what to put in. Still I would advocate to keep those two changes coupled, as I fear making a breaking change to --extras default later would introduce a more silent breaking change.

[qmarcou]

One more specific question, dev information could be used to create the lockfile name template but not extras, should I add support for a similar functionality with generic extras or just deprecate?

See lock

conda-lock/conda_lock/conda_lock.py

Lines 1347 to 1386 in df4280c

	def lock(
	ctx: click.Context,
	conda: Optional[str],
	mamba: bool,
	micromamba: bool,
	platform: Sequence[str],
	channel_overrides: Sequence[str],
	dev_dependencies: bool,
	files: Sequence[PathLike],
	kind: Sequence[Union[Literal["lock"], Literal["env"], Literal["explicit"]]],
	filename_template: str,
	lockfile: Optional[PathLike],
	strip_auth: bool,
	extras: Sequence[str],
	filter_categories: bool,
	check_input_hash: bool,
	log_level: TLogLevel,
	pdb: bool,
	virtual_package_spec: Optional[PathLike],
	pypi_to_conda_lookup_file: Optional[str],
	with_cuda: Optional[str] = None,
	update: Optional[Sequence[str]] = None,
	metadata_choices: Sequence[str] = (),
	metadata_yamls: Sequence[PathLike] = (),
	) -> None:
	"""Generate fully reproducible lock files for conda environments.
	By default, a multi-platform lock file is written to conda-lock.yml.
	When choosing the "explicit" or "env" kind, lock files are written to
	conda-{platform}.lock. These filenames can be customized using the
	--filename-template argument. The following tokens are available:
	\b
	platform: The platform this lock file was generated for (conda subdir).
	dev-dependencies: Whether or not dev dependencies are included in this lock file.
	input-hash: A sha256 hash of the lock file input specification.
	version: The version of conda-lock used to generate this lock file.
	timestamp: The approximate timestamp of the output file in ISO8601 basic format.
	"""

and do_render

conda-lock/conda_lock/conda_lock.py

Lines 516 to 528 in df4280c

	for plat in platforms:
	for kind in kinds:
	if filename_template:
	context = {
	"platform": plat,
	"dev-dependencies": str(include_dev_dependencies).lower(),
	"input-hash": lockfile.metadata.content_hash,
	"version": distribution("conda_lock").version,
	"timestamp": datetime.datetime.now(datetime.timezone.utc).strftime(
	"%Y%m%dT%H%M%SZ"
	),
	}

[maresb]

Solving using conda-lock lock --files env1.yml env2.yml or conda-lock lock --files env1.yml env2.yml --extras my_extras will produce two different environment, the first with latest python (e.g. 3.13.2 as of today) the latter with latest python 3.12 (e.g. 3.12.9 as of today).

@qmarcou, I'm having trouble reproducing this claim (both on main and v2.5.7):

conda-lock lock --file env_core.yml --file env_extra.yml -p linux-64 --lockfile=lock-without-extras.yml
conda-lock lock --file env_core.yml --file env_extra.yml -p linux-64 --extras my_extras --lockfile=lock-with-extras.yml
md5sum lock-with-extras.yml
cat lock-without-extras.yml | sed s/lock-without-extras.yml/lock-with-extras.yml/g | md5sum

The checksum shows that the two lockfiles are identical (up to the filenames appearing in the metadata). The lockfile lists Python 3.12.0.

[qmarcou]

Ooww sorry I had not tested that example.
Do you get the same hash generating these two files?

conda-lock lock --file env_core.yml --file env_extra.yml -p linux-64 --lockfile=lock-without-extras1.yml
conda-lock lock --file env_core.yml  -p linux-64 --lockfile=lock-without-extras2.yml

This is just to test whether my example is ill-chosen, or if solving with extra categories but without using --extra switches has a different behavior than I expected

Thanks for this check!

[maresb]

The two commands you just provided lead to very different results because the second command doesn't know about jake or the version constraint on Python. This seems so far to me like expected behavior.

[qmarcou]

Ok, then I guess I am misunderstanding something.

Assuming the default behavior of lock is to not include any extra by default, I expected the two latter commands to output the same file (meaning both disregard jake and python version constraint). In a nutshell I expected that within the lock command the --extra switch controlled what dependencies were sent to the solver. I thought we were on the same page because you wrote:

I'm concerned that with this approach, some users may potentially suddenly inherit a bunch of new extras after upgrading conda-lock, introducing weird constraints coming from the extra dependencies, possibly rendering the environment unsolvable.

, and I do not understand how changing the extras after calling the solver may render the environment unsolvable (since it must have been already solved by the solver.

Given this:

@qmarcou, I'm having trouble reproducing this claim (both on main and v2.5.7):

conda-lock lock --file env_core.yml --file env_extra.yml -p linux-64 --lockfile=lock-without-extras.yml
conda-lock lock --file env_core.yml --file env_extra.yml -p linux-64 --extras my_extras --lockfile=lock-with-extras.yml
md5sum lock-with-extras.yml
cat lock-without-extras.yml | sed s/lock-without-extras.yml/lock-with-extras.yml/g | md5sum

The checksum shows that the two lockfiles are identical (up to the filenames appearing in the metadata). The lockfile lists Python 3.12.0.

Is the lock command --extra switch only used when --kind is not the unified lock format then (i.e. an output format that cannot handle categories/extras) ? In other words is it only used for the render step of the lock command?

[maresb]

I think I might see what the confusion is.

The "extra" terminology comes from the PyPI metadata as defined in pyproject.toml and similar, and is the PyPI analog of a "category".

So what I originally thought you meant is that if you run conda-lock lock -f pyproject.toml, and that pyproject.toml defines a bunch of extras, then they'll by default get ignored by conda-lock lock unless explicitly specified. Hence my concern. (Note: I have not verified if this is actually the current behavior.)

[qmarcou]

Thanks for the context, I did not have time to dig into this misunderstanding just yet. I will get back to you about this.

In the meantime I have submitted a first batch of changes to deprecate all --dev switches. As mentioned in the PR, there is no automated test to assert this deprecation functions as expected yet. It seems there were already no tests to regarding --dev and --extra switches behavior.

[maresb]

I did not have time to dig into this misunderstanding just yet. I will get back to you about this.

I should have provided more context. The "extras" terminology comes from:

1. A deprecated Poetry setting: https://python-poetry.org/docs/pyproject/#extras
2. The extras_require from SetupTools setup.cfg format that's been since replaced by optional-dependencies in the pyproject.toml format
3. The Provides-Extra field in the PyPI Metadata spec, i.e. what you find in the METADATA file in a wheel.

Thus it's sort of an outdated PyPI-centric terminology for categories.

For this reason, in qmarcou#2 I'm aiming to prefer the --category terminology.