Feature request: disconnect package repository better from stack provisioning

[wdconinc]

Related to AIDASoft/DD4hep#1379, I wanted to outline some other EIC frustrations in the interest of resolving them. (Due to teaching I haven't been able to attend any North America-friendly key4HEP meetings all fall term.)

I think some of the issues we experience downstream in EIC are ultimately caused by the dual role of this repository: it is both a place where package recipes are collected as an external repository, and it is the place where the key4hep stack is maintained and tested (including all the tweaks needed for such). This dual role leads to issues, the main of which are caused by (ref):

• tests here assume that they are running against .last_commit, not a released spack version,
• tests here use a spack with .cherry-pick added to it, including from personal repos.

At CHEP we discussed (@jmcarcell) why we cannot just use the key4hep stack, or why we don't integrate our packages in key4hep-spack (since we already use k4fwcore, k4actstracking). The reason is that we have not found it feasible to use a released version of spack with the key4hep-spack repository. More often than workable, the .last_commit is a requirement for using this repository (e.g. it imposes dependencies and conflicts that result in unsolvable environments when using the last released version of spack). The same thing happens with the .cherry-pick where additions there result in unsolvable environments. In some cases, the updates are really hard to disentange with very limited information included in PRs with many changes (e.g. #648).

So, we are now reducing our risk by using the key4hep-stack using hard-pinned commits only, and we only update it rarely, driving us further away from key4hep-spack.

What I wish could happen is that the key4hep-spack repository has a policy of supporting the last spack release, without needing to jump to a specific develop commit or cherry-pick some other commits. Those cherry-picks should not be requirements for the repository, even if you want to use last-commit or cherry-picks (as we also do) in building the key4hep software stack.

Once there is key4hep-spack repository testing here against v0.23.0 (or whatever thellatest release is), then the unsolvable environment issues (which are related to the backwards incompatibilities in various components, starting from the podio v1 transition upwards) would become manifest here as well.

While the title seems to indicate the need for splitting this repository, that's not what I am suggestion. I'm just calling for independently testing both aspects.

cc: @jmcarcell @andresailer @tmadlener

[andresailer]

Hi @wdconinc

Could you let us know which pieces from this key4hep-spack repository you are currently using?
We are investigating provisioning of the key4hep deployments with a different tool at the moment, which would lead to the desired disconnect as well.

[wdconinc]

Could you let us know which pieces from this key4hep-spack repository you are currently using?

I verified that it is just k4fwcore and k4actstracking.

[tmadlener]

IIUC the major grievances correctly, the minimal solution would be to ensure that k4fwcore and k4actstracking build on (a set of) the latest(?) spack release(s) without any cherry-picks or special fixes, right? Would the latest tags (and maybe main) be sufficient for this, or would you want to have more versions?

I am going out on a limb here, but I think that should be somewhat straight forward to setup via github actions that complete in a reasonable amount of time. What I think should be possible (correct my if I am wrong):

• checkout the desired spack version
• add the key4hep release on cvmfs as an upstream (to cover hopefully almost all of the dependencies with their latest tag).
• Build k4fwcore and k4actstracking on top of that (probably using the common packages.yaml)

Am I missing something obvious here?