Orion & Hercules module config issue for spack-stack-1.9.0

[AlexanderRichert-NOAA]

Describe the bug
Using the spack-managed-x86-64_v3/v1.0 module on Orion & Hercules is problematic because the "pushenv" undoes spack-stack's changes to $MODULEPATH, making our stack-oneapi module unable to load. For now on Orion, I've modified /apps/contrib/spack-stack/spack-stack-1.9.0/envs/ue-oneapi-2024.2.1/install/modulefiles/oneapi/2024.2.1/stack-intel-oneapi-mpi/2021.13.lua and /apps/contrib/spack-stack/spack-stack-1.9.0/envs/ue-oneapi-2024.2.1/install/modulefiles/Core/stack-oneapi/2024.2.1.lua to prepend $MODULEPATH using the same set of paths that would have come from spack-managed-x86-64_v3/v1.0, and this seems to work. We should bake the modified MODULEPATH into the site configs.

[rickgrubin-noaa]

@AlexanderRichert-NOAA the same will be required for hercules; just verified this behavior.

% module use /apps/contrib/spack-stack/spack-stack-1.9.0/envs/ue-oneapi-2024.2.1/install/modulefiles/Core
% module load stack-oneapi/2024.2.1
Lmod has detected the following error:  The following module(s) are unknown: "intel-oneapi-compilers/2024.2.1"

Please check the spelling or version number. Also try "module spider ..."
It is also possible your cache file is out-of-date; it may help to try:
  $ module --ignore_cache load "intel-oneapi-compilers/2024.2.1"

Also make sure that all modulefiles written in TCL start with the string #%Module

Executing this command requires loading "intel-oneapi-compilers/2024.2.1" which failed while processing the following
module(s):

    Module fullname        Module Filename
    ---------------        ---------------
    stack-oneapi/2024.2.1  /apps/contrib/spack-stack/spack-stack-1.9.0/envs/ue-oneapi-2024.2.1/install/modulefiles/Core/stack-oneapi/2024.2.1.lua
While processing the following module(s):
    Module fullname        Module Filename
    ---------------        ---------------
    stack-oneapi/2024.2.1  /apps/contrib/spack-stack/spack-stack-1.9.0/envs/ue-oneapi-2024.2.1/install/modulefiles/Core/stack-oneapi/2024.2.1.lua

Would this bug be mitigated if the spack-managed-x86-64_v3/v1.0 module used prepend_path instead of pushenv? It would seem so, based on the change you made.

Perhaps a request for a change to this module on both hosts is worth a try?

[AlexanderRichert-NOAA]

That's fine with me, I suspect they may have some reservations if the purpose of the module is to have a separate clean module hierarchy, but it can't hurt to consult with them and see what they say.

[AlexanderRichert-NOAA]

Thanks for checking Hercules, I'll update the description.

[rickgrubin-noaa]

@AlexanderRichert-NOAA I'm having conversations with hercules / orion sysadmins about changing the spack-managed-x86-64_v3/v1.0 module file; they outlined reasons for using pushenv() and don't seem amenable to changing the file.

In the meantime, the following line:

prepend_path("MODULEPATH", "/apps/spack-managed-x86_64_v3-v1.0/modulefiles/Core:/apps/other/modulefiles:/apps/containers/modulefiles:/apps/licensed/modulefiles")

is missing on hercules from this stanza in the stack-oneapi/2024.2.1 module file (it's present on orion):

-- prerequisite modules
--load("spack-managed-x86-64_v3")
prepend_path("MODULEPATH", "/apps/spack-managed-x86_64_v3-v1.0/modulefiles/Core:/apps/other/modulefiles:/apps/containers/modulefiles:/apps/licensed/modulefiles")
load("intel-oneapi-compilers/2024.2.1")
--prereq("spack-managed-x86-64_v3")
prereq("intel-oneapi-compilers/2024.2.1")

The stack won't load without it; I don't know the proper incantation as user role-epic to write a file on what's reported as a read-only filesystem.

[AlexanderRichert-NOAA]

I didn't make the change there yet. I'll do so shortly, and I'll follow up over email re: modifying the stacks on Orion & Hercules.