Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Name clashes error in "spack module lmod refresh", Spack Stack 1.9.0 on S4 #1543

Open
InnocentSouopgui-NOAA opened this issue Mar 5, 2025 · 9 comments
Open

Name clashes error in "spack module lmod refresh", Spack Stack 1.9.0 on S4 #1543

InnocentSouopgui-NOAA opened this issue Mar 5, 2025 · 9 comments
Labels
bug Something is not working

Comments

@InnocentSouopgui-NOAA
Copy link
Collaborator

Describe the bug
spack module lmod refresh step fails on S4 with Error: Name clashes detected in module files error message;
A couple of packages seem to be installed twice with the same version number

To Reproduce
Steps to reproduce the behavior:

  • Clone the branch release/1.9.0
  • Navigate to the cloned directory
  • Source the setup script source setup.sh
  • Create the environment: spack stack create env --site s4 --template unified-dev --name ue.s4.oneapi --compiler oneapi
  • Navigate to the newly created environment cd envs/ue.s4.oneapi
  • Run the concretize step: spack concretize --force 2>&1 | tee concretize.log
  • Run the Install step: spack install --source --verbose --fail-fast 2>&1 | tee install.log
  • Run the module refresh step: spack module lmod refresh

Expected behavior
The module refresh step should run smoothly without error message

System:
What system(s) are you running the code on? S4

Additional context

Below is the whole error message

Error: Name clashes detected in module files:

file: <PATH_TO_SPACK_STACK>/spack-stack/ss-1.9.0/envs/ue.s4.intel/install/modulefiles/intel/2021.10.0/bufr/12.1.0.lua
spec: bufr@=12.1.0%intel@=2021.10.0/pqqvn5b ~ipo+python+shared+utils build_system=cmake build_type=Release generator=make test_files=none arch=linux-rocky8-skylake
spec: bufr@=12.1.0%intel@=2021.10.0/3fv3qfa ~ipo+python+shared+utils build_system=cmake build_type=Release generator=make test_files=none arch=linux-rocky8-skylake_avx512

file: <PATH_TO_SPACK_STACK>/spack-stack/ss-1.9.0/envs/ue.s4.intel/install/modulefiles/intel-oneapi-mpi/2021.10.0-dazbmdo/intel/2021.10.0/bufr-query/0.0.4.lua
spec: bufr-query@=0.0.4%intel@=2021.10.0/x6mdnvj ~ipo+python build_system=cmake build_type=Release generator=make arch=linux-rocky8-skylake_avx512
spec: bufr-query@=0.0.4%intel@=2021.10.0/fespkhj ~ipo+python build_system=cmake build_type=Release generator=make arch=linux-rocky8-skylake_avx512

file: <PATH_TO_SPACK_STACK>/spack-stack/ss-1.9.0/envs/ue.s4.intel/install/modulefiles/intel-oneapi-mpi/2021.10.0-dazbmdo/intel/2021.10.0/esmf/8.6.1.lua
spec: esmf@=8.6.1%intel@=2021.10.0/qntcmd7 ~debug~external-lapack+external-parallelio+mpi+netcdf~pnetcdf+python+shared~xerces build_system=makefile esmf_comm=auto esmf_os=auto esmf_pio=auto patches=f63d405 snapshot=none arch=linux-rocky8-skylake
spec: esmf@=8.6.1%intel@=2021.10.0/oczi5mc ~debug~external-lapack+external-parallelio+mpi+netcdf~pnetcdf+python+shared~xerces build_system=makefile esmf_comm=auto esmf_os=auto esmf_pio=auto patches=f63d405 snapshot=none arch=linux-rocky8-skylake_avx512

file: <PATH_TO_SPACK_STACK>/spack-stack/ss-1.9.0/envs/ue.s4.intel/install/modulefiles/intel-oneapi-mpi/2021.10.0-dazbmdo/intel/2021.10.0/jedi-base-env/1.0.0.lua
spec: jedi-base-env@=1.0.0%intel@=2021.10.0/p2vkkkg +bufrquery+fftw+hdf4 build_system=bundle arch=linux-rocky8-skylake_avx512
spec: jedi-base-env@=1.0.0%intel@=2021.10.0/zlz4rft +bufrquery+fftw+hdf4 build_system=bundle arch=linux-rocky8-skylake_avx512

file: <PATH_TO_SPACK_STACK>/spack-stack/ss-1.9.0/envs/ue.s4.intel/install/modulefiles/intel-oneapi-mpi/2021.10.0-dazbmdo/intel/2021.10.0/mapl/2.53.0-esmf-8.6.1.lua
spec: mapl@=2.53.0%intel@=2021.10.0/oew7n6v ~debug+extdata2g~f2py+fargparse~ipo+pflogger~pfunit~shared+zstd build_system=cmake build_type=Release generator=make arch=linux-rocky8-skylake
spec: mapl@=2.53.0%intel@=2021.10.0/opuin4t ~debug+extdata2g~f2py+fargparse~ipo+pflogger~pfunit~shared+zstd build_system=cmake build_type=Release generator=make arch=linux-rocky8-skylake_avx512

file: <PATH_TO_SPACK_STACK>/spack-stack/ss-1.9.0/envs/ue.s4.intel/install/modulefiles/intel/2021.10.0/py-cftime/1.0.3.4.lua
spec: py-cftime@=1.0.3.4%intel@=2021.10.0/dzvgi7e  build_system=python_pip arch=linux-rocky8-skylake
spec: py-cftime@=1.0.3.4%intel@=2021.10.0/lbu7qhl  build_system=python_pip arch=linux-rocky8-skylake_avx512

file: <PATH_TO_SPACK_STACK>/spack-stack/ss-1.9.0/envs/ue.s4.intel/install/modulefiles/intel/2021.10.0/py-eccodes/1.5.0.lua
spec: py-eccodes@=1.5.0%intel@=2021.10.0/k4demqu  build_system=python_pip arch=linux-rocky8-skylake_avx512
spec: py-eccodes@=1.5.0%intel@=2021.10.0/dt2awcj  build_system=python_pip arch=linux-rocky8-skylake_avx512

file: <PATH_TO_SPACK_STACK>/spack-stack/ss-1.9.0/envs/ue.s4.intel/install/modulefiles/intel-oneapi-mpi/2021.10.0-dazbmdo/intel/2021.10.0/py-h5py/3.12.1.lua
spec: py-h5py@=3.12.1%intel@=2021.10.0/2oh3qnn ~mpi build_system=python_pip arch=linux-rocky8-skylake_avx512
spec: py-h5py@=3.12.1%intel@=2021.10.0/sq7b7jn ~mpi build_system=python_pip arch=linux-rocky8-skylake_avx512

file: <PATH_TO_SPACK_STACK>/spack-stack/ss-1.9.0/envs/ue.s4.intel/install/modulefiles/intel-oneapi-mpi/2021.10.0-dazbmdo/intel/2021.10.0/py-netcdf4/1.7.1.post2.lua
spec: py-netcdf4@=1.7.1.post2%intel@=2021.10.0/b7ly3rf ~mpi build_system=python_pip patches=255b5ae,2bce09b arch=linux-rocky8-skylake
spec: py-netcdf4@=1.7.1.post2%intel@=2021.10.0/fanxnys ~mpi build_system=python_pip patches=255b5ae,2bce09b arch=linux-rocky8-skylake_avx512

file: <PATH_TO_SPACK_STACK>/spack-stack/ss-1.9.0/envs/ue.s4.intel/install/modulefiles/intel/2021.10.0/py-pandas/2.2.3.lua
spec: py-pandas@=2.2.3%intel@=2021.10.0/hhvvyz2 ~excel~performance build_system=python_pip arch=linux-rocky8-skylake
spec: py-pandas@=2.2.3%intel@=2021.10.0/tnyendm ~excel~performance build_system=python_pip arch=linux-rocky8-skylake_avx512

file: <PATH_TO_SPACK_STACK>/spack-stack/ss-1.9.0/envs/ue.s4.intel/install/modulefiles/intel/2021.10.0/py-pyhdf/0.11.4.lua
spec: py-pyhdf@=0.11.4%intel@=2021.10.0/htohkub  build_system=python_pip arch=linux-rocky8-skylake_avx512
spec: py-pyhdf@=0.11.4%intel@=2021.10.0/3verqjw  build_system=python_pip arch=linux-rocky8-skylake_avx512

file: <PATH_TO_SPACK_STACK>/spack-stack/ss-1.9.0/envs/ue.s4.intel/install/modulefiles/intel/2021.10.0/py-pythran/0.16.1.lua
spec: py-pythran@=0.16.1%intel@=2021.10.0/26uogks  build_system=python_pip arch=linux-rocky8-skylake_avx512
spec: py-pythran@=0.16.1%intel@=2021.10.0/kb4ydjb  build_system=python_pip arch=linux-rocky8-skylake_avx512

file: <PATH_TO_SPACK_STACK>/spack-stack/ss-1.9.0/envs/ue.s4.intel/install/modulefiles/intel/2021.10.0/py-scipy/1.13.1.lua
spec: py-scipy@=1.13.1%intel@=2021.10.0/pjglr7i  build_system=python_pip patches=3720932 arch=linux-rocky8-skylake_avx512
spec: py-scipy@=1.13.1%intel@=2021.10.0/3v75ekn  build_system=python_pip patches=3720932 arch=linux-rocky8-skylake_avx512

file: <PATH_TO_SPACK_STACK>/spack-stack/ss-1.9.0/envs/ue.s4.intel/install/modulefiles/intel/2021.10.0/py-xarray/2024.7.0.lua
spec: py-xarray@=2024.7.0%intel@=2021.10.0/opls6au ~io~parallel~viz build_system=python_pip arch=linux-rocky8-skylake
spec: py-xarray@=2024.7.0%intel@=2021.10.0/bbqedvp ~io~parallel~viz build_system=python_pip arch=linux-rocky8-skylake_avx512

==> Error: Operation aborted
@InnocentSouopgui-NOAA InnocentSouopgui-NOAA added the bug Something is not working label Mar 5, 2025
@climbfuji
Copy link
Collaborator

Looks like you skipped the check for duplicate packages after concretizing?

@InnocentSouopgui-NOAA
Copy link
Collaborator Author

Looks like you skipped the check for duplicate packages after concretizing?

Spot on, I forgot the duplicate checking step.

I restarted the whole process and after concretizing I checked and found duplicates.

How do I eliminate those duplicates?

@climbfuji
Copy link
Collaborator

Looks like you skipped the check for duplicate packages after concretizing?

Spot on, I forgot the duplicate checking step.

I restarted the whole process and after concretizing I checked and found duplicates.

How do I eliminate those duplicates?

This is where the fun begins. First, you need to identify which package is installed twice and why. The ones you listed above with module name clashes all seem to be consequences of some lower-level package being built twice. Can you post the output from the check for duplicates here?

@InnocentSouopgui-NOAA
Copy link
Collaborator Author

@climbfuji,
Below is the output from the check for duplicates

More details about the problem. I got this problem trying to have intel classic in addition to oneapi. Initially I said that it was oneapi, but after double check, it is only intel classic (2023) that is causing duplicates. oneapi 2024 is installing without problem.

===
Duplicates found!
ztfs3z6  [email protected]%[email protected]  fflags='-diag-disable=10448'  ~ipo+python+shared+utils  build_system=cmake  build_type=Release  generator=make  test_files=none  arch=linux-rocky8-skylake
ckweycq  [email protected]%[email protected]  fflags='-diag-disable=10448'  ~ipo+python+shared+utils  build_system=cmake  build_type=Release  generator=make  test_files=none  arch=linux-rocky8-skylake
muyrhkt  [email protected]%[email protected]  fflags='-diag-disable=10448'  ~ipo+python  build_system=cmake  build_type=Release  generator=make  arch=linux-rocky8-skylake
sedcosf  [email protected]%[email protected]  fflags='-diag-disable=10448'  ~ipo+python  build_system=cmake  build_type=Release  generator=make  arch=linux-rocky8-skylake
7ts5miz  [email protected]%[email protected]  fflags='-diag-disable=10448'  +fix~ipo  build_system=cmake  build_type=Release  generator=make  arch=linux-rocky8-skylake
owhziaq  [email protected]%[email protected]  fflags='-diag-disable=10448'  +fix~ipo  build_system=cmake  build_type=Release  generator=make  arch=linux-rocky8-skylake
w3kzm5w  [email protected]%[email protected]  fflags='-diag-disable=10448'  +fix~ipo  build_system=cmake  build_type=Release  generator=make  arch=linux-rocky8-skylake
qgyl3qi  [email protected]%[email protected]  fflags='-diag-disable=10448'  +fix~ipo  build_system=cmake  build_type=Release  generator=make  arch=linux-rocky8-skylake
q4bgy7a  [email protected]%[email protected]  fflags='-diag-disable=10448'  +big_endian~little_endian+netcdf~testfiles  build_system=generic  arch=linux-rocky8-skylake
5dzsuby  [email protected]_emc%[email protected]  fflags='-diag-disable=10448'  +big_endian~little_endian+netcdf  build_system=generic  arch=linux-rocky8-skylake
w7xdnib  [email protected]%[email protected]  cxxflags='-fp-model  precise'  fflags='-diag-disable=10448  -fp-model  precise'  ~debug~external-lapack+external-parallelio+mpi+netcdf~pnetcdf+python+shared~xerces  build_system=makefile  esmf_comm=auto  esmf_os=auto  esmf_pio=auto  patches=f63d405  snapshot=none  arch=linux-rocky8-skylake
3vye3a2  [email protected]%[email protected]  cxxflags='-fp-model  precise'  fflags='-diag-disable=10448  -fp-model  precise'  ~debug~external-lapack+external-parallelio+mpi+netcdf~pnetcdf+python+shared~xerces  build_system=makefile  esmf_comm=auto  esmf_os=auto  esmf_pio=auto  patches=f63d405  snapshot=none  arch=linux-rocky8-skylake
25dlh3n  [email protected]%[email protected]  cxxflags='-fp-model  precise'  fflags='-diag-disable=10448  -fp-model  precise'  ~debug~external-lapack+external-parallelio+mpi+netcdf~pnetcdf+python+shared~xerces  build_system=makefile  esmf_comm=auto  esmf_os=auto  esmf_pio=auto  patches=f63d405  snapshot=none  arch=linux-rocky8-skylake
67wojo3  [email protected]%[email protected]  fflags='-diag-disable=10448'  +bufrquery+fftw+hdf4  build_system=bundle  arch=linux-rocky8-skylake
2crqz3g  [email protected]%[email protected]  fflags='-diag-disable=10448'  +bufrquery+fftw+hdf4  build_system=bundle  arch=linux-rocky8-skylake
sw2u2qc  [email protected]%[email protected]  fflags='-diag-disable=10448'  ~debug+extdata2g~f2py+fargparse~ipo+pflogger~pfunit~shared+zstd  build_system=cmake  build_type=Release  generator=make  arch=linux-rocky8-skylake
qwtu4km  [email protected]%[email protected]  fflags='-diag-disable=10448'  ~debug+extdata2g~f2py+fargparse~ipo+pflogger~pfunit~shared+zstd  build_system=cmake  build_type=Release  generator=make  arch=linux-rocky8-skylake
syz7rcv  [email protected]%[email protected]  fflags='-diag-disable=10448'  build_system=python_pip  arch=linux-rocky8-skylake
sd6mvkf  [email protected]%[email protected]  fflags='-diag-disable=10448'  build_system=python_pip  arch=linux-rocky8-skylake
zki32tu  [email protected]%[email protected]  fflags='-diag-disable=10448'  build_system=python_pip  arch=linux-rocky8-skylake
x6h3zkh  [email protected]%[email protected]  fflags='-diag-disable=10448'  build_system=python_pip  arch=linux-rocky8-skylake
pbnsje7  [email protected]%[email protected]  fflags='-diag-disable=10448'  ~mpi  build_system=python_pip  arch=linux-rocky8-skylake
atjfrm6  [email protected]%[email protected]  fflags='-diag-disable=10448'  ~mpi  build_system=python_pip  arch=linux-rocky8-skylake
yedvihz  [email protected]%[email protected]  fflags='-diag-disable=10448'  ~mpi  build_system=python_pip  patches=255b5ae,2bce09b  arch=linux-rocky8-skylake
jdvddpl  [email protected]%[email protected]  fflags='-diag-disable=10448'  ~mpi  build_system=python_pip  patches=255b5ae,2bce09b  arch=linux-rocky8-skylake
l72qunt  [email protected]%[email protected]  fflags='-diag-disable=10448'  build_system=python_pip  patches=873745d  arch=linux-rocky8-skylake
mn6nkkx  [email protected]%[email protected]  fflags='-diag-disable=10448'  build_system=python_pip  patches=873745d  arch=linux-rocky8-skylake
a33pp4l  [email protected]%[email protected]  fflags='-diag-disable=10448'  ~excel~performance  build_system=python_pip  arch=linux-rocky8-skylake
vbnkfbp  [email protected]%[email protected]  fflags='-diag-disable=10448'  ~excel~performance  build_system=python_pip  arch=linux-rocky8-skylake
4aapzc7  [email protected]%[email protected]  fflags='-diag-disable=10448'  build_system=python_pip  arch=linux-rocky8-skylake
amygpdi  [email protected]%[email protected]  fflags='-diag-disable=10448'  build_system=python_pip  arch=linux-rocky8-skylake
5cfupjb  [email protected]%[email protected]  fflags='-diag-disable=10448'  build_system=python_pip  arch=linux-rocky8-skylake
zvmjt63  [email protected]%[email protected]  fflags='-diag-disable=10448'  build_system=python_pip  arch=linux-rocky8-skylake
porwmra  [email protected]%[email protected]  fflags='-diag-disable=10448'  build_system=python_pip  patches=3720932  arch=linux-rocky8-skylake
qmsms2b  [email protected]%[email protected]  fflags='-diag-disable=10448'  build_system=python_pip  patches=3720932  arch=linux-rocky8-skylake
6szschz  [email protected]%[email protected]  fflags='-diag-disable=10448'  ~io~parallel~viz  build_system=python_pip  arch=linux-rocky8-skylake
vlijiae  [email protected]%[email protected]  fflags='-diag-disable=10448'  ~io~parallel~viz  build_system=python_pip  arch=linux-rocky8-skylake

@climbfuji
Copy link
Collaborator

I see two different versions of py-numpy, this must be the problem.

l72qunt  [email protected]%[email protected]  fflags='-diag-disable=10448'  build_system=python_pip  patches=873745d  arch=linux-rocky8-skylake
mn6nkkx  [email protected]%[email protected]  fflags='-diag-disable=10448'  build_system=python_pip  patches=873745d  arch=linux-rocky8-skylake

I guess there are some conflicts/limitations on which versions of numpy work with all packages in the unified environment, in addition to conflicts of certain versions of numpy with Intel classic. Please try to figure this out with JCSDA.

@rickgrubin-noaa
Copy link
Collaborator

Almost certainly it's related to py-numpy; try

  # To avoid duplicate packages
  py-numpy:
    require:
    - '@1.26'

in packages.yaml

@InnocentSouopgui-NOAA
Copy link
Collaborator Author

When I fully remove py-numpy from my package file, the concretization fails with the message cannot satisfy a requirement for package 'py-numpy'.

Let me add the content of packages_intel.yaml if it can help spot more problems. ; I am using split package.yaml to support oneapi and intel classic

packages:
  all:
    compiler:: [[email protected]]
    providers:
      mpi:: [[email protected]]
      # Remove the next three lines to switch to intel-oneapi-mkl
      blas:: [openblas]
      fftw-api:: [fftw]
      lapack:: [openblas]
    # spack concretizer gets confused and wants an AVX512 architecture for some packages
    require:
      - 'arch=linux-rocky8-skylake'

### MPI, Python, MKL
  mpi:
    buildable: False
  intel-oneapi-mpi:
    externals:
    - spec: [email protected]%[email protected]
      prefix: /opt/intel/oneapi/2023.2
      modules:
      - intel/2023.2
  intel-oneapi-mkl:
    # Remove buildable: False and uncomment externals section below to use intel-oneapi-mkl
    buildable: False
    #externals:
    #- spec: [email protected]%[email protected]
    #  prefix: /opt/intel/oneapi/2023.2
    #  modules:
    #  - intel/2023.2
  # Remove this section to switch to intel-oneapi-mkl
  ectrans:
    require::
    - '@1.2.0 ~mkl +fftw'
  gsibec:
    require::
    - '@1.2.1 ~mkl'
  py-numpy:
    require::
    - '^openblas'
  # *DH
  zlib-ng:
    require:
      - '~shared'

@InnocentSouopgui-NOAA
Copy link
Collaborator Author

I removed intel-oneapi-mkl, ectrans, gsibec, py-numpy and the concretization went smoothly.
The only duplicates were crtm and esmf which is expected if I am not mistaking.

@climbfuji
Copy link
Collaborator

Please be careful. We added those settings for a reason. Please try what @rickgrubin-noaa suggested, namely adding

    - '@1.26'

to the packages_intel.yaml file:

  py-numpy:
    require::
    - '@1.26'
    - '^openblas'

I am still hopeful that NOAA one day will tell us that we can switch back to using Intel MKL with the Intel Classic and LLVM compilers. Then all of this wouldn't be necessary.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something is not working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@climbfuji @InnocentSouopgui-NOAA @rickgrubin-noaa