Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ATS-9990: SPEC - expanse/0.17.3.b - User requested Intel TBB installation #127

Open
mkandes opened this issue Aug 27, 2024 · 5 comments
Open

ATS-9990: SPEC - expanse/0.17.3.b - User requested Intel TBB installation #127

mkandes opened this issue Aug 27, 2024 · 5 comments

Comments

@mkandes
Copy link
Member

mkandes commented Aug 27, 2024

https://access-ci.atlassian.net/browse/ATS-9980
@mkandes
Copy link
Member Author

mkandes commented Aug 27, 2024

Before creating a new featture branch to deploy Intel TBB, you need to resync the deployment branch of your fork with the sdsc/spack upstream.

[mkandes@login01 spack]$ pwd
/home/mkandes/software/spack/repositories/mkandes/spack
[mkandes@login01 spack]$ git log -1
commit 2eccf0894939f5e1f34d8bde648884b6ad5f0d79 (HEAD -> sdsc-0.17.3-gh-125-spec-exp-cpu-pgi-netcdf, origin/sdsc-0.17.3-gh-125-spec-exp-cpu-pgi-netcdf)
Author: Marty Kandes <[email protected]>
Date:   Fri Aug 16 11:33:00 2024 -0700

    Add [email protected] % [email protected] + netcdf-* to exp/0.17.3/cpu/b
    
    We are deploying [email protected] to exp/0.17.3/cpu/b as a compiler option to
    support a specific use case [1], where the end user can currently only
    compile their custom code with PGI-based compilers and a set of
    compatible NetCDF libraries.
    
    [1] https://github.com/sdsc/spack/issues/125
[mkandes@login01 spack]$ git branch
  sdsc-0.17.3
  sdsc-0.17.3-gh-124-inst-exp-cpu-dev
* sdsc-0.17.3-gh-125-spec-exp-cpu-pgi-netcdf
[mkandes@login01 spack]$ git checkout sdsc-0.17.3
Switched to branch 'sdsc-0.17.3'
Your branch is up to date with 'origin/sdsc-0.17.3'.
[mkandes@login01 spack]$

@mkandes
Copy link
Member Author

mkandes commented Aug 27, 2024

Resyncing from upstream ...

[mkandes@login01 spack]$ git pull upstream sdsc-0.17.3
From https://github.com/sdsc/spack
 * branch                  sdsc-0.17.3 -> FETCH_HEAD
Updating 3cbfd1ba45..a3029ae4c3
Fast-forward
 .../cpu/b/specs/[email protected]   | 653 ++++++++++++++++++
 .../expanse/0.17.3/cpu/b/specs/[email protected]     |  68 ++
 .../[email protected]/[email protected] | 662 ++++++++++++++++++
 .../cpu/b/specs/[email protected]/[email protected]      |   5 +-
 .../0.17.3/cpu/b/specs/[email protected]/[email protected]   |   2 +-
 .../[email protected]            | 742 +++++++++++++++++++++
 .../[email protected]/[email protected]/[email protected]   |  81 +++
 .../specs/[email protected]/[email protected]/[email protected] |   2 +-
 .../specs/[email protected]/[email protected]/[email protected]  |   2 +-
 .../cpu/b/specs/[email protected]     | 643 ++++++++++++++++++
 .../sdsc/expanse/0.17.3/cpu/b/specs/[email protected]  |  72 ++
 .../[email protected]/[email protected]     | 695 +++++++++++++++++++
 .../0.17.3/cpu/b/specs/[email protected]/[email protected]   |  68 ++
 .../[email protected]/[email protected]  | 704 +++++++++++++++++++
 .../cpu/b/specs/[email protected]/[email protected]       |  68 ++
 .../[email protected]       | 669 +++++++++++++++++++
 .../cpu/b/specs/[email protected]/[email protected] |  67 ++
 .../sdsc/expanse/0.17.3/cpu/b/yamls/compilers.yaml |  16 +
 18 files changed, 5213 insertions(+), 6 deletions(-)
 create mode 100644 etc/spack/sdsc/expanse/0.17.3/cpu/b/specs/[email protected]
 create mode 100644 etc/spack/sdsc/expanse/0.17.3/cpu/b/specs/[email protected]
 create mode 100644 etc/spack/sdsc/expanse/0.17.3/cpu/b/specs/[email protected]/[email protected]
 create mode 100644 etc/spack/sdsc/expanse/0.17.3/cpu/b/specs/[email protected]/[email protected]/[email protected]
 create mode 100644 etc/spack/sdsc/expanse/0.17.3/cpu/b/specs/[email protected]/[email protected]/[email protected]
 create mode 100644 etc/spack/sdsc/expanse/0.17.3/cpu/b/specs/[email protected]
 create mode 100644 etc/spack/sdsc/expanse/0.17.3/cpu/b/specs/[email protected]
 create mode 100644 etc/spack/sdsc/expanse/0.17.3/cpu/b/specs/[email protected]/[email protected]
 create mode 100644 etc/spack/sdsc/expanse/0.17.3/cpu/b/specs/[email protected]/[email protected]
 create mode 100644 etc/spack/sdsc/expanse/0.17.3/cpu/b/specs/[email protected]/[email protected]
 create mode 100644 etc/spack/sdsc/expanse/0.17.3/cpu/b/specs/[email protected]/[email protected]
 create mode 100644 etc/spack/sdsc/expanse/0.17.3/cpu/b/specs/[email protected]/[email protected]
 create mode 100644 etc/spack/sdsc/expanse/0.17.3/cpu/b/specs/[email protected]/[email protected]
[mkandes@login01 spack]$ git log
commit a3029ae4c3bc806522bf34949c3e4ed995e6206b (HEAD -> sdsc-0.17.3, upstream/sdsc-0.17.3)
Author: Marty Kandes <[email protected]>
Date:   Sat Aug 17 07:58:30 2024 -0700

    [email protected] % [email protected] +mpi +openmp failed builds
    
    Unfortauntely, I have not been able to resolve this issue yet. It's a
    bit odd that enabling +openmp causes the package build to fail when we
    already have successful builds with +mpi ~openmp. We will need to
    investigate this further in the future. For the time being, I will
    invetigate alternative compiler, MPI, and FFTW options for QE.

commit 5c15c7b17659e27a70d0dacf62449b589f65edca
Author: Marty Kandes <[email protected]>
Date:   Sat Aug 17 07:19:07 2024 -0700

    Add [email protected] % [email protected] +openmp to exp/0.17.3/cpu/b
    
    Adding +openmp-enabled FFTW3 libraries to the production instances in an
    attempt to satisfy a requirement for the latest versions of Quantum
    Espresso, which appears to require a multithreaded FFTW-compatible
    libary.  For example, this is the build error I'm currently running into
[mkandes@login01 spack]$ git push
Username for 'https://github.com': ^C     
[mkandes@login01 spack]$ git push
Username for 'https://github.com': mkandes
Password for 'https://[email protected]': 
Total 0 (delta 0), reused 0 (delta 0), pack-reused 0
To https://github.com/mkandes/spack.git
   3cbfd1ba45..a3029ae4c3  sdsc-0.17.3 -> sdsc-0.17.3
[mkandes@login01 spack]$

@mkandes
Copy link
Member Author

mkandes commented Aug 27, 2024

Create new feature branch for testing spec installation ...

[mkandes@login01 spack]$ git branch
* sdsc-0.17.3
  sdsc-0.17.3-gh-124-inst-exp-cpu-dev
  sdsc-0.17.3-gh-125-spec-exp-cpu-pgi-netcdf
[mkandes@login01 spack]$ git checkout -b sdsc-0.17.3-gh-127-spec-exp-cpu-intel-tbb
Switched to a new branch 'sdsc-0.17.3-gh-127-spec-exp-cpu-intel-tbb'
[mkandes@login01 spack]$ git branch
  sdsc-0.17.3
  sdsc-0.17.3-gh-124-inst-exp-cpu-dev
  sdsc-0.17.3-gh-125-spec-exp-cpu-pgi-netcdf
* sdsc-0.17.3-gh-127-spec-exp-cpu-intel-tbb
[mkandes@login01 spack]$

mkandes added a commit to mkandes/spack that referenced this issue Aug 27, 2024
@mkandes
Add [email protected] % [email protected] to exp/0.17.3/cpu/b
8cac7a7 
This explicit installation of [email protected] % [email protected] is to
satisfy a user request [1]. Note, however, it also appears as if the TBB
libraries are already included inplicitly within the existing Intel MKL
variants already deployed on Expanse.

[1] sdsc#127
@mkandes mkandes mentioned this issue Aug 27, 2024
Merged
@mkandes
Copy link
Member Author

mkandes commented Aug 27, 2024

Deploying [email protected] to expanse/0.17.3/cpu/b ...

[spack_cpu@login01 ~]$ !7463
srun --partition=ind-shared --reservation=root_73  --account=use300 --nodes=1 --nodelist=exp-15-56 --ntasks-per-node=1 --cpus-per-task=16 --mem=32G --time=12:00:00 --pty --wait=0 /bin/bash
[spack_cpu@exp-15-56 ~]$ git log -1
fatal: not a git repository (or any parent up to mount point /home)
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
[spack_cpu@exp-15-56 ~]$ cd /cm/shared/apps/spack/0.17.3/cpu/b
[spack_cpu@exp-15-56 b]$ git log -1
commit a3029ae4c3bc806522bf34949c3e4ed995e6206b (HEAD -> sdsc-0.17.3, origin/sdsc-0.17.3, origin/HEAD)
Author: Marty Kandes <[email protected]>
Date:   Sat Aug 17 07:58:30 2024 -0700

    [email protected] % [email protected] +mpi +openmp failed builds
    
    Unfortauntely, I have not been able to resolve this issue yet. It's a
    bit odd that enabling +openmp causes the package build to fail when we
    already have successful builds with +mpi ~openmp. We will need to
    investigate this further in the future. For the time being, I will
    invetigate alternative compiler, MPI, and FFTW options for QE.
[spack_cpu@exp-15-56 b]$ git fetch origin
remote: Enumerating objects: 16, done.
remote: Counting objects: 100% (16/16), done.
remote: Compressing objects: 100% (6/6), done.
Unpacking objects: 100% (16/16), 3.03 KiB | 119.00 KiB/s, done.
remote: Total 16 (delta 7), reused 15 (delta 7), pack-reused 0 (from 0)
From https://github.com/sdsc/spack
   a3029ae4c3..7db33edafe  sdsc-0.17.3 -> origin/sdsc-0.17.3
[spack_cpu@exp-15-56 b]$ git log -1
commit a3029ae4c3bc806522bf34949c3e4ed995e6206b (HEAD -> sdsc-0.17.3)
Author: Marty Kandes <[email protected]>
Date:   Sat Aug 17 07:58:30 2024 -0700

    [email protected] % [email protected] +mpi +openmp failed builds
    
    Unfortauntely, I have not been able to resolve this issue yet. It's a
    bit odd that enabling +openmp causes the package build to fail when we
    already have successful builds with +mpi ~openmp. We will need to
    investigate this further in the future. For the time being, I will
    invetigate alternative compiler, MPI, and FFTW options for QE.
[spack_cpu@exp-15-56 b]$ git pull
Updating a3029ae4c3..7db33edafe
Fast-forward
 activate-shared-spack-instance.sh                  |  2 +-
 configure-shared-spack-instance.sh                 |  2 +-
 .../specs/[email protected]/[email protected] |  2 +-
 .../b/specs/[email protected]/[email protected]   | 73 ++++++++++++++++++++++
 4 files changed, 76 insertions(+), 3 deletions(-)
 mode change 100644 => 100755 configure-shared-spack-instance.sh
 create mode 100644 etc/spack/sdsc/expanse/0.17.3/cpu/b/specs/[email protected]/[email protected]
[spack_cpu@exp-15-56 b]$ cd etc/spack/sdsc/expanse/0.17.3/cpu/b/specs/
[spack_cpu@exp-15-56 specs]$ cd [email protected]/
[spack_cpu@exp-15-56 [email protected]]$ sbatch [email protected] 
sbatch: error: Batch job submission failed: Requested reservation is invalid
[spack_cpu@exp-15-56 [email protected]]$ vi [email protected] 
[spack_cpu@exp-15-56 [email protected]]$ sbatch [email protected] 
Submitted batch job 33452916
[spack_cpu@exp-15-56 [email protected]]$ squeue -u $USER
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
          33452916 ind-share intel-tb spack_cp  R       0:04      1 exp-15-56
          33452904 ind-share     bash spack_cp  R       1:59      1 exp-15-56
[spack_cpu@exp-15-56 [email protected]]$

@mkandes
Copy link
Member Author

mkandes commented Aug 27, 2024

Deployed successfully ...

[mkandes@login02 ~]$ module load intel/19.1.3.304
[mkandes@login02 ~]$ module avail

 /cm/shared/apps/spack/0.17.3/cpu/b/share/spack/lmod/linux-rocky8-x86_64/intel/19.1.3.304 
   arpack-ng/3.8.0/ndrjsnd          netcdf-cxx4/4.3.1/d5kubqf
   cmake/3.21.4/bpzre3q             netcdf-fortran/4.5.3/hg7lhvi
   eigen/3.4.0/w2iz4rf              netlib-lapack/3.9.1/64ytaps
   fftw/3.3.10/jq4mbmk              openblas/0.3.18/sctoo2e-omp
   gsl/2.7/ix3kgnc                  openblas/0.3.18/xxq5a3j      (D)
   hdf5/1.10.7/butdr5c              openmpi/4.1.3/luhyajc
   hypre/2.23.0/w7eerxg             papi/6.0.0.1/tbdvjsm
   intel-mkl/2020.4.304/vg6aq26     petsc/3.16.1/ax5r6e3
   intel-mpi/2019.10.317/ezrfjne    sprng/5.0/u2op57v
   intel-tbb/2020.3/lfesfxm         superlu/5.3.0/tycq2th

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@mkandes