Metis/Parmetis missing from rpath when CUDA is enabled

[balos1]

When linking to libsuperlu_dist.so, I have noticed that in the CUDA build metis and parmetis seem to be missing from the RPATH.

Without CUDA, objdump -p libsuperlu_dist.so shows metis and parmetis are there.

  RPATH                /usr/tce/packages/mvapich2/mvapich2-2.3-gcc-8.3.1/lib:/usr/WS1/sundials/spack_installs/v0.19.1/quartz/gcc-10.2.1/superlu-dist-8.1.1-jwm6ft5i3snhbyartfknpzaebjcmqisa/lib:/usr/WS1/sundials/spack_installs/v0.19.1/quartz/gcc-10.2.1/netlib-lapack-3.10.1-gwmchskxbvt6asi22dgiopv47c2ltdwr/lib64:/usr/WS1/sundials/spack_installs/v0.19.1/quartz/gcc-10.2.1/superlu-dist-8.1.1-jwm6ft5i3snhbyartfknpzaebjcmqisa/lib:/usr/WS1/sundials/spack_installs/v0.19.1/quartz/gcc-10.2.1/superlu-dist-8.1.1-jwm6ft5i3snhbyartfknpzaebjcmqisa/lib64:/usr/WS1/sundials/spack_installs/v0.19.1/quartz/gcc-10.2.1/parmetis-4.0.3-ewnu4kbnfqv7qa4vu3tpn2ymetdfpotg/lib:/usr/WS1/sundials/spack_installs/v0.19.1/quartz/gcc-10.2.1/metis-5.1.0-hh6vphqtfun2c7tc6kreiu5vimhohto2/lib:/usr/tce/packages/gcc/gcc-10.2.1/rh/usr/lib/gcc/x86_64-redhat-linux/10

With CUDA, they are missing from the RPATH

  RPATH                /usr/tce/packages/spectrum-mpi/ibm/spectrum-mpi-rolling-release/lib:/usr/WS1/sundials/spack_installs/v0.19.1/lassen/gcc-8.3.1/superlu-dist-8.1.1-5egktuoh4uoufrfle6ctg3qjvixajhw6/lib:/usr/WS1/sundials/spack_installs/v0.19.1/lassen/gcc-8.3.1/netlib-lapack-3.10.1-3hg7zjqlepgwv6nkhn7kgvgcfpiai554/lib64

I am building [email protected] via [email protected].

This results in an error when linking to libsuperlu_dist.so unless the dependent also links directly to metis/parmetis.

[liuyangzhuan]

I'm not sure why this is happening, the use of parmetis/metis in superlu defined by the cmakelist.txt and the spack package.py file has nothing to do with whether cuda is used or not. But from your output it seems that you are comparing with different compiler and MPIs as well? Would doing a static build solve the problem?

[balos1]

Yes, sorry, that was a CUDA build on a different machine than the non CUDA build. Here is a non CUDA build with the same compiler and MPI as with CUDA. Again you can see that Parmetis/metis are in the RPATH, so it does seem to be related to CUDA.

  RPATH                /usr/tce/packages/spectrum-mpi/ibm/spectrum-mpi-rolling-release/lib:/usr/WS1/sundials/spack_installs/v0.19.1/lassen/gcc-8.3.1/superlu-dist-8.1.1-tbl66wgihlgum2mwdig5dzr3clpmvl3t/lib:/usr/WS1/sundials/spack_installs/v0.19.1/lassen/gcc-8.3.1/netlib-lapack-3.10.1-3hg7zjqlepgwv6nkhn7kgvgcfpiai554/lib64:/usr/WS1/sundials/spack_installs/v0.19.1/lassen/gcc-8.3.1/superlu-dist-8.1.1-tbl66wgihlgum2mwdig5dzr3clpmvl3t/lib:/usr/WS1/sundials/spack_installs/v0.19.1/lassen/gcc-8.3.1/superlu-dist-8.1.1-tbl66wgihlgum2mwdig5dzr3clpmvl3t/lib64:/usr/WS1/sundials/spack_installs/v0.19.1/lassen/gcc-8.3.1/parmetis-4.0.3-3tcjaxanguyk6rdhrgckegyqcd7asyks/lib:/usr/WS1/sundials/spack_installs/v0.19.1/lassen/gcc-8.3.1/metis-5.1.0-ytubmn5mgiyrnbsjufexeh7xsf4prxez/lib:/usr/tce/packages/spectrum-mpi/spectrum-mpi-rolling-release-gcc-8.3.1/lib:/usr/tce/packages/gcc/gcc-8.3.1/rh/usr/lib/gcc/ppc64le-redhat-linux/8

[xiaoyeli]

This doesn't make sense. How is parmetis built? (dynamic or static?)

[balos1]

It is built as a shared library.

[xiaoyeli]

What's the difference between the 2 build scripts (without CUDA vs. with CUDA) ?

[balos1]

I did the build with Spack, so Ill have to dig up the logs and see what cmake was generated. This was done on Lassen at LLNL, which is Summit-like, you might be able to reproduce it there.