Device to Device transfers don't work with OpenMPI + LinkX provider on AMD GPUs

[teodorhk]

I do not know if this is a bug, if I'm doing something wrong, or if this is simply not (yet?) expected to work. I understand that the LinkX provider is experimental - just trying to make it work and would appreciate any insights.

Describe the bug
OpenMPI with shm+cxi:lnx fails to perform Device - Device transfers on LUMI system (AMD GPUs) with OSU benchmark. Host - Host transfers work as expected for intra- and inter-node transfers.

To Reproduce
I have compiled libfabric 2.0.0 on LUMI, with all the new https://github.com/HewlettPackard/shs-* repositories for cassini headers, cxi driver, and libcxi. I have then compiled OpenMPI against that libfabric and I'm trying to use the LinxX provider. Things work for Host - Host transfers (OSU), both for intra-node communication:

export FI_LNX_PROV_LINKS=shm+cxi
export FI_SHM_USE_XPMEM=1
mpirun --mca opal_common_ofi_provider_include "shm+cxi:lnx" -np 2 -map-by numa ./osu_bibw -m 131072: H H

# OSU MPI Bi-Directional Bandwidth Test v7.4
# Datatype: MPI_CHAR.
# Size      Bandwidth (MB/s)
131072              79654.27
262144              69814.38
524288              75581.93
1048576             75997.54
2097152             78447.48
4194304             77570.98

and inter-node communication:

mpirun --mca opal_common_ofi_provider_include "shm+cxi:lnx" -np 2 -map-by node ./osu_bibw -m 131072: H H

# OSU MPI Bi-Directional Bandwidth Test v7.4
# Datatype: MPI_CHAR.
# Size      Bandwidth (MB/s)
131072              28439.17
262144              32406.37
524288              33389.49
1048576             33877.41
2097152             34119.12
4194304             34305.67

but the Device - Device transfers fail both for intra- and inter-node

export FI_LNX_PROV_LINKS=shm+cxi
mpirun --mca opal_common_ofi_provider_include "shm+cxi:lnx" -np 2 -map-by numa ./osu_bibw -m 131072: D D

# OSU MPI-ROCM Bi-Directional Bandwidth Test v7.4
# Datatype: MPI_CHAR.
# Size      Bandwidth (MB/s)
--------------------------------------------------------------------------
Open MPI failed to register your buffer.
This error is fatal, your job will abort

  Buffer Type: rocm
  Buffer Address: 0x154beaa00000
  Buffer Length: 131072
  Error: Required key not available (4294967030)
--------------------------------------------------------------------------

Looking at OpenMPI code, a call to fi_mr_regattr fails with the above message (probably FI_EKEYREJECTED? the cryptic 4294967030 is in fact -9)

Environment:
This is on Cray LUMI system, but the cxi driver is taken from https://github.com/HewlettPackard/shs-* repos.

[hppritcha]

This is probably related to open-mpi/ompi#11076

[angainor]

@hppritcha Ah, thanks! do you think it's enough to apply open-mpi/ompi#12290 ? I can check with master

[angainor]

@hppritcha Yes, that was it 👍 It seems the PR was merged into main, but not 5.0. I modified 5.0.6 manually and now all works fine! Thanks for the hint!

[hppritcha]

maybe add a patch to the Open mpi spackage for PR 12290?

[hppritcha]

sorry i'm going back and forth between spack world and this one and got things mixed up.

[angainor]

Should I report this in the OpenMPI issues? I guess it belongs there and not really here..

[hppritcha]

yes please open an issue and I'll assign it to the author of 12290.

[angainor]

I have reported this to ompi issues: open-mpi/ompi#13048

This issue can be closed.