Calling multiple GPU devices from a single thread #107
jatkinson1000
started this conversation in
General
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
In the example @jwallwork23 wrote in #96 we have
n
MPI processes forn
GPUs.In a discussion I just had the question was raised - what if we gather data into one process to send to multiple GPUs, or if we have multiple nets and want to cast each one to a different GPU.
My gut feeling is that since Fortran is sequential/imperative, calling multiple nets on different devices from a single process will be no faster than calling them sequentially on the same device (possibly even worse depending on proximity on the node of the CPU to the GPU).
More interesting might be how to assign device indices if we want to gather
m >> n
processes inton
for distribution ton
GPUs, but this isn't particularly an "us" problem, nor is it unique to FTorch but occurs more generally in MPI programs.Beta Was this translation helpful? Give feedback.
All reactions