Reply To: Problem with multi GPU

[Adrian]

The problem is that each process by default uses the first visible GPU. You can restrict this to assign each process their own GPU via:

mpirun -np 2 bash -c 'export CUDA_VISIBLE_DEVICES=${OMPI_COMM_WORLD_LOCAL_RANK}; ./cylinder3d'

This is also documented in the config/gpu_openmpi.mk: (excerpt)

#  - Start the simulation using <code>mpirun -np 2 ./cavity3d</code> (All processes share default GPU, not optimal)
#
# Usage on a multi GPU system: (recommended when using MPI, use non-MPI version on single GPU systems)
#  - Run  "mpirun -np 4 bash -c 'export CUDA_VISIBLE_DEVICES=${OMPI_COMM_WORLD_LOCAL_RANK}; ./cavity3d'"
#    (for a 4 GPU system, further process mapping advisable, consult cluster documentation)