Multi GPUs Calculation
- This topic has 26 replies, 2 voices, and was last updated 9 months, 2 weeks ago by .
-
Posts
-
Hello team openLB,
I am tyring GPU calculatons.
1. with one GPU, it worked well when I used gpu_only.mk at `examples/laminar/cavity3dBenchmark’2. with 2 GPUs (because my workstation has 2 of 3080Ti), it doesn’t work well when I use gpu_openmpi_mixed.mk or gpu_openmpi.mk. The both of error message are same
The call to cuIpcOpenMemHandle failed. This is an unrecoverable error
and will cause the program to abort.
Hostname: user
cuIpcOpenMemHandle return value: 217
address: 0x1310200000
Check the cuda.h file for what the return value means. A possible cause
for this is not enough free device memory. Try to reduce the device
memory footprint of your application.
————————————————————————–
[user:06042] Failed to register remote memory, rc=-1
[user:06041] Failed to register remote memory, rc=-1
corrupted size vs. prev_size while consolidating
[user:06041] *** Process received signal ***
[user:06041] Signal: Aborted (6)
[user:06041] Signal code: (-6)
[user:06037] 1 more process has sent help message help-mpi-common-cuda.txt / cuIpcOpenMemHandle failed
[user:06037] Set MCA parameter “orte_base_help_aggregate” to 0 to see all help / error messages(the shell comand is $mpirun -np 2 bash -c ‘export CUDA_VISIBLE_DEVICES=${OMPI_COMM_WORLD_LOCAL_RANK}; ./cavity3d’)
Moreover, I tried a comannd of $ mpirun -np 2 ./cavity3d. The error message was [GPU_CUDA:0] Found 2 CUDA devices but only one can be used per MPI process.
[GPU_CUDA:1] Found 2 CUDA devices but only one can be used per MPI process.
corrupted size vs. prev_size while consolidating
[user:06021] *** Process received signal ***
[user:06021] Signal: Aborted (6)
[user:06021] Signal code: (-6)Do you know what is wrong?
I am wondering if I need NVLINK as to use multi GPUs?Best,
In general you do not need NVlink interconnect to use multiple GPUs in OpenLB (as MPI will transparently fall back to PCI device-cpu-device communication, although it is recommended for optimal performance due to better inter-GPU bandwidth).
I assume that OpenLB did not issue a warning on missing CUDA-awareness of MPI (e.g. “The used MPI Library is not CUDA-aware. Multi-GPU execution will fail.”) and that you compiled / installed MPI with CUDA-awareness?
Can you provide me with more details on your system and software environment? (CUDA versions, modified config.mk and so on)
If you use
mpirun -np 2 ./cavity3donly the first of all visible GPUs will be used (as per the warning message). This is why the example configs contain thempirun -np 2 bash -c ‘export CUDA_VISIBLE_DEVICES=${OMPI_COMM_WORLD_LOCAL_RANK}; ./program’command which will assign each rank its own GPU.Ok, this looks as if the wrong MPI version is selected for compilation. Likely you will get a compiler / linker error if you remove opempi-bin and libopenmpi-dev.
Does
ompi_info --parsable -l 9 --all | grep mpi_built_with_cuda_support:valuereturn true (as per the user guide)? Did you modify the PATH accordingly?I would suggest to make sure that the correct MPI version is set and use the flags as provided by
mpic++ -showme. (Or you can exactly follow the user guide / tech report).Ok, this seems to be a precedence issue then – if you add
-I/home/ubuntu2204/opt/openmpi/includeto your CXXFLAGS and-L/home/ubuntu2204/opt/openmpi/libto the LDFLAGS it should work.How do you know that it is not actually using the MPI library in
/home/ubuntu2204/opt/openmpi/lib? The core library compilation was successful as per the log. Did you already try recompiling one of the examples to confirm whether it still doesn’t work?To make sure you can also try to put the CUDA-aware MPI paths first in:
export PATH=$PATH:$MPI_HOME/bin export CPATH=$CPATH:$MPI_HOME/include export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$MPI_HOME/libDid you verify whether single-GPU execution works correctly on the other GPU by setting CUDA_VISIBLE_DEVICES to 1?
I will only be back from holiday tomorrow so can not verify right now but ” -mca btl_smcuda_use_cuda_ipc 0″ should disable NVlink for local GPU-GPU communication in CUDA-aware MPI.
No worries 🙂 So the multi-GPU execution now works with the additional flag?
The
gpu::cuda::device::synchronizecalls are conditionally enabled only if GPU support is enabled. In general this is an artifact of the work-in-progress nature of heterogeneous computation support in OpenLB. Both this and theSuperLattice::setProcessingContextcalls will be transparently hidden in the future.What do you mean exactly by your last question? Ignoring the mpirun / hardware setup issues, multi-GPU support in OpenLB is transparent in the sense that if A) CUDA-aware MPI support is enabled during comilation and B) the application works on a single GPU then it will work in multi-GPU mode also.
OpenMP is only used for CPU-side parallelization on shared memory systems. Most commonly we use it in HYBRID mode for CPU-only simulations (i.e. each socket of a cluster is assigned a single OpenMPI process using OpenMP parallelization internally).
The performance-critical parts of OpenMPI usage are contained in the
SuperCommunicator(and its support infrastructure). This is the class responsible for handling all overlap communication between the individual blocks of the decomposition.
- You must be logged in to reply to this topic.