Multi-GPU MPI library is not CUDA-aware
- This topic has 3 replies, 2 voices, and was last updated 1 month ago by .
-
Posts
-
Hello,
We are trying to run some multi-gpu examples (aorta3d) and get the following error upon execution:
[GPU_CUDA] The used MPI Library is not CUDA-aware. Multi-GPU execution will fail.Some info that may be useful:
System is a 48-core EPYC with 2x NVIDIA RTX PRO 6000 Blackwell GPUs, 384GB memory.
Ubuntu 24.04
CUDA 13.0 with drivers 580.65.06
OpenMPI 5.0.8running the command nvidia-smi I get:
ubuntu@hpc:~/OpenLB_GPU/examples/turbulence/aorta3d$ nvidia-smi Mon Oct 6 14:14:23 2025 +-----------------------------------------------------------------------------------------+ | NVIDIA-SMI 580.65.06 Driver Version: 580.65.06 CUDA Version: 13.0 | +-----------------------------------------+------------------------+----------------------+ | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |=========================================+========================+======================| | 0 NVIDIA RTX PRO 6000 Blac... Off | 00000000:01:00.0 Off | Off | | 30% 25C P8 12W / 300W | 15MiB / 97887MiB | 0% Default | | | | N/A | +-----------------------------------------+------------------------+----------------------+ | 1 NVIDIA RTX PRO 6000 Blac... Off | 00000000:02:00.0 Off | Off | | 30% 24C P8 9W / 300W | 15MiB / 97887MiB | 0% Default | | | | N/A | +-----------------------------------------+------------------------+----------------------+ +-----------------------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=========================================================================================| | 0 N/A N/A 2309 G /usr/lib/xorg/Xorg 4MiB | | 1 N/A N/A 2309 G /usr/lib/xorg/Xorg 4MiB | +-----------------------------------------------------------------------------------------+NVCC:
ubuntu@hpc:~/OpenLB_GPU/examples/turbulence/aorta3d$ nvcc --version nvcc: NVIDIA (R) Cuda compiler driver Copyright (c) 2005-2025 NVIDIA Corporation Built on Wed_Aug_20_01:58:59_PM_PDT_2025 Cuda compilation tools, release 13.0, V13.0.88 Build cuda_13.0.r13.0/compiler.36424714_0ompi_info:
ubuntu@hpc:~/OpenLB_GPU/examples/turbulence/aorta3d$ ompi_info | grep -i cuda Configure command line: '--prefix=/opt/openmpi-5.0.8' '--with-cuda=/usr/local/cuda' MPI extensions: affinity, cuda, ftmpi, rocm, shortfloat MCA accelerator: cuda (MCA v2.1.0, API v1.0.0, Component v5.0.8) MCA btl: smcuda (MCA v2.1.0, API v3.3.0, Component v5.0.8) MCA coll: cuda (MCA v2.1.0, API v2.4.0, Component v5.0.8)ubuntu@hpc:~/OpenLB_GPU/examples/turbulence/aorta3d$ ompi_info --parsable --all | grep mpi_built_with_cuda_support:value mca:mpi:base:param:mpi_built_with_cuda_support:value:trueOpenLB config.mk:
CXX := nvcc CC := nvcc CXXFLAGS := -O3 CXXFLAGS += -std=c++20 --forward-unknown-to-host-compiler PARALLEL_MODE := MPI # CPU/MPI compiler flags CXXFLAGS += -I/opt/openmpi-5.0.8/include CCFLAGS += -I/opt/openmpi-5.0.8/include # MPI linker flags LDFLAGS += -L/opt/openmpi-5.0.8/lib -lmpi PLATFORMS := CPU_SISD GPU_CUDA USE_CUDA_AWARE_MPI := ON # for e.g. RTX 30* (Ampere), see table in 'rules.mk' for other options CUDA_ARCH := 100 FLOATING_POINT_TYPE := float USE_EMBEDDED_DEPENDENCIES := ONSingle-GPU simulations run fine and as far as I can tell OpenMPI is installed with CUDA support enabled, however when compiling the OpenLB examples it doesn’t detect the CUDA-aware installation for multi-GPU runs. Any advice is appreciated, hopefully I have provided enough information.
Thanks in advance,
AlexThe command outputs (thanks!) all look fine so this may just be the automated check failing despite CUDA-awareness being available. The logic we use to check this
#ifdef PARALLEL_MODE_MPI #if defined(MPIX_CUDA_AWARE_SUPPORT) && MPIX_CUDA_AWARE_SUPPORT if (!MPIX_Query_cuda_support()) { clout << "The used MPI Library is not CUDA-aware. Multi-GPU execution will fail." << std::endl; } #endif #if defined(MPIX_CUDA_AWARE_SUPPORT) && !MPIX_CUDA_AWARE_SUPPORT clout << "The used MPI Library is not CUDA-aware. Multi-GPU execution will fail." << std::endl; #endif #if !defined(MPIX_CUDA_AWARE_SUPPORT) clout << "Unable to check for CUDA-aware MPI support. Multi-GPU execution may fail." << std::endl; #endif #endif // PARALLEL_MODE_MPIcan definitely have gaps. Does the program proceed as usual in multi-GPU after this? (If CUDA-awareness is indeed not working for some reason despite the command output I would expect it to instantly segfault on communication).
Ok, weird. I just re-tested the release on my dual GPU system and the example works as it should.
One other possibility is that the
nvccselected in the environment is a different one than in your/usr/local/cuda. You could also try the “mixed” mode (see the example configs inconfig/) to directly use yourmpic++together withnvcc.Did you test any other CUDA-aware MPI apps in the same environment?
- You must be logged in to reply to this topic.