Mixed Mode Compilation issue on HPC

[leonc]

Hello there,

I am a new openlb (version 1.8.1) user and I am trying to compile it on the Gadi HPC, Australia (https://nci.org.au/our-systems/hpc-systems) utilising the V100 GPUs. I load the following modules

module load gcc/12.2.0
module load openmpi/4.1.7
module load cuda/12.3.2

and in the config.mk file

CXX := mpic++
CC := gcc

CXXFLAGS := -O3 -Wall -march=x86-64-v3
CXXFLAGS += -std=c++20

PARALLEL_MODE := MPI

PLATFORMS := CPU_SISD GPU_CUDA

CUDA_CXX := nvcc
CUDA_CXXFLAGS := -O3 -std=c++20

CUDA_ARCH := 70

FLOATING_POINT_TYPE := float

USE_EMBEDDED_DEPENDENCIES := ON

OLB_MIXED_MODE_INCLUDE_CPP := ON

The compilation works (and runs) on cavity example but I have compilations issues when trying to run the turbulent channel3d case. Initially I had issues with the default.mixed.mk file with an unescaped “#'” in so edit that with “#” in default.mixed.mk in the indented line:

OLB_MIXED_MODE_INCLUDE := \#include <olb.h>\n
ifdef OLB_MIXED_MODE_INCLUDE_CPP
OLB_MIXED_MODE_INCLUDE := $(CPP_FILES:%=\n\#include “../%”\n)
endif

However, I still faced other issues with the default.mixed.mk file. I get an error that it is missing some symbol:

channel3d.cpp:(.text._ZNSt17_Function_handlerIFPN3olb18AbstractCollisionOIdNS0_11descriptors5D3Q19IJNS2_5FORCEENS2_6TENSORENS2_8VELOCITYENS2_8POROSITYENS2_9VELOCITY2ENS2_16AVERAGE_VELOCITYEEEEEENS0_8PlatformEEZNS0_15DynamicsPromiseIdSA_EC4INS0_8dynamics5TupleIdSA_NS0_7momenta5TupleINSK_11BulkDensityENSK_31MovingPorousMomentumCombinationINSK_12BulkMomentumEEENSK_10BulkStressENSK_11DefineToNEqEEENS0_10equilibria10ThirdOrderENS0_9collision17ParameterFromCellINSV_6HYBRIDENSW_INSV_10HYBRID_RHOENSV_39LocalVanDriestSmagorinskyEffectiveOmegaINSV_14ExternalRhoHRREEEEEEENSI_18DefaultCombinationEEEEENS0_4meta2idIT_EEEUlSD_E0_E9_M_invokeERKSt9_Any_dataOSD_[_ZNSt17_Function_handlerIFPN3olb18AbstractCollisionOIdNS0_11descriptors5D3Q19IJNS2_5FORCEENS2_6TENSORENS2_8VELOCITYENS2_8POROSITYENS2_9VELOCITY2ENS2_16AVERAGE_VELOCITYEEEEEENS0_8PlatformEEZNS0_15DynamicsPromiseIdSA_EC4INS0_8dynamics5TupleIdSA_NS0_7momenta5TupleINSK_11BulkDensityENSK_31MovingPorousMomentumCombinationINSK_12BulkMomentumEEENSK_10BulkStressENSK_11DefineToNEqEEENS0_10equilibria10ThirdOrderENS0_9collision17ParameterFromCellINSV_6HYBRIDENSW_INSV_10HYBRID_RHOENSV_39LocalVanDriestSmagorinskyEffectiveOmegaINSV_14ExternalRhoHRREEEEEEENSI_18DefaultCombinationEEEEENS0_4meta2idIT_EEEUlSD_E0_E9_M_invokeERKSt9_Any_dataOSD_]+0x85): undefined reference to `_ZN3olb23ConcreteBlockCollisionOIdNS_11descriptors5D3Q19IJNS1_5FORCEENS1_6TENSORENS1_8VELOCITYENS1_8POROSITYENS1_9VELOCITY2ENS1_16AVERAGE_VELOCITYEEEELNS_8PlatformE2ENS_8dynamics5TupleIdS9_NS_7momenta5TupleINSD_11BulkDensityENSD_31MovingPorousMomentumCombinationINSD_12BulkMomentumEEENSD_10BulkStressENSD_11DefineToNEqEEENS_10equilibria10ThirdOrderENS_9collision17ParameterFromCellINSO_6HYBRIDENSP_INSO_10HYBRID_RHOENSO_39LocalVanDriestSmagorinskyEffectiveOmegaINSO_14ExternalRhoHRREEEEEEENSB_18DefaultCombinationEEEEC1Ev’

The HPC staff is not too sure what the issue is and suggested I get help. Thank you in advance!

Leon

[Adrian]

This is unusual, can you share which version of GNU Make you are using? The change to the escape in the mixed config should not be necessary (I also just re-tested the build of channel3d in mixed mode to be sure).

In any case: Did you do a full rebuild of the example after you tweaked the config? (make clean; make) Also, just to be safe: Did you change anything else beside the config? Is there a specific reason why you added the OLB_MIXED_MODE_INCLUDE_CPP := ON to the config.mk? This flag is already set by default in the example’s Makefile.

[leonc]

Hello Adrian,

Thank you for your response. I am using GNU Make 4.2.1.

Yes, I did make clean; make when recompiling.

No, I did not change anything else other than the config.mk and the default.mixed.mk files.

OLB_MIXED_MODE_INCLUDE_CPP := ON. The HPC staff advised us to include it when they were running the cavity testcase. Yes, I agree that it would be redundant as it is already in the Makefile. I did switch it to OFF but still had a compilation error.

Thank you!

Leon

[Adrian]

Sorry for the delay, I can reproduce the issue in GNU Make 4.2.1, it works for 4.4. I’ll look into it and provide a fix for you.

[Adrian]

So I think I was able to reproduce your exact issue by:

1. fixing the escape issue for GNU Make 4.2.1
2. compiling without GPU support (e.g. by using the default config.mk)
3. Switching on mixed compilation mode GPU support
4. Recompiling just the case, not the core library
5. => linker errors for the same symbols that you got

So your setup should be fixable by executing make clean-core additionally and then rebuilding (forgot to mention this before)

I am still looking into a general fix for the escape issue as escaping the include fixes 4.2.1 but breaks 4.4.1. Will keep you updated.

[Author]

Posts