Issues with time averaged velocity field in turbulence model running in GPU

[jflorezgi]

Hi everyone, I’m trying to run an OpenLB code to simulate a turbulent flow (using different turbulent dynamics), and I want to obtain the time averaged velocity field at the end of the simulation. For this case I’m using the squareCavity2d benchmark as a guide but only for the velocity field (it can be obtained the time averaged temperature field in this example also). The lines that I’m using are:

This part is after the prepareLattice function call to define de instantaneous and time averaged velocity functors

#ifdef PRINT_VTKSTATISTICS
SuperVTMwriter3D<T> vtmWriter(“enclosedspace3d”);
SuperLatticePhysVelocity3D<T,DESCRIPTOR> sVel(sLattice,converter);
SuperLatticeTimeAveragedF3D<T> sAveragedVel(sVel);
SuperLatticeTimeAveragedCrossCorrelationF3D<T> sAveragedVelVelCross(sVel,sVel);
#endif

When the simulation converges the time averaged velocity field is computed and printed

#ifdef PRINT_VTKSTATISTICS
if ((converge.hasConverged() || iT == converter.getLatticeTime( maxPhysT )-1) && iT % statisticsIntervall == 0) {
sLattice.communicate();
sAveragedVel.addEnsemble();
sAveragedVelVelCross.addEnsemble();
if ( sAveragedVel.getEnsembles() >= statisticsEnsembles ) {
break;
}
}
#endif
}
#ifdef PRINT_VTKSTATISTICS
vtmWriter.write(sAveragedVel);
vtmWriter.write(sVel);
#endif

running with cpu_gcc_openmpi.mk as config.mk the output console is (there is no compilation problem):

make -C ../../../../external
make[1]: se entra en el directorio ‘/home/jonathan/Documentos/Proyecto/OpenLB/olb-1.5r0/external’
make -C zlib
make[2]: se entra en el directorio ‘/home/jonathan/Documentos/Proyecto/OpenLB/olb-1.5r0/external/zlib’
make[2]: No se hace nada para ‘all’.
make[2]: se sale del directorio ‘/home/jonathan/Documentos/Proyecto/OpenLB/olb-1.5r0/external/zlib’
cp zlib/build/libz.a lib/
make -C tinyxml
make[2]: se entra en el directorio ‘/home/jonathan/Documentos/Proyecto/OpenLB/olb-1.5r0/external/tinyxml’
make[2]: No se hace nada para ‘all’.
make[2]: se sale del directorio ‘/home/jonathan/Documentos/Proyecto/OpenLB/olb-1.5r0/external/tinyxml’
cp tinyxml/build/libtinyxml.a lib/
make[1]: se sale del directorio ‘/home/jonathan/Documentos/Proyecto/OpenLB/olb-1.5r0/external’
mpic++ -DPARALLEL_MODE_MPI -O3 -Wall -march=native -mtune=native -std=c++17 -DPLATFORM_CPU_SISD -I../../../../src -I../../../../external/zlib -I../../../../external/tinyxml -c -o GPU-ESVcase01.o GPU-ESVcase01.cpp
mpic++ GPU-ESVcase01.o -o GPU-ESVcase01 -L../../../../external/lib -lz -ltinyxml

but running with gpu_only.mk as config.mk, there is an error:

make -C ../../../../external
make[1]: Entering directory ‘/home/jflorez/proyecto/olb-1.5r0/external’
make -C zlib
make[2]: Entering directory ‘/home/jflorez/proyecto/olb-1.5r0/external/zlib’
make[2]: Nothing to be done for ‘all’.
make[2]: Leaving directory ‘/home/jflorez/proyecto/olb-1.5r0/external/zlib’
cp zlib/build/libz.a lib/
make -C tinyxml
make[2]: Entering directory ‘/home/jflorez/proyecto/olb-1.5r0/external/tinyxml’
make[2]: Nothing to be done for ‘all’.
make[2]: Leaving directory ‘/home/jflorez/proyecto/olb-1.5r0/external/tinyxml’
cp tinyxml/build/libtinyxml.a lib/
make[1]: Leaving directory ‘/home/jflorez/proyecto/olb-1.5r0/external’
nvcc -O3 -std=c++17 –generate-code=arch=compute_75,code=[compute_75,sm_75] –extended-lambda –expt-relaxed-constexpr -x cu -Xcudafe “–diag_suppress=implicit_return_from_non_void_function –display_error_number –diag_suppress=20014 –diag_suppress=20011” -DPLATFORM_CPU_SISD -DPLATFORM_GPU_CUDA -I../../../../src -I../../../../external/zlib -I../../../../external/tinyxml -c -o GPU-ESVcase01.o GPU-ESVcase01.cpp
../../../../src/functors/lattice/timeAveraged/superLatticeTimeAveraged3D.hh(85): error: Internal Compiler Error (codegen): “variable length arrays are not supported!”

make: *** [../../../../default.mk:31: GPU-ESVcase01.o] Error 2

it seems that there is a problem with superLatticeTimeAveraged3D functor when the code is running in gpu.

Thanks in advance for any help

[Adrian]

Thanks for the report. The issue is that some places in OpenLB use non-standard runtime-defined C-style array sizes which do not work in all compilers. This will be fixed in the next release, until then you can replace lines 85 and 86 in src/functors/lattice/timeAveraged/superLatticeTimeAveraged3D.hh by e.g.:

      std::vector<BaseType<T>> tmp(_sFunctor.getTargetDim(), 0);
      _sFunctor(tmp.data(), i);

[jflorezgi]

Thank you Adrian

[Author]

Posts