Using MRT with GPU

[sfraniatte]

Hello,

I can’t get the MRT method to work with the Smagorinsky model on GPU (e.g., with the aorta3d example), but it runs fine on CPU in parallel mode with the latest version of OpenLB. Everything worked correctly with version 1.7. I’m using CUDA 12.9, and everything works properly with the BGK method.

Best regards,
Sylvain

[sfraniatte]

I forgot to mention that it’s only the outlet pressure condition that doesn’t work (the flow remains zero) with the MRT method.

[Adrian]

Can you share your modifications of the case so we can reproduce your issue?

[sfraniatte]

Thank you for your respond ! Here is the code that does not work but succeed to compile on GPU and works on CPU. It is basically the aorta example with the MRT. To compile OpenLB, I used the gpu_only.mk file and with the ampere generation (nvidia rtx a6000) and with CUDA 12.9. This setup works with my case (without MRT) and the other examples.

/* Lattice Boltzmann sample, written in C++, using the OpenLB
* library
*
* Copyright (C) 2011-2014 Mathias J. Krause
* E-mail contact: info@openlb.net
* The most recent release of OpenLB can be downloaded at
* <http://www.openlb.net/&gt;
*
* This program is free software; you can redistribute it and/or
* modify it under the terms of the GNU General Public License
* as published by the Free Software Foundation; either version 2
* of the License, or (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public
* License along with this program; if not, write to the Free
* Software Foundation, Inc., 51 Franklin Street, Fifth Floor,
* Boston, MA 02110-1301, USA.
*/

/* aorta3d.cpp:
* In this example the fluid flow through a bifurcation is
* simulated. The geometry is obtained from a mesh in stl-format.
* With Bouzidi boundary conditions the curved boundary is
* adequately mapped and initialized fully automatically. As
* dynamics a Smagorinsky turbulent BGK model is used to stabilize
* the simulation for low resolutions. As output the flux at the
* inflow and outflow region is computed. The wall stress can be
* visualized on the stl Mesh with the Mesh.pvd file in paraview.
* The results has been validated by comparison with other results
* obtained with FEM and FVM.
*/

#include <olb.h>

using namespace olb;
using namespace olb::descriptors;
using namespace olb::graphics;

using T = FLOATING_POINT_TYPE;
using DESCRIPTOR = D3Q19<tag::MRT>;
using BulkDynamics = SmagorinskyMRTdynamics<T,DESCRIPTOR>;

//simulation parameters
const int N = 40; // resolution of the model
const int M = 20; // time discretization refinement
const bool bouzidiOn = true; // choice of boundary condition
const T maxPhysT = 2.; // max. simulation time in s, SI unit

// Stores data from stl file in geometry in form of material numbers
void prepareGeometry( UnitConverter<T,DESCRIPTOR> const& converter, IndicatorF3D<T>& indicator,
STLreader<T>& stlReader, SuperGeometry<T,3>& superGeometry )
{

OstreamManager clout( std::cout,”prepareGeometry” );
clout << “Prepare Geometry …” << std::endl;

superGeometry.rename( 0,2,indicator );
superGeometry.rename( 2,1,stlReader );

superGeometry.clean();

// Set material number for inflow
IndicatorCircle3D<T> inflow( 0.218125,0.249987,0.0234818, 0., 1.,0., 0.0112342 );
IndicatorCylinder3D<T> layerInflow( inflow, 2.*converter.getPhysDeltaX() );
superGeometry.rename( 2,3,1,layerInflow );

// Set material number for outflow0
//IndicatorCircle3D<T> outflow0(0.2053696,0.0900099,0.0346537, 2.5522,5.0294,-1.5237, 0.0054686 );
IndicatorCircle3D<T> outflow0( 0.2053696,0.0900099,0.0346537, 0.,-1.,0., 0.0054686 );
IndicatorCylinder3D<T> layerOutflow0( outflow0, 2.*converter.getPhysDeltaX() );
superGeometry.rename( 2,4,1,layerOutflow0 );

// Set material number for outflow1
//IndicatorCircle3D<T> outflow1(0.2388403,0.0900099,0.0343228, -1.5129,5.1039,-2.8431, 0.0058006 );
IndicatorCircle3D<T> outflow1( 0.2388403,0.0900099,0.0343228, 0.,-1.,0., 0.0058006 );
IndicatorCylinder3D<T> layerOutflow1( outflow1, 2.*converter.getPhysDeltaX() );
superGeometry.rename( 2,5,1,layerOutflow1 );

// Removes all not needed boundary voxels outside the surface
superGeometry.clean();
// Removes all not needed boundary voxels inside the surface
superGeometry.innerClean( 3 );
superGeometry.checkForErrors();

superGeometry.print();
clout << “Prepare Geometry … OK” << std::endl;
}

// Set up the geometry of the simulation
void prepareLattice( SuperLattice<T, DESCRIPTOR>& lattice,
UnitConverter<T,DESCRIPTOR> const& converter,
STLreader<T>& stlReader, SuperGeometry<T,3>& superGeometry )
{

OstreamManager clout( std::cout,”prepareLattice” );
clout << “Prepare Lattice …” << std::endl;

const T omega = converter.getLatticeRelaxationFrequency();

// material=1 –> bulk dynamics
lattice.defineDynamics<BulkDynamics>(superGeometry, 1);

if ( bouzidiOn ) {
// material=2 –> no dynamics + bouzidi zero velocity
setBouzidiBoundary<T,DESCRIPTOR>(lattice, superGeometry, 2, stlReader);
// material=3 –> no dynamics + bouzidi velocity (inflow)
setBouzidiBoundary<T,DESCRIPTOR,BouzidiVelocityPostProcessor>(lattice, superGeometry, 3, stlReader);
}
else {
// material=2 –> bounceBack dynamics
boundary::set<boundary::BounceBack>(lattice, superGeometry, 2);
// material=3 –> bulk dynamics + velocity (inflow)
lattice.defineDynamics<BulkDynamics>(superGeometry, 3);
boundary::set<boundary::InterpolatedVelocity>(lattice, superGeometry, 3);
}

// material=4,5 –> bulk dynamics + pressure (outflow)
lattice.defineDynamics<BulkDynamics>(superGeometry.getMaterialIndicator({4, 5}));
boundary::set<boundary::InterpolatedPressure>(lattice, superGeometry, 4);
boundary::set<boundary::InterpolatedPressure>(lattice, superGeometry, 5);

// Initial conditions
AnalyticalConst3D<T,T> rhoF( 1 );
std::vector<T> velocity( 3,T() );
AnalyticalConst3D<T,T> uF( velocity );

// Initialize all values of distribution functions to their local equilibrium
lattice.defineRhoU( superGeometry.getMaterialIndicator({1, 3, 4, 5}),rhoF,uF );
lattice.iniEquilibrium( superGeometry.getMaterialIndicator({1, 3, 4, 5}),rhoF,uF );

lattice.setParameter<descriptors::OMEGA>(omega);
lattice.setParameter<collision::LES::SMAGORINSKY>(T(0.1));
// Lattice initialize
lattice.initialize();

clout << “Prepare Lattice … OK” << std::endl;
}

// Generates a slowly increasing sinuidal inflow
void setBoundaryValues( SuperLattice<T, DESCRIPTOR>& sLattice,
UnitConverter<T,DESCRIPTOR> const& converter, std::size_t iT,
SuperGeometry<T,3>& superGeometry )
{

// No of time steps for smooth start-up
std::size_t iTperiod = converter.getLatticeTime( 0.5 );
std::size_t iTupdate = 50;

if ( iT%iTupdate == 0 ) {
// Smooth start curve, sinus
SinusStartScale<T,std::size_t> nSinusStartScale( iTperiod,converter.getCharLatticeVelocity() );

// Creates and sets the Poiseuille inflow profile using functors
std::size_t iTvec[1]= {iT};
T maxVelocity[1]= {T()};
nSinusStartScale( maxVelocity,iTvec );
CirclePoiseuille3D<T> velocity( superGeometry,3,maxVelocity[0], T() );

if ( bouzidiOn ) {
setBouzidiVelocity(sLattice, superGeometry, 3, velocity);
sLattice.setProcessingContext<Array<descriptors::BOUZIDI_VELOCITY>>(
ProcessingContext::Simulation);
}
else {
sLattice.defineU(superGeometry, 3, velocity);
sLattice.setProcessingContext<Array<momenta::FixedVelocityMomentumGeneric::VELOCITY>>(
ProcessingContext::Simulation);
}
}
}

// Computes flux at inflow and outflow
void getResults(SuperLattice<T, DESCRIPTOR>& sLattice,
const UnitConverter<T,DESCRIPTOR>& converter,
std::size_t iT,
SuperGeometry<T,3>& superGeometry,
util::Timer<T>& timer,
STLreader<T>& stlReader,
VTUsurfaceWriter<T>& vtuWriter)
{
OstreamManager clout( std::cout,”getResults” );

const std::size_t vtkIter = converter.getLatticeTime( .1 );
const std::size_t statIter = converter.getLatticeTime( .1 );

if ( iT==0 ) {
SuperVTMwriter3D<T> vtmWriter(“aorta3d”);
// Writes the geometry, cuboid no. and rank no. as vti file for visualization

SuperLatticeCuboid3D<T, DESCRIPTOR> cuboid( sLattice );
SuperLatticeRank3D<T, DESCRIPTOR> rank( sLattice );
vtmWriter.write( cuboid );
vtmWriter.write( rank );

vtmWriter.createMasterFile();
vtuWriter.createMasterFile();
}

// Writes the vtk files
if ( iT%vtkIter==0 ) {
sLattice.setProcessingContext(ProcessingContext::Evaluation);
sLattice.scheduleBackgroundOutputVTK([&,iT](auto task) {
SuperVTMwriter3D<T> vtmWriter(“aorta3d”);
SuperLatticePhysVelocity3D velocity(sLattice, converter);
SuperLatticePhysPressure3D pressure(sLattice, converter);
vtmWriter.addFunctor(velocity);
vtmWriter.addFunctor(pressure);
task(vtmWriter, iT);
});

// Write interpolated wall shear stress on the STL surface
{
SuperLatticeDensity3D densityF(sLattice);
AnalyticalFfromSuperF3D smoothDensityF(densityF);

SuperLatticeStress3D stressF(sLattice);
AnalyticalFfromSuperF3D smoothStressF(stressF);

PhysWallShearStressOnSurface3D<T,DESCRIPTOR> interpolatedWssF(converter, smoothDensityF, smoothStressF, stlReader);
interpolatedWssF.getName() = “interpolatedWss”;

vtuWriter.addFunctor(interpolatedWssF);

vtuWriter.write(iT);
}
}

// Writes output on the console
if ( iT%statIter==0 ) {
// Timer console output
timer.update( iT );
timer.printStep();

// Lattice statistics console output
sLattice.getStatistics().print( iT,converter.getPhysTime( iT ) );

// Flux at the inflow and outflow region
std::vector<int> materials = { 1, 3, 4, 5 };

IndicatorCircle3D<T> inflow( 0.218125,0.249987-2.*converter.getPhysDeltaX(),0.0234818, 0., -1.,0., 0.0112342+2*converter.getPhysDeltaX() );
SuperPlaneIntegralFluxVelocity3D<T> vFluxInflow( sLattice, converter, superGeometry, inflow, materials, BlockDataReductionMode::Discrete );
vFluxInflow.print( “inflow”,”ml/s” );
SuperPlaneIntegralFluxPressure3D<T> pFluxInflow( sLattice, converter, superGeometry, inflow, materials, BlockDataReductionMode::Discrete );
pFluxInflow.print( “inflow”,”N”,”mmHg” );

IndicatorCircle3D<T> outflow0( 0.2053696,0.0900099+2.*converter.getPhysDeltaX(),0.0346537, 0.,1.,0., 0.0054686+2*converter.getPhysDeltaX() );
SuperPlaneIntegralFluxVelocity3D<T> vFluxOutflow0( sLattice, converter, superGeometry, outflow0, materials, BlockDataReductionMode::Discrete );
vFluxOutflow0.print( “outflow0″,”ml/s” );
SuperPlaneIntegralFluxPressure3D<T> pFluxOutflow0( sLattice, converter, superGeometry, outflow0, materials, BlockDataReductionMode::Discrete );
pFluxOutflow0.print( “outflow0″,”N”,”mmHg” );

IndicatorCircle3D<T> outflow1( 0.2388403,0.0900099+2.*converter.getPhysDeltaX(),0.0343228, 0.,1.,0., 0.0058006+2*converter.getPhysDeltaX() );
SuperPlaneIntegralFluxVelocity3D<T> vFluxOutflow1( sLattice, converter, superGeometry, outflow1, materials, BlockDataReductionMode::Discrete );
vFluxOutflow1.print( “outflow1″,”ml/s” );
SuperPlaneIntegralFluxPressure3D<T> pFluxOutflow1( sLattice, converter, superGeometry, outflow1, materials, BlockDataReductionMode::Discrete );
pFluxOutflow1.print( “outflow1″,”N”,”mmHg” );

if ( bouzidiOn ) {
SuperLatticeYplus3D<T, DESCRIPTOR> yPlus( sLattice, converter, superGeometry, stlReader, 3 );
SuperMax3D<T> yPlusMaxF( yPlus, superGeometry, 1 );
int input[4]= {};
T yPlusMax[1];
yPlusMaxF( yPlusMax,input );
clout << “yPlusMax=” << yPlusMax[0] << std::endl;
}
}

if ( sLattice.getStatistics().getMaxU() > 0.3 ) {
clout << “PROBLEM uMax=” << sLattice.getStatistics().getMaxU() << std::endl;
std::exit(0);
}
}

int main( int argc, char* argv[] )
{
// === 1st Step: Initialization ===
initialize( &argc, &argv );
singleton::directories().setOutputDir( “./tmp/” );
OstreamManager clout( std::cout,”main” );
// display messages from every single mpi process
//clout.setMultiOutput(true);

const UnitConverter<T,DESCRIPTOR> converter(
(T) 0.02246/N, // physDeltaX: spacing between two lattice cells in __m__
(T) 0.02246/(M*N), // physDeltaT: time step in __s__
(T) 0.02246, // charPhysLength: reference length of simulation geometry
(T) 0.45, // charPhysVelocity: maximal/highest expected velocity during simulation in __m / s__
(T) 0.003/1055., // physViscosity: physical kinematic viscosity in __m^2 / s__
(T) 1055 // physDensity: physical density in __kg / m^3__
);
// Prints the converter log as console output
converter.print();
// Writes the converter log in a file
converter.write(“aorta3d”);

// === 2nd Step: Prepare Geometry ===

// Instantiation of the STLreader class
// file name, voxel size in meter, stl unit in meter, outer voxel no., inner voxel no.
STLreader<T> stlReader( “aorta3d.stl”, converter.getPhysDeltaX(), 0.001, olb::RayMode::FastRayZ, true );
IndicatorLayer3D<T> extendedDomain( stlReader, converter.getPhysDeltaX() );

// Instantiation of a cuboidDecomposition with weights
#ifdef PARALLEL_MODE_MPI
const int noOfCuboids = util::min(16*N, 8*singleton::mpi().getSize());
#else
const int noOfCuboids = 2;
#endif
CuboidDecomposition3D<T> cuboidDecomposition( extendedDomain, converter.getPhysDeltaX(), noOfCuboids, “volume” );
// Instantiation of a loadBalancer
HeuristicLoadBalancer<T> loadBalancer( cuboidDecomposition );

// Instantiation of a superGeometry
SuperGeometry<T,3> superGeometry( cuboidDecomposition, loadBalancer );

prepareGeometry( converter, extendedDomain, stlReader, superGeometry );

// === 3rd Step: Prepare Lattice ===
SuperLattice<T, DESCRIPTOR> sLattice( superGeometry );

util::Timer<T> timer1( converter.getLatticeTime( maxPhysT ), superGeometry.getStatistics().getNvoxel() );
timer1.start();

prepareLattice( sLattice, converter, stlReader, superGeometry );

VTUsurfaceWriter<T> vtuWriter(“surface”, cuboidDecomposition, loadBalancer);
vtuWriter.addSTL( stlReader );

timer1.stop();
timer1.printSummary();

// === 4th Step: Main Loop with Timer ===
clout << “starting simulation…” << std::endl;
util::Timer<T> timer( converter.getLatticeTime( maxPhysT ), superGeometry.getStatistics().getNvoxel() );
timer.start();

for ( std::size_t iT = 0; iT <= converter.getLatticeTime( maxPhysT ); iT++ ) {
// === 5th Step: Definition of Initial and Boundary Conditions ===
setBoundaryValues( sLattice, converter, iT, superGeometry );

// === 6th Step: Collide and Stream Execution ===
sLattice.collideAndStream();

// === 7th Step: Computation and Output of the Results ===
getResults( sLattice, converter, iT, superGeometry, timer, stlReader, vtuWriter );
}

timer.stop();
timer.printSummary();
}

Here is the output with GPU :

utilisateur@LABO-ACC:~/release-1.8.1/examples/turbulence/aorta3d$ make clean; make; ./aorta3d
rm -f tmp/*.* tmp/vtkData/*.* tmp/vtkData/data/*.* tmp/imageData/*.* tmp/imageData/data/*.* tmp/gnuplotData/*.* tmp/gnuplotData/data/*.*
rm -f aorta3d.o aorta3d.d aorta3d.wasm aorta3d.js aorta3d
make -C ../../../external
make[1] : on entre dans le répertoire « /home/utilisateur/release-1.8.1/external »
make -C zlib
make[2] : on entre dans le répertoire « /home/utilisateur/release-1.8.1/external/zlib »
make[2]: rien à faire pour « all ».
make[2] : on quitte le répertoire « /home/utilisateur/release-1.8.1/external/zlib »
cp zlib/build/libz.a lib/
make -C tinyxml2
make[2] : on entre dans le répertoire « /home/utilisateur/release-1.8.1/external/tinyxml2 »
make[2]: rien à faire pour « all ».
make[2] : on quitte le répertoire « /home/utilisateur/release-1.8.1/external/tinyxml2 »
cp tinyxml2/build/libtinyxml2.a lib/
make[1] : on quitte le répertoire « /home/utilisateur/release-1.8.1/external »
make -C ../../.. core
make[1] : on entre dans le répertoire « /home/utilisateur/release-1.8.1 »
make[1]: rien à faire pour « core ».
make[1] : on quitte le répertoire « /home/utilisateur/release-1.8.1 »
nvcc -O3 -std=c++20 –forward-unknown-to-host-compiler -pthread –forward-unknown-to-host-compiler -x cu -O3 -std=c++20 –generate-code=arch=compute_86,code=[compute_86,sm_86] –extended-lambda –expt-relaxed-constexpr -rdc=true -Xcudafe “–diag_suppress=implicit_return_from_non_void_function –display_error_number –diag_suppress=20014 –diag_suppress=20011” -DPLATFORM_CPU_SISD -DPLATFORM_GPU_CUDA -DDEFAULT_FLOATING_POINT_TYPE=double -I../../../src -I../../../external/zlib -I../../../external/tinyxml2 -c -o aorta3d.o aorta3d.cpp
../../../src/utilities/typeIndexedContainers.h(189): warning #20012-D: __device__ annotation is ignored on a function(“TypeIndexedTuple”) that is explicitly defaulted on its first declaration
TypeIndexedTuple() __attribute__((device)) __attribute__((host)) = default;
^

Remark: The warnings can be suppressed with “-diag-suppress <warning-number>”

../../../src/utilities/typeIndexedContainers.h(189): warning #20012-D: __host__ annotation is ignored on a function(“TypeIndexedTuple”) that is explicitly defaulted on its first declaration
TypeIndexedTuple() __attribute__((device)) __attribute__((host)) = default;
^

../../../src/utilities/typeIndexedContainers.h(190): warning #20012-D: __device__ annotation is ignored on a function(“TypeIndexedTuple”) that is explicitly defaulted on its first declaration
TypeIndexedTuple(TypeIndexedTuple&) __attribute__((device)) __attribute__((host)) = default;
^

../../../src/utilities/typeIndexedContainers.h(190): warning #20012-D: __host__ annotation is ignored on a function(“TypeIndexedTuple”) that is explicitly defaulted on its first declaration
TypeIndexedTuple(TypeIndexedTuple&) __attribute__((device)) __attribute__((host)) = default;
^

../../../src/utilities/typeIndexedContainers.h(191): warning #20012-D: __device__ annotation is ignored on a function(“TypeIndexedTuple”) that is explicitly defaulted on its first declaration
TypeIndexedTuple(TypeIndexedTuple&& rhs) __attribute__((device)) __attribute__((host)) = default;
^

../../../src/utilities/typeIndexedContainers.h(191): warning #20012-D: __host__ annotation is ignored on a function(“TypeIndexedTuple”) that is explicitly defaulted on its first declaration
TypeIndexedTuple(TypeIndexedTuple&& rhs) __attribute__((device)) __attribute__((host)) = default;
^

../../../src/io/stlReader.h(219): warning #611-D: overloaded virtual function “olb::IndicatorF3D<S>::distance [with S=T]” is only partially overridden in class “olb::STLreader<T>”
class STLreader : public IndicatorF3D<T> {
^
detected during instantiation of class “olb::STLreader<T> [with T=T]” at line 65 of aorta3d.cpp

../../../src/utilities/typeIndexedContainers.h(189): warning #20012-D: __device__ annotation is ignored on a function(“TypeIndexedTuple”) that is explicitly defaulted on its first declaration
TypeIndexedTuple() __attribute__((device)) __attribute__((host)) = default;
^

Remark: The warnings can be suppressed with “-diag-suppress <warning-number>”

../../../src/utilities/typeIndexedContainers.h(189): warning #20012-D: __host__ annotation is ignored on a function(“TypeIndexedTuple”) that is explicitly defaulted on its first declaration
TypeIndexedTuple() __attribute__((device)) __attribute__((host)) = default;
^

../../../src/utilities/typeIndexedContainers.h(190): warning #20012-D: __device__ annotation is ignored on a function(“TypeIndexedTuple”) that is explicitly defaulted on its first declaration
TypeIndexedTuple(TypeIndexedTuple&) __attribute__((device)) __attribute__((host)) = default;
^

../../../src/utilities/typeIndexedContainers.h(190): warning #20012-D: __host__ annotation is ignored on a function(“TypeIndexedTuple”) that is explicitly defaulted on its first declaration
TypeIndexedTuple(TypeIndexedTuple&) __attribute__((device)) __attribute__((host)) = default;
^

../../../src/utilities/typeIndexedContainers.h(191): warning #20012-D: __device__ annotation is ignored on a function(“TypeIndexedTuple”) that is explicitly defaulted on its first declaration
TypeIndexedTuple(TypeIndexedTuple&& rhs) __attribute__((device)) __attribute__((host)) = default;
^

../../../src/utilities/typeIndexedContainers.h(191): warning #20012-D: __host__ annotation is ignored on a function(“TypeIndexedTuple”) that is explicitly defaulted on its first declaration
TypeIndexedTuple(TypeIndexedTuple&& rhs) __attribute__((device)) __attribute__((host)) = default;
^

../../../src/io/stlReader.h(219): warning #611-D: overloaded virtual function “olb::IndicatorF3D<S>::distance [with S=T]” is only partially overridden in class “olb::STLreader<T>”
class STLreader : public IndicatorF3D<T> {
^
detected during instantiation of class “olb::STLreader<T> [with T=T]” at line 65 of aorta3d.cpp

../../../src/functors/analytical/indicator/indicatorF2D.h(121): warning #611-D: overloaded virtual function “olb::IndicatorF2D<S>::operator() [with S=T]” is only partially overridden in class “olb::IndicatorCircle2D<T>”
class IndicatorCircle2D : public IndicatorF2D<S> {
^
detected during:
instantiation of class “olb::IndicatorCircle2D<S> [with S=T]” at line 248 of ../../../src/functors/lattice/integral/superPlaneIntegralF3D.hh
instantiation of “olb::SuperPlaneIntegralF3D<T>::SuperPlaneIntegralF3D(olb::FunctorPtr<olb::SuperF3D<T, T>> &&, olb::SuperGeometry<T, 3U> &, const olb::IndicatorCircle3D<T> &, std::vector<int, std::allocator<int>>, olb::BlockDataReductionMode) [with T=T]” at line 221 of ../../../src/functors/lattice/integral/superPlaneIntegralFluxF3D.hh
instantiation of “olb::SuperPlaneIntegralFluxF3D<T, F>::SuperPlaneIntegralFluxF3D(olb::SuperLattice<T, DESCRIPTOR> &, const olb::UnitConverter<T, DESCRIPTOR> &, olb::SuperGeometry<T, 3U> &, olb::IndicatorCircle3D<T> &, std::vector<int, std::allocator<int>>, olb::BlockDataReductionMode) [with T=T, F=olb::SuperLatticePhysVelocity3D, DESCRIPTOR=DESCRIPTOR]” at line 164 of ../../../src/functors/lattice/integral/superPlaneIntegralFluxF3D.h

nvcc aorta3d.o -o aorta3d -lolbcore -L../../../external/lib -lpthread -lz -ltinyxml2 -lcuda -lcudadevrt -lcudart -L../../../build/lib
nvcc warning : Support for offline compilation for architectures prior to ‘<compute/sm/lto>_75’ will be removed in a future release (Use -Wno-deprecated-gpu-targets to suppress warning).
nvlink warning : SM Arch (‘sm_52’) not found in ‘aorta3d.o’
[ThreadPool] Sucessfully initialized, numThreads=1
[UnitConverter] —————– UnitConverter information —————–
[UnitConverter] — Parameters:
[UnitConverter] Resolution: N= 40
[UnitConverter] Lattice velocity: latticeU= 0.0225
[UnitConverter] Lattice relaxation frequency: omega= 1.99697
[UnitConverter] Lattice relaxation time: tau= 0.50076
[UnitConverter] Characteristical length(m): charL= 0.02246
[UnitConverter] Characteristical speed(m/s): charU= 0.45
[UnitConverter] Phys. kinematic viscosity(m^2/s): charNu= 2.8436e-06
[UnitConverter] Phys. density(kg/m^d): charRho= 1055
[UnitConverter] Characteristical pressure(N/m^2): charPressure= 0
[UnitConverter] Mach number: machNumber= 0.0389711
[UnitConverter] Reynolds number: reynoldsNumber= 3554.3
[UnitConverter] Knudsen number: knudsenNumber= 1.09645e-05
[UnitConverter] Characteristical CFL number: charCFLnumber= 0.0225
[UnitConverter]
[UnitConverter] — Conversion factors:
[UnitConverter] Voxel length(m): physDeltaX= 0.0005615
[UnitConverter] Time step(s): physDeltaT= 2.8075e-05
[UnitConverter] Velocity factor(m/s): physVelocity= 20
[UnitConverter] Density factor(kg/m^3): physDensity= 1055
[UnitConverter] Mass factor(kg): physMass= 1.86768e-07
[UnitConverter] Viscosity factor(m^2/s): physViscosity= 0.01123
[UnitConverter] Force factor(N): physForce= 0.133049
[UnitConverter] Pressure factor(N/m^2): physPressure= 422000
[UnitConverter] ————————————————————-
[UnitConverter] WARNING:
[UnitConverter] Potentially UNSTABLE combination of relaxation time (tau=0.50076)
[UnitConverter] and characteristical CFL number (lattice velocity) charCFLnumber=0.0225!
[UnitConverter] Potentially maximum characteristical CFL number (maxCharCFLnumber=0.00607715)
[UnitConverter] Actual characteristical CFL number (charCFLnumber=0.0225) > 0.00607715
[UnitConverter] Please reduce the the cell size or the time step size!
[UnitConverter] We recommend to use the cell size of 0.000151659 m and the time step size of 7.58294e-06 s.
[UnitConverter] ————————————————————-
[STLreader] Voxelizing …
[STLmesh] nTriangles=2654; maxDist2=0.000610779
[STLmesh] minPhysR(StlMesh)=(0.199901,0.0900099,0.0117236); maxPhysR(StlMesh)=(0.243584,0.249987,0.0398131)
[Octree] radius=0.143744; center=(0.221602,0.169858,0.025628)
[STLreader] voxelSize=0.0005615; stlSize=0.001
[STLreader] minPhysR(VoxelMesh)=(0.199984,0.0904058,0.0118712); maxPhysR(VoxelMesh)=(0.24322,0.249872,0.0393847)
[STLreader] Voxelizing … OK
[prepareGeometry] Prepare Geometry …
[SuperGeometry3D] cleaned 0 outer boundary voxel(s)
[SuperGeometry3D] cleaned 0 outer boundary voxel(s)
[SuperGeometry3D] cleaned 0 inner boundary voxel(s) of Type 3
[SuperGeometryStatistics3D] updated
[SuperGeometry3D] the model is correct!
[CuboidDecomposition] —Cuboid Structure Statistics—
[CuboidDecomposition] Number of Cuboids: 2
[CuboidDecomposition] Delta : 0.0005615
[CuboidDecomposition] Ratio (min): 0.293706
[CuboidDecomposition] (max): 3.40476
[CuboidDecomposition] Nodes (min): 252252
[CuboidDecomposition] (max): 557424
[CuboidDecomposition] Weight (min): 96042
[CuboidDecomposition] (max): 117790
[CuboidDecomposition] ——————————–
[SuperGeometryStatistics3D] materialNumber=0; count=595844; minPhysR=(0.199984,0.0898443,0.0113097); maxPhysR=(0.243781,0.250433,0.0399462)
[SuperGeometryStatistics3D] materialNumber=1; count=171218; minPhysR=(0.200546,0.0904058,0.0118712); maxPhysR=(0.24322,0.249872,0.0393847)
[SuperGeometryStatistics3D] materialNumber=2; count=41071; minPhysR=(0.199984,0.0898443,0.0113097); maxPhysR=(0.243781,0.250433,0.0399462)
[SuperGeometryStatistics3D] materialNumber=3; count=1059; minPhysR=(0.208407,0.250433,0.0124327); maxPhysR=(0.228059,0.250433,0.0332082)
[SuperGeometryStatistics3D] materialNumber=4; count=245; minPhysR=(0.200546,0.0898443,0.0298392); maxPhysR=(0.210653,0.0898443,0.0388232)
[SuperGeometryStatistics3D] materialNumber=5; count=239; minPhysR=(0.234236,0.0898443,0.0287162); maxPhysR=(0.24322,0.0898443,0.0388232)
[SuperGeometryStatistics3D] countTotal[1e6]=0.809676
[prepareGeometry] Prepare Geometry … OK
[prepareLattice] Prepare Lattice …
[prepareLattice] Prepare Lattice … OK
[Timer]
[Timer] —————-Summary:Timer—————-
[Timer] measured time (rt) : 0.420s
[Timer] measured time (cpu): 0.419s
[Timer] ———————————————
[main] starting simulation…
[Timer] step=0; percent=0; passedTime=0.232; remTime=16527; MLUPs=0
[LatticeStatistics] step=0; physT=0; maxCFL=1.49167e-154; avgEnergy=0; avgRho=1
[SuperPlaneIntegralFluxVelocity3D] regionName=inflow; regionSize[m^2]=0.000310868; volumetricFlowRate[ml/s]=0; meanVelocity[m/s]=0
[SuperPlaneIntegralFluxPressure3D] regionName=inflow; regionSize[m^2]=0.000310868; force[N]=0; meanPressure[mmHg]=0
[SuperPlaneIntegralFluxVelocity3D] regionName=outflow0; regionSize[m^2]=6.40023e-05; volumetricFlowRate[ml/s]=0; meanVelocity[m/s]=0
[SuperPlaneIntegralFluxPressure3D] regionName=outflow0; regionSize[m^2]=6.40023e-05; force[N]=0; meanPressure[mmHg]=0
[SuperPlaneIntegralFluxVelocity3D] regionName=outflow1; regionSize[m^2]=6.21106e-05; volumetricFlowRate[ml/s]=0; meanVelocity[m/s]=0
[SuperPlaneIntegralFluxPressure3D] regionName=outflow1; regionSize[m^2]=6.21106e-05; force[N]=0; meanPressure[mmHg]=0
[getResults] yPlusMax=2.22507e-308
[Timer] step=3562; percent=5.00014; passedTime=8.042; remTime=152.793; MLUPs=97.5124
[LatticeStatistics] step=3562; physT=0.100003; maxCFL=1.49167e-154; avgEnergy=0; avgRho=1
[SuperPlaneIntegralFluxVelocity3D] regionName=inflow; regionSize[m^2]=0.000310868; volumetricFlowRate[ml/s]=5.35481; meanVelocity[m/s]=0.0172253
[SuperPlaneIntegralFluxPressure3D] regionName=inflow; regionSize[m^2]=0.000310868; force[N]=3.21631; meanPressure[mmHg]=77.6032
[SuperPlaneIntegralFluxVelocity3D] regionName=outflow0; regionSize[m^2]=6.40023e-05; volumetricFlowRate[ml/s]=0; meanVelocity[m/s]=0
[SuperPlaneIntegralFluxPressure3D] regionName=outflow0; regionSize[m^2]=6.40023e-05; force[N]=0; meanPressure[mmHg]=0
[SuperPlaneIntegralFluxVelocity3D] regionName=outflow1; regionSize[m^2]=6.21106e-05; volumetricFlowRate[ml/s]=0; meanVelocity[m/s]=0
[SuperPlaneIntegralFluxPressure3D] regionName=outflow1; regionSize[m^2]=6.21106e-05; force[N]=0; meanPressure[mmHg]=0
[getResults] yPlusMax=7.14391
[Timer] step=7124; percent=10.0003; passedTime=15.87; remTime=142.826; MLUPs=97.3007
[LatticeStatistics] step=7124; physT=0.200006; maxCFL=1.49167e-154; avgEnergy=0; avgRho=1
[SuperPlaneIntegralFluxVelocity3D] regionName=inflow; regionSize[m^2]=0.000310868; volumetricFlowRate[ml/s]=12.8358; meanVelocity[m/s]=0.04129
[SuperPlaneIntegralFluxPressure3D] regionName=inflow; regionSize[m^2]=0.000310868; force[N]=24.4992; meanPressure[mmHg]=591.118
[SuperPlaneIntegralFluxVelocity3D] regionName=outflow0; regionSize[m^2]=6.40023e-05; volumetricFlowRate[ml/s]=0; meanVelocity[m/s]=0
[SuperPlaneIntegralFluxPressure3D] regionName=outflow0; regionSize[m^2]=6.40023e-05; force[N]=0; meanPressure[mmHg]=0
[SuperPlaneIntegralFluxVelocity3D] regionName=outflow1; regionSize[m^2]=6.21106e-05; volumetricFlowRate[ml/s]=0; meanVelocity[m/s]=0
[SuperPlaneIntegralFluxPressure3D] regionName=outflow1; regionSize[m^2]=6.21106e-05; force[N]=0; meanPressure[mmHg]=0
[getResults] yPlusMax=23.7621
[Timer] step=10686; percent=15.0004; passedTime=23.708; remTime=134.341; MLUPs=97.1765
[LatticeStatistics] step=10686; physT=0.300009; maxCFL=1.49167e-154; avgEnergy=0; avgRho=1
[SuperPlaneIntegralFluxVelocity3D] regionName=inflow; regionSize[m^2]=0.000310868; volumetricFlowRate[ml/s]=13.5503; meanVelocity[m/s]=0.0435886
[SuperPlaneIntegralFluxPressure3D] regionName=inflow; regionSize[m^2]=0.000310868; force[N]=75.1471; meanPressure[mmHg]=1813.15
[SuperPlaneIntegralFluxVelocity3D] regionName=outflow0; regionSize[m^2]=6.40023e-05; volumetricFlowRate[ml/s]=0; meanVelocity[m/s]=0
[SuperPlaneIntegralFluxPressure3D] regionName=outflow0; regionSize[m^2]=6.40023e-05; force[N]=0; meanPressure[mmHg]=0
[SuperPlaneIntegralFluxVelocity3D] regionName=outflow1; regionSize[m^2]=6.21106e-05; volumetricFlowRate[ml/s]=0; meanVelocity[m/s]=0
[SuperPlaneIntegralFluxPressure3D] regionName=outflow1; regionSize[m^2]=6.21106e-05; force[N]=0; meanPressure[mmHg]=0
[getResults] yPlusMax=126.305
[Timer] step=14248; percent=20.0006; passedTime=31.56; remTime=126.236; MLUPs=96.9909
[LatticeStatistics] step=14248; physT=0.400013; maxCFL=1.49167e-154; avgEnergy=0; avgRho=1
[SuperPlaneIntegralFluxVelocity3D] regionName=inflow; regionSize[m^2]=0.000310868; volumetricFlowRate[ml/s]=11.5981; meanVelocity[m/s]=0.0373087
[SuperPlaneIntegralFluxPressure3D] regionName=inflow; regionSize[m^2]=0.000310868; force[N]=155.24; meanPressure[mmHg]=3745.63
[SuperPlaneIntegralFluxVelocity3D] regionName=outflow0; regionSize[m^2]=6.40023e-05; volumetricFlowRate[ml/s]=0; meanVelocity[m/s]=0
[SuperPlaneIntegralFluxPressure3D] regionName=outflow0; regionSize[m^2]=6.40023e-05; force[N]=0; meanPressure[mmHg]=0
[SuperPlaneIntegralFluxVelocity3D] regionName=outflow1; regionSize[m^2]=6.21106e-05; volumetricFlowRate[ml/s]=0; meanVelocity[m/s]=0
[SuperPlaneIntegralFluxPressure3D] regionName=outflow1; regionSize[m^2]=6.21106e-05; force[N]=0; meanPressure[mmHg]=0
[getResults] yPlusMax=49.5797
[Timer] step=17810; percent=25.0007; passedTime=39.439; remTime=118.313; MLUPs=96.6708
[LatticeStatistics] step=17810; physT=0.500016; maxCFL=1.49167e-154; avgEnergy=0; avgRho=1
[SuperPlaneIntegralFluxVelocity3D] regionName=inflow; regionSize[m^2]=0.000310868; volumetricFlowRate[ml/s]=8.12449; meanVelocity[m/s]=0.0261348
[SuperPlaneIntegralFluxPressure3D] regionName=inflow; regionSize[m^2]=0.000310868; force[N]=253.61; meanPressure[mmHg]=6119.11
[SuperPlaneIntegralFluxVelocity3D] regionName=outflow0; regionSize[m^2]=6.40023e-05; volumetricFlowRate[ml/s]=0; meanVelocity[m/s]=0
[SuperPlaneIntegralFluxPressure3D] regionName=outflow0; regionSize[m^2]=6.40023e-05; force[N]=0; meanPressure[mmHg]=0
[SuperPlaneIntegralFluxVelocity3D] regionName=outflow1; regionSize[m^2]=6.21106e-05; volumetricFlowRate[ml/s]=0; meanVelocity[m/s]=0
[SuperPlaneIntegralFluxPressure3D] regionName=outflow1; regionSize[m^2]=6.21106e-05; force[N]=0; meanPressure[mmHg]=0
[getResults] yPlusMax=48.3407
[Timer] step=21372; percent=30.0008; passedTime=47.333; remTime=110.439; MLUPs=96.4749
[LatticeStatistics] step=21372; physT=0.600019; maxCFL=1.49167e-154; avgEnergy=0; avgRho=1
[SuperPlaneIntegralFluxVelocity3D] regionName=inflow; regionSize[m^2]=0.000310868; volumetricFlowRate[ml/s]=5.51686; meanVelocity[m/s]=0.0177466
[SuperPlaneIntegralFluxPressure3D] regionName=inflow; regionSize[m^2]=0.000310868; force[N]=352.107; meanPressure[mmHg]=8495.64
[SuperPlaneIntegralFluxVelocity3D] regionName=outflow0; regionSize[m^2]=6.40023e-05; volumetricFlowRate[ml/s]=0; meanVelocity[m/s]=0
[SuperPlaneIntegralFluxPressure3D] regionName=outflow0; regionSize[m^2]=6.40023e-05; force[N]=0; meanPressure[mmHg]=0
[SuperPlaneIntegralFluxVelocity3D] regionName=outflow1; regionSize[m^2]=6.21106e-05; volumetricFlowRate[ml/s]=0; meanVelocity[m/s]=0
[SuperPlaneIntegralFluxPressure3D] regionName=outflow1; regionSize[m^2]=6.21106e-05; force[N]=0; meanPressure[mmHg]=0
[getResults] yPlusMax=70.7212
[Timer] step=24934; percent=35.001; passedTime=55.232; remTime=102.569; MLUPs=96.4261
[LatticeStatistics] step=24934; physT=0.700022; maxCFL=1.49167e-154; avgEnergy=0; avgRho=1
[SuperPlaneIntegralFluxVelocity3D] regionName=inflow; regionSize[m^2]=0.000310868; volumetricFlowRate[ml/s]=3.34209; meanVelocity[m/s]=0.0107508
[SuperPlaneIntegralFluxPressure3D] regionName=inflow; regionSize[m^2]=0.000310868; force[N]=432.532; meanPressure[mmHg]=10436.1
[SuperPlaneIntegralFluxVelocity3D] regionName=outflow0; regionSize[m^2]=6.40023e-05; volumetricFlowRate[ml/s]=0; meanVelocity[m/s]=0
[SuperPlaneIntegralFluxPressure3D] regionName=outflow0; regionSize[m^2]=6.40023e-05; force[N]=0; meanPressure[mmHg]=0
[SuperPlaneIntegralFluxVelocity3D] regionName=outflow1; regionSize[m^2]=6.21106e-05; volumetricFlowRate[ml/s]=0; meanVelocity[m/s]=0
[SuperPlaneIntegralFluxPressure3D] regionName=outflow1; regionSize[m^2]=6.21106e-05; force[N]=0; meanPressure[mmHg]=0
[getResults] yPlusMax=70.9493
[Timer] step=28496; percent=40.0011; passedTime=63.095; remTime=94.6381; MLUPs=96.8552
[LatticeStatistics] step=28496; physT=0.800025; maxCFL=1.49167e-154; avgEnergy=0; avgRho=1
[SuperPlaneIntegralFluxVelocity3D] regionName=inflow; regionSize[m^2]=0.000310868; volumetricFlowRate[ml/s]=1.59319; meanVelocity[m/s]=0.00512497
[SuperPlaneIntegralFluxPressure3D] regionName=inflow; regionSize[m^2]=0.000310868; force[N]=483.591; meanPressure[mmHg]=11668.1
[SuperPlaneIntegralFluxVelocity3D] regionName=outflow0; regionSize[m^2]=6.40023e-05; volumetricFlowRate[ml/s]=0; meanVelocity[m/s]=0
[SuperPlaneIntegralFluxPressure3D] regionName=outflow0; regionSize[m^2]=6.40023e-05; force[N]=0; meanPressure[mmHg]=0
[SuperPlaneIntegralFluxVelocity3D] regionName=outflow1; regionSize[m^2]=6.21106e-05; volumetricFlowRate[ml/s]=0; meanVelocity[m/s]=0
[SuperPlaneIntegralFluxPressure3D] regionName=outflow1; regionSize[m^2]=6.21106e-05; force[N]=0; meanPressure[mmHg]=0
[getResults] yPlusMax=20.4464
[Timer] step=32058; percent=45.0013; passedTime=70.964; remTime=86.7294; MLUPs=96.7937
[LatticeStatistics] step=32058; physT=0.900028; maxCFL=1.49167e-154; avgEnergy=0; avgRho=1
[SuperPlaneIntegralFluxVelocity3D] regionName=inflow; regionSize[m^2]=0.000310868; volumetricFlowRate[ml/s]=0.4174; meanVelocity[m/s]=0.00134269
[SuperPlaneIntegralFluxPressure3D] regionName=inflow; regionSize[m^2]=0.000310868; force[N]=505.206; meanPressure[mmHg]=12189.6
[SuperPlaneIntegralFluxVelocity3D] regionName=outflow0; regionSize[m^2]=6.40023e-05; volumetricFlowRate[ml/s]=0; meanVelocity[m/s]=0
[SuperPlaneIntegralFluxPressure3D] regionName=outflow0; regionSize[m^2]=6.40023e-05; force[N]=0; meanPressure[mmHg]=0
[SuperPlaneIntegralFluxVelocity3D] regionName=outflow1; regionSize[m^2]=6.21106e-05; volumetricFlowRate[ml/s]=0; meanVelocity[m/s]=0
[SuperPlaneIntegralFluxPressure3D] regionName=outflow1; regionSize[m^2]=6.21106e-05; force[N]=0; meanPressure[mmHg]=0
[getResults] yPlusMax=20.5769
[Timer] step=35620; percent=50.0014; passedTime=78.814; remTime=78.8096; MLUPs=97.028
[LatticeStatistics] step=35620; physT=1.00003; maxCFL=1.49167e-154; avgEnergy=0; avgRho=1
[SuperPlaneIntegralFluxVelocity3D] regionName=inflow; regionSize[m^2]=0.000310868; volumetricFlowRate[ml/s]=-8.46732e-06; meanVelocity[m/s]=-2.72376e-08
[SuperPlaneIntegralFluxPressure3D] regionName=inflow; regionSize[m^2]=0.000310868; force[N]=508.549; meanPressure[mmHg]=12270.3
[SuperPlaneIntegralFluxVelocity3D] regionName=outflow0; regionSize[m^2]=6.40023e-05; volumetricFlowRate[ml/s]=0; meanVelocity[m/s]=0
[SuperPlaneIntegralFluxPressure3D] regionName=outflow0; regionSize[m^2]=6.40023e-05; force[N]=0; meanPressure[mmHg]=0
[SuperPlaneIntegralFluxVelocity3D] regionName=outflow1; regionSize[m^2]=6.21106e-05; volumetricFlowRate[ml/s]=0; meanVelocity[m/s]=0
[SuperPlaneIntegralFluxPressure3D] regionName=outflow1; regionSize[m^2]=6.21106e-05; force[N]=0; meanPressure[mmHg]=0
[getResults] yPlusMax=20.5963
[Timer] step=39182; percent=55.0015; passedTime=86.7; remTime=70.9319; MLUPs=96.5728
[LatticeStatistics] step=39182; physT=1.10003; maxCFL=1.49167e-154; avgEnergy=0; avgRho=1
[SuperPlaneIntegralFluxVelocity3D] regionName=inflow; regionSize[m^2]=0.000310868; volumetricFlowRate[ml/s]=0.406064; meanVelocity[m/s]=0.00130622
[SuperPlaneIntegralFluxPressure3D] regionName=inflow; regionSize[m^2]=0.000310868; force[N]=511.771; meanPressure[mmHg]=12348
[SuperPlaneIntegralFluxVelocity3D] regionName=outflow0; regionSize[m^2]=6.40023e-05; volumetricFlowRate[ml/s]=0; meanVelocity[m/s]=0
[SuperPlaneIntegralFluxPressure3D] regionName=outflow0; regionSize[m^2]=6.40023e-05; force[N]=0; meanPressure[mmHg]=0
[SuperPlaneIntegralFluxVelocity3D] regionName=outflow1; regionSize[m^2]=6.21106e-05; volumetricFlowRate[ml/s]=0; meanVelocity[m/s]=0
[SuperPlaneIntegralFluxPressure3D] regionName=outflow1; regionSize[m^2]=6.21106e-05; force[N]=0; meanPressure[mmHg]=0
[getResults] yPlusMax=20.6148
[Timer] step=42744; percent=60.0017; passedTime=94.599; remTime=63.0616; MLUPs=96.4139
[LatticeStatistics] step=42744; physT=1.20004; maxCFL=1.49167e-154; avgEnergy=0; avgRho=1
[SuperPlaneIntegralFluxVelocity3D] regionName=inflow; regionSize[m^2]=0.000310868; volumetricFlowRate[ml/s]=1.42205; meanVelocity[m/s]=0.00457443
[SuperPlaneIntegralFluxPressure3D] regionName=inflow; regionSize[m^2]=0.000310868; force[N]=533.068; meanPressure[mmHg]=12861.9
[SuperPlaneIntegralFluxVelocity3D] regionName=outflow0; regionSize[m^2]=6.40023e-05; volumetricFlowRate[ml/s]=0; meanVelocity[m/s]=0
[SuperPlaneIntegralFluxPressure3D] regionName=outflow0; regionSize[m^2]=6.40023e-05; force[N]=0; meanPressure[mmHg]=0
[SuperPlaneIntegralFluxVelocity3D] regionName=outflow1; regionSize[m^2]=6.21106e-05; volumetricFlowRate[ml/s]=0; meanVelocity[m/s]=0
[SuperPlaneIntegralFluxPressure3D] regionName=outflow1; regionSize[m^2]=6.21106e-05; force[N]=0; meanPressure[mmHg]=0
[getResults] yPlusMax=20.7326
[Timer] step=46306; percent=65.0018; passedTime=102.522; remTime=55.1997; MLUPs=96.134
[LatticeStatistics] step=46306; physT=1.30004; maxCFL=1.49167e-154; avgEnergy=0; avgRho=1
[SuperPlaneIntegralFluxVelocity3D] regionName=inflow; regionSize[m^2]=0.000310868; volumetricFlowRate[ml/s]=2.49626; meanVelocity[m/s]=0.00802995
[SuperPlaneIntegralFluxPressure3D] regionName=inflow; regionSize[m^2]=0.000310868; force[N]=583.735; meanPressure[mmHg]=14084.4
[SuperPlaneIntegralFluxVelocity3D] regionName=outflow0; regionSize[m^2]=6.40023e-05; volumetricFlowRate[ml/s]=0; meanVelocity[m/s]=0
[SuperPlaneIntegralFluxPressure3D] regionName=outflow0; regionSize[m^2]=6.40023e-05; force[N]=0; meanPressure[mmHg]=0
[SuperPlaneIntegralFluxVelocity3D] regionName=outflow1; regionSize[m^2]=6.21106e-05; volumetricFlowRate[ml/s]=0; meanVelocity[m/s]=0
[SuperPlaneIntegralFluxPressure3D] regionName=outflow1; regionSize[m^2]=6.21106e-05; force[N]=0; meanPressure[mmHg]=0
[getResults] yPlusMax=21.8774
[Timer] step=49868; percent=70.002; passedTime=110.432; remTime=47.3236; MLUPs=96.292
[LatticeStatistics] step=49868; physT=1.40004; maxCFL=1.49167e-154; avgEnergy=0; avgRho=1
[SuperPlaneIntegralFluxVelocity3D] regionName=inflow; regionSize[m^2]=0.000310868; volumetricFlowRate[ml/s]=3.0358; meanVelocity[m/s]=0.00976554
[SuperPlaneIntegralFluxPressure3D] regionName=inflow; regionSize[m^2]=0.000310868; force[N]=663.841; meanPressure[mmHg]=16017.2
[SuperPlaneIntegralFluxVelocity3D] regionName=outflow0; regionSize[m^2]=6.40023e-05; volumetricFlowRate[ml/s]=0; meanVelocity[m/s]=0
[SuperPlaneIntegralFluxPressure3D] regionName=outflow0; regionSize[m^2]=6.40023e-05; force[N]=0; meanPressure[mmHg]=0
[SuperPlaneIntegralFluxVelocity3D] regionName=outflow1; regionSize[m^2]=6.21106e-05; volumetricFlowRate[ml/s]=0; meanVelocity[m/s]=0
[SuperPlaneIntegralFluxPressure3D] regionName=outflow1; regionSize[m^2]=6.21106e-05; force[N]=0; meanPressure[mmHg]=0
[getResults] yPlusMax=70.2012
[Timer] step=53430; percent=75.0021; passedTime=118.364; remTime=39.4502; MLUPs=96.0249
[LatticeStatistics] step=53430; physT=1.50005; maxCFL=1.49167e-154; avgEnergy=0; avgRho=1
[SuperPlaneIntegralFluxVelocity3D] regionName=inflow; regionSize[m^2]=0.000310868; volumetricFlowRate[ml/s]=2.20282; meanVelocity[m/s]=0.00708603
[SuperPlaneIntegralFluxPressure3D] regionName=inflow; regionSize[m^2]=0.000310868; force[N]=762.216; meanPressure[mmHg]=18390.8
[SuperPlaneIntegralFluxVelocity3D] regionName=outflow0; regionSize[m^2]=6.40023e-05; volumetricFlowRate[ml/s]=0; meanVelocity[m/s]=0
[SuperPlaneIntegralFluxPressure3D] regionName=outflow0; regionSize[m^2]=6.40023e-05; force[N]=0; meanPressure[mmHg]=0
[SuperPlaneIntegralFluxVelocity3D] regionName=outflow1; regionSize[m^2]=6.21106e-05; volumetricFlowRate[ml/s]=0; meanVelocity[m/s]=0
[SuperPlaneIntegralFluxPressure3D] regionName=outflow1; regionSize[m^2]=6.21106e-05; force[N]=0; meanPressure[mmHg]=0
[getResults] yPlusMax=21.6485
[Timer] step=56992; percent=80.0022; passedTime=126.298; remTime=31.5701; MLUPs=95.9886
[LatticeStatistics] step=56992; physT=1.60005; maxCFL=1.49167e-154; avgEnergy=0; avgRho=1
[SuperPlaneIntegralFluxVelocity3D] regionName=inflow; regionSize[m^2]=0.000310868; volumetricFlowRate[ml/s]=2.40087; meanVelocity[m/s]=0.00772312
[SuperPlaneIntegralFluxPressure3D] regionName=inflow; regionSize[m^2]=0.000310868; force[N]=860.708; meanPressure[mmHg]=20767.2
[SuperPlaneIntegralFluxVelocity3D] regionName=outflow0; regionSize[m^2]=6.40023e-05; volumetricFlowRate[ml/s]=0; meanVelocity[m/s]=0
[SuperPlaneIntegralFluxPressure3D] regionName=outflow0; regionSize[m^2]=6.40023e-05; force[N]=0; meanPressure[mmHg]=0
[SuperPlaneIntegralFluxVelocity3D] regionName=outflow1; regionSize[m^2]=6.21106e-05; volumetricFlowRate[ml/s]=0; meanVelocity[m/s]=0
[SuperPlaneIntegralFluxPressure3D] regionName=outflow1; regionSize[m^2]=6.21106e-05; force[N]=0; meanPressure[mmHg]=0
[getResults] yPlusMax=21.9221
[Timer] step=60554; percent=85.0024; passedTime=134.257; remTime=23.688; MLUPs=95.6992
[LatticeStatistics] step=60554; physT=1.70005; maxCFL=1.49167e-154; avgEnergy=0; avgRho=1
[SuperPlaneIntegralFluxVelocity3D] regionName=inflow; regionSize[m^2]=0.000310868; volumetricFlowRate[ml/s]=1.54919; meanVelocity[m/s]=0.00498344
[SuperPlaneIntegralFluxPressure3D] regionName=inflow; regionSize[m^2]=0.000310868; force[N]=941.118; meanPressure[mmHg]=22707.3
[SuperPlaneIntegralFluxVelocity3D] regionName=outflow0; regionSize[m^2]=6.40023e-05; volumetricFlowRate[ml/s]=0; meanVelocity[m/s]=0
[SuperPlaneIntegralFluxPressure3D] regionName=outflow0; regionSize[m^2]=6.40023e-05; force[N]=0; meanPressure[mmHg]=0
[SuperPlaneIntegralFluxVelocity3D] regionName=outflow1; regionSize[m^2]=6.21106e-05; volumetricFlowRate[ml/s]=0; meanVelocity[m/s]=0
[SuperPlaneIntegralFluxPressure3D] regionName=outflow1; regionSize[m^2]=6.21106e-05; force[N]=0; meanPressure[mmHg]=0
[getResults] yPlusMax=22.1479
[Timer] step=64116; percent=90.0025; passedTime=142.197; remTime=15.7952; MLUPs=95.9282
[LatticeStatistics] step=64116; physT=1.80006; maxCFL=1.49167e-154; avgEnergy=0; avgRho=1
[SuperPlaneIntegralFluxVelocity3D] regionName=inflow; regionSize[m^2]=0.000310868; volumetricFlowRate[ml/s]=-0.217778; meanVelocity[m/s]=-0.000700546
[SuperPlaneIntegralFluxPressure3D] regionName=inflow; regionSize[m^2]=0.000310868; force[N]=992.16; meanPressure[mmHg]=23938.9
[SuperPlaneIntegralFluxVelocity3D] regionName=outflow0; regionSize[m^2]=6.40023e-05; volumetricFlowRate[ml/s]=0; meanVelocity[m/s]=0
[SuperPlaneIntegralFluxPressure3D] regionName=outflow0; regionSize[m^2]=6.40023e-05; force[N]=0; meanPressure[mmHg]=0
[SuperPlaneIntegralFluxVelocity3D] regionName=outflow1; regionSize[m^2]=6.21106e-05; volumetricFlowRate[ml/s]=0; meanVelocity[m/s]=0
[SuperPlaneIntegralFluxPressure3D] regionName=outflow1; regionSize[m^2]=6.21106e-05; force[N]=0; meanPressure[mmHg]=0
[getResults] yPlusMax=22.2753
[Timer] step=67678; percent=95.0027; passedTime=150.175; remTime=7.89951; MLUPs=95.4712
[LatticeStatistics] step=67678; physT=1.90006; maxCFL=1.49167e-154; avgEnergy=0; avgRho=1
[SuperPlaneIntegralFluxVelocity3D] regionName=inflow; regionSize[m^2]=0.000310868; volumetricFlowRate[ml/s]=0.239828; meanVelocity[m/s]=0.000771479
[SuperPlaneIntegralFluxPressure3D] regionName=inflow; regionSize[m^2]=0.000310868; force[N]=1013.76; meanPressure[mmHg]=24460.1
[SuperPlaneIntegralFluxVelocity3D] regionName=outflow0; regionSize[m^2]=6.40023e-05; volumetricFlowRate[ml/s]=0; meanVelocity[m/s]=0
[SuperPlaneIntegralFluxPressure3D] regionName=outflow0; regionSize[m^2]=6.40023e-05; force[N]=0; meanPressure[mmHg]=0
[SuperPlaneIntegralFluxVelocity3D] regionName=outflow1; regionSize[m^2]=6.21106e-05; volumetricFlowRate[ml/s]=0; meanVelocity[m/s]=0
[SuperPlaneIntegralFluxPressure3D] regionName=outflow1; regionSize[m^2]=6.21106e-05; force[N]=0; meanPressure[mmHg]=0
[getResults] yPlusMax=22.3259
[Timer]
[Timer] —————-Summary:Timer—————-
[Timer] measured time (rt) : 158.10s
[Timer] measured time (cpu): 163.815s
[Timer] average MLUPs : 91.587
[Timer] average MLUPps: 91.587
[Timer] ———————————————

And here is the output with CPU (the compilation of OpenLB has be done with the cpu_gcc_openmpi.mk file) :

utilisateur@LABO-ACC:~/release-1.8.1/examples/turbulence/aorta3d$ make clean; make; mpirun -np 32 ./aorta3d
rm -f tmp/*.* tmp/vtkData/*.* tmp/vtkData/data/*.* tmp/imageData/*.* tmp/imageData/data/*.* tmp/gnuplotData/*.* tmp/gnuplotData/data/*.*
rm -f aorta3d.o aorta3d.d aorta3d.wasm aorta3d.js aorta3d
make -C ../../../external
make[1] : on entre dans le répertoire « /home/utilisateur/release-1.8.1/external »
make -C zlib
make[2] : on entre dans le répertoire « /home/utilisateur/release-1.8.1/external/zlib »
make[2]: rien à faire pour « all ».
make[2] : on quitte le répertoire « /home/utilisateur/release-1.8.1/external/zlib »
cp zlib/build/libz.a lib/
make -C tinyxml2
make[2] : on entre dans le répertoire « /home/utilisateur/release-1.8.1/external/tinyxml2 »
make[2]: rien à faire pour « all ».
make[2] : on quitte le répertoire « /home/utilisateur/release-1.8.1/external/tinyxml2 »
cp tinyxml2/build/libtinyxml2.a lib/
make[1] : on quitte le répertoire « /home/utilisateur/release-1.8.1/external »
make -C ../../.. core
make[1] : on entre dans le répertoire « /home/utilisateur/release-1.8.1 »
make[1]: rien à faire pour « core ».
make[1] : on quitte le répertoire « /home/utilisateur/release-1.8.1 »
mpic++ -O3 -Wall -march=native -mtune=native -std=c++20 -pthread -DPARALLEL_MODE_MPI -DPLATFORM_CPU_SISD -DDEFAULT_FLOATING_POINT_TYPE=double -I../../../src -I../../../external/zlib -I../../../external/tinyxml2 -c -o aorta3d.o aorta3d.cpp
mpic++ aorta3d.o -o aorta3d -lolbcore -L../../../external/lib -lpthread -lz -ltinyxml2 -L../../../build/lib
Authorization required, but no authorization protocol specified

Authorization required, but no authorization protocol specified

[MpiManager] Sucessfully initialized, numThreads=32
[ThreadPool] Sucessfully initialized, numThreads=1
[UnitConverter] —————– UnitConverter information —————–
[UnitConverter] — Parameters:
[UnitConverter] Resolution: N= 40
[UnitConverter] Lattice velocity: latticeU= 0.0225
[UnitConverter] Lattice relaxation frequency: omega= 1.99697
[UnitConverter] Lattice relaxation time: tau= 0.50076
[UnitConverter] Characteristical length(m): charL= 0.02246
[UnitConverter] Characteristical speed(m/s): charU= 0.45
[UnitConverter] Phys. kinematic viscosity(m^2/s): charNu= 2.8436e-06
[UnitConverter] Phys. density(kg/m^d): charRho= 1055
[UnitConverter] Characteristical pressure(N/m^2): charPressure= 0
[UnitConverter] Mach number: machNumber= 0.0389711
[UnitConverter] Reynolds number: reynoldsNumber= 3554.3
[UnitConverter] Knudsen number: knudsenNumber= 1.09645e-05
[UnitConverter] Characteristical CFL number: charCFLnumber= 0.0225
[UnitConverter]
[UnitConverter] — Conversion factors:
[UnitConverter] Voxel length(m): physDeltaX= 0.0005615
[UnitConverter] Time step(s): physDeltaT= 2.8075e-05
[UnitConverter] Velocity factor(m/s): physVelocity= 20
[UnitConverter] Density factor(kg/m^3): physDensity= 1055
[UnitConverter] Mass factor(kg): physMass= 1.86768e-07
[UnitConverter] Viscosity factor(m^2/s): physViscosity= 0.01123
[UnitConverter] Force factor(N): physForce= 0.133049
[UnitConverter] Pressure factor(N/m^2): physPressure= 422000
[UnitConverter] ————————————————————-
[UnitConverter] WARNING:
[UnitConverter] Potentially UNSTABLE combination of relaxation time (tau=0.50076)
[UnitConverter] and characteristical CFL number (lattice velocity) charCFLnumber=0.0225!
[UnitConverter] Potentially maximum characteristical CFL number (maxCharCFLnumber=0.00607715)
[UnitConverter] Actual characteristical CFL number (charCFLnumber=0.0225) > 0.00607715
[UnitConverter] Please reduce the the cell size or the time step size!
[UnitConverter] We recommend to use the cell size of 0.000151659 m and the time step size of 7.58294e-06 s.
[UnitConverter] ————————————————————-
[STLreader] Voxelizing …
[STLmesh] nTriangles=2654; maxDist2=0.000610779
[STLmesh] minPhysR(StlMesh)=(0.199901,0.0900099,0.0117236); maxPhysR(StlMesh)=(0.243584,0.249987,0.0398131)
[Octree] radius=0.143744; center=(0.221602,0.169858,0.025628)
[STLreader] voxelSize=0.0005615; stlSize=0.001
[STLreader] minPhysR(VoxelMesh)=(0.199984,0.0904058,0.0118712); maxPhysR(VoxelMesh)=(0.24322,0.249872,0.0393847)
[STLreader] Voxelizing … OK
[prepareGeometry] Prepare Geometry …
[SuperGeometry3D] cleaned 0 outer boundary voxel(s)
[SuperGeometry3D] cleaned 0 outer boundary voxel(s)
[SuperGeometry3D] cleaned 0 inner boundary voxel(s) of Type 3
[SuperGeometryStatistics3D] updated
[SuperGeometry3D] the model is correct!
[CuboidDecomposition] —Cuboid Structure Statistics—
[CuboidDecomposition] Number of Cuboids: 256
[CuboidDecomposition] Delta : 0.0005615
[CuboidDecomposition] Ratio (min): 0.444444
[CuboidDecomposition] (max): 2.5
[CuboidDecomposition] Nodes (min): 360
[CuboidDecomposition] (max): 1800
[CuboidDecomposition] Weight (min): 277
[CuboidDecomposition] (max): 1800
[CuboidDecomposition] ——————————–
[SuperGeometryStatistics3D] materialNumber=0; count=60575; minPhysR=(0.199984,0.0898443,0.0113097); maxPhysR=(0.243781,0.250433,0.0399462)
[SuperGeometryStatistics3D] materialNumber=1; count=171218; minPhysR=(0.200546,0.0904058,0.0118712); maxPhysR=(0.24322,0.249872,0.0393847)
[SuperGeometryStatistics3D] materialNumber=2; count=41071; minPhysR=(0.199984,0.0898443,0.0113097); maxPhysR=(0.243781,0.250433,0.0399462)
[SuperGeometryStatistics3D] materialNumber=3; count=1059; minPhysR=(0.208407,0.250433,0.0124327); maxPhysR=(0.228059,0.250433,0.0332082)
[SuperGeometryStatistics3D] materialNumber=4; count=245; minPhysR=(0.200546,0.0898443,0.0298392); maxPhysR=(0.210653,0.0898443,0.0388232)
[SuperGeometryStatistics3D] materialNumber=5; count=239; minPhysR=(0.234236,0.0898443,0.0287162); maxPhysR=(0.24322,0.0898443,0.0388232)
[SuperGeometryStatistics3D] countTotal[1e6]=0.274407
[prepareGeometry] Prepare Geometry … OK
[prepareLattice] Prepare Lattice …
[prepareLattice] Prepare Lattice … OK
[Timer]
[Timer] —————-Summary:Timer—————-
[Timer] measured time (rt) : 0.103s
[Timer] measured time (cpu): 0.103s
[Timer] ———————————————
[main] starting simulation…
[Timer] step=0; percent=0; passedTime=0.111; remTime=7907.31; MLUPs=0
[LatticeStatistics] step=0; physT=0; maxCFL=1.49167e-154; avgEnergy=0; avgRho=1
[SuperPlaneIntegralFluxVelocity3D] regionName=inflow; regionSize[m^2]=0.000310868; volumetricFlowRate[ml/s]=0; meanVelocity[m/s]=0
[SuperPlaneIntegralFluxPressure3D] regionName=inflow; regionSize[m^2]=0.000310868; force[N]=0; meanPressure[mmHg]=0
[SuperPlaneIntegralFluxVelocity3D] regionName=outflow0; regionSize[m^2]=6.40023e-05; volumetricFlowRate[ml/s]=0; meanVelocity[m/s]=0
[SuperPlaneIntegralFluxPressure3D] regionName=outflow0; regionSize[m^2]=6.40023e-05; force[N]=0; meanPressure[mmHg]=0
[SuperPlaneIntegralFluxVelocity3D] regionName=outflow1; regionSize[m^2]=6.21106e-05; volumetricFlowRate[ml/s]=0; meanVelocity[m/s]=0
[SuperPlaneIntegralFluxPressure3D] regionName=outflow1; regionSize[m^2]=6.21106e-05; force[N]=0; meanPressure[mmHg]=0
[getResults] yPlusMax=2.22507e-308
[Timer] step=3562; percent=5.00014; passedTime=134.9; remTime=2563.02; MLUPs=5.65083
[LatticeStatistics] step=3562; physT=0.100003; maxCFL=0.0360234; avgEnergy=5.51886e-05; avgRho=1.00067
[SuperPlaneIntegralFluxVelocity3D] regionName=inflow; regionSize[m^2]=0.000310868; volumetricFlowRate[ml/s]=6.88058; meanVelocity[m/s]=0.0221334
[SuperPlaneIntegralFluxPressure3D] regionName=inflow; regionSize[m^2]=0.000310868; force[N]=0.0477156; meanPressure[mmHg]=1.15128
[SuperPlaneIntegralFluxVelocity3D] regionName=outflow0; regionSize[m^2]=6.40023e-05; volumetricFlowRate[ml/s]=-2.97519; meanVelocity[m/s]=-0.0464857
[SuperPlaneIntegralFluxPressure3D] regionName=outflow0; regionSize[m^2]=6.40023e-05; force[N]=0.000318598; meanPressure[mmHg]=0.0373375
[SuperPlaneIntegralFluxVelocity3D] regionName=outflow1; regionSize[m^2]=6.21106e-05; volumetricFlowRate[ml/s]=-2.84465; meanVelocity[m/s]=-0.0457997
[SuperPlaneIntegralFluxPressure3D] regionName=outflow1; regionSize[m^2]=6.21106e-05; force[N]=0.000319367; meanPressure[mmHg]=0.0385676
[getResults] yPlusMax=1.4707

I killed the simulation here because it is enough to see the difference with the outflow.

Thanks in advance and good luck !

[sfraniatte]

I forgot to say that I have changed the floating point type to double for a good voxelization. Here is my config.mk file :

# Example build config for OpenLB using CUDA on single GPU systems
#
# Tested using CUDA 11.4
#
# Usage:
#  - Copy this file to OpenLB root as <code>config.mk</code>
#  - Adjust CUDA_ARCH to match your specifc GPU
#  - Run <code>make clean; make</code>
#  - Switch to example directory, e.g. <code>examples/laminar/cavity3dBenchmark</code>
#  - Run <code>make</code>
#  - Start the simulation using <code>./cavity3d</code>

CXX             := nvcc
CC              := nvcc

CXXFLAGS        := -O3
CXXFLAGS        += -std=c++20 --forward-unknown-to-host-compiler

PARALLEL_MODE   := NONE

PLATFORMS       := CPU_SISD GPU_CUDA

# for e.g. RTX 30* (Ampere), see table in <code>rules.mk</code> for other options
CUDA_ARCH       := 86

FLOATING_POINT_TYPE := double

USE_EMBEDDED_DEPENDENCIES := ON

[Adrian]

Sorry for the late reply – the issue was that the MRT collision implementation was missing the any_platform macro that is needed to trigger device code generation (and unfortunately nvcc silentlly produces garbage instead of failing). I pushed an update to the release repository. In my tests aorta3d now works with MRT on GPUs.

[sfraniatte]

Awesome ! Thank you !

[Author]

Posts