[sfraniatte]

Hello,

In fact, I converted the showCase/centrifugalPump3d/ example to make my case and it works really well without grid refinement. Then, I tried to make the refinement around the moving part with the sphere3d example. The mouving part is in the refined grid and can not move through refinement regions.

I have solved my problem this morning and it was a mistake in the parameters of the simulation. The motion was physically impossible.

Thanks a lot again for your time to respond me !

Best regards,

Sylvain

[sfraniatte]

Hello,

I succeeded to make it in my case but the FSI part of my case does not work anymore. Is it possible to have FSI and grid refinement in the same case ?

Best regards,

Sylvain

[sfraniatte]

Hello,

The sketch of my refinement domain is very simple. It is basically the spher3d example with the lines 221 and 222 which are commented. The lines are the following :
cuboidDecompositionLevel0.splitFractional(1, 1, {0.1,0.8,0.1});
cuboidDecompositionLevel0.splitFractional(3, 2, {0.1,0.8,0.1});

So, the localized refinement part is on the entire cross-section of the pipe and no longer on 80% of the cross-section.

Thanks to your respond, I understood how to solve my problem. The solution is to replace “sGeometry.rename(0,1);” by the following lines in “prepareGeometryFine” :
IndicatorCuboid3D<T> domainI(Vector<T,3>{2.5,0.5,0.5},
Vector<T,3>{0.0,0.0,0.0});
IndicatorLayer3D<T> domainLayerI(domainI, converter.getPhysDeltaX());

sGeometry.rename(0,2, domainLayerI);
sGeometry.rename(2,1, domainI);

Thanks a lot !

I will check the cell-centered approach by Rohde when I will have time.

Best regards,

Sylvain

[sfraniatte]

I forgot to say thank you again for all this work. It is a great tool !

Best regards,

Sylvain

[sfraniatte]

Interesting. I have donloaded the last release on GitLab to retry if it works and the compilation still does not work with this error :

../../../src/fsi/platform/cpu/integral.h(77): error: no instance of function template “lambda [](olb::meta::id<FIELD>, const olb::FieldD<T, DESCRIPTOR, FIELD> &)->auto” matches the argument list
argument types are: (olb::meta::id<olb::fields::fsi::ELEMENT_TORQUE>, const olb::concepts::placeholder::Parameters2D::value_t)
object type is: lambda [](olb::meta::id<FIELD>, const olb::FieldD<T, DESCRIPTOR, FIELD> &)->auto
((set(meta::id<FIELDS>{}, fields.template get<FIELDS>())), …);
^

I have CUDA 12.9 with an Ampere generation GPU. The others cases work properly and it runs properly with gcc and mpic++.

I did not know for the centrifugal pump case. Thank you !

[sfraniatte]

Thank you for this quick answer ! I noticed that only the taylorCouette3d example works properly on GPU though. The rigidValve2d example does not compile and the deformableParticleInShearFlow3d runs but all the fields remain equal to zero. It is not a big deal though.

Best regards,
Sylvain

[sfraniatte]

Awesome ! Thank you !

[sfraniatte]

I forgot to say that I have changed the floating point type to double for a good voxelization. Here is my config.mk file :

# Example build config for OpenLB using CUDA on single GPU systems
#
# Tested using CUDA 11.4
#
# Usage:
#  - Copy this file to OpenLB root as <code>config.mk</code>
#  - Adjust CUDA_ARCH to match your specifc GPU
#  - Run <code>make clean; make</code>
#  - Switch to example directory, e.g. <code>examples/laminar/cavity3dBenchmark</code>
#  - Run <code>make</code>
#  - Start the simulation using <code>./cavity3d</code>

CXX             := nvcc
CC              := nvcc

CXXFLAGS        := -O3
CXXFLAGS        += -std=c++20 --forward-unknown-to-host-compiler

PARALLEL_MODE   := NONE

PLATFORMS       := CPU_SISD GPU_CUDA

# for e.g. RTX 30* (Ampere), see table in <code>rules.mk</code> for other options
CUDA_ARCH       := 86

FLOATING_POINT_TYPE := double

USE_EMBEDDED_DEPENDENCIES := ON

[sfraniatte]

Thank you for your respond ! Here is the code that does not work but succeed to compile on GPU and works on CPU. It is basically the aorta example with the MRT. To compile OpenLB, I used the gpu_only.mk file and with the ampere generation (nvidia rtx a6000) and with CUDA 12.9. This setup works with my case (without MRT) and the other examples.

/* Lattice Boltzmann sample, written in C++, using the OpenLB
* library
*
* Copyright (C) 2011-2014 Mathias J. Krause
* E-mail contact: info@openlb.net
* The most recent release of OpenLB can be downloaded at
* <http://www.openlb.net/&gt;
*
* This program is free software; you can redistribute it and/or
* modify it under the terms of the GNU General Public License
* as published by the Free Software Foundation; either version 2
* of the License, or (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public
* License along with this program; if not, write to the Free
* Software Foundation, Inc., 51 Franklin Street, Fifth Floor,
* Boston, MA 02110-1301, USA.
*/

/* aorta3d.cpp:
* In this example the fluid flow through a bifurcation is
* simulated. The geometry is obtained from a mesh in stl-format.
* With Bouzidi boundary conditions the curved boundary is
* adequately mapped and initialized fully automatically. As
* dynamics a Smagorinsky turbulent BGK model is used to stabilize
* the simulation for low resolutions. As output the flux at the
* inflow and outflow region is computed. The wall stress can be
* visualized on the stl Mesh with the Mesh.pvd file in paraview.
* The results has been validated by comparison with other results
* obtained with FEM and FVM.
*/

#include <olb.h>

using namespace olb;
using namespace olb::descriptors;
using namespace olb::graphics;

using T = FLOATING_POINT_TYPE;
using DESCRIPTOR = D3Q19<tag::MRT>;
using BulkDynamics = SmagorinskyMRTdynamics<T,DESCRIPTOR>;

//simulation parameters
const int N = 40; // resolution of the model
const int M = 20; // time discretization refinement
const bool bouzidiOn = true; // choice of boundary condition
const T maxPhysT = 2.; // max. simulation time in s, SI unit

// Stores data from stl file in geometry in form of material numbers
void prepareGeometry( UnitConverter<T,DESCRIPTOR> const& converter, IndicatorF3D<T>& indicator,
STLreader<T>& stlReader, SuperGeometry<T,3>& superGeometry )
{

OstreamManager clout( std::cout,”prepareGeometry” );
clout << “Prepare Geometry …” << std::endl;

superGeometry.rename( 0,2,indicator );
superGeometry.rename( 2,1,stlReader );

superGeometry.clean();

// Set material number for inflow
IndicatorCircle3D<T> inflow( 0.218125,0.249987,0.0234818, 0., 1.,0., 0.0112342 );
IndicatorCylinder3D<T> layerInflow( inflow, 2.*converter.getPhysDeltaX() );
superGeometry.rename( 2,3,1,layerInflow );

// Set material number for outflow0
//IndicatorCircle3D<T> outflow0(0.2053696,0.0900099,0.0346537, 2.5522,5.0294,-1.5237, 0.0054686 );
IndicatorCircle3D<T> outflow0( 0.2053696,0.0900099,0.0346537, 0.,-1.,0., 0.0054686 );
IndicatorCylinder3D<T> layerOutflow0( outflow0, 2.*converter.getPhysDeltaX() );
superGeometry.rename( 2,4,1,layerOutflow0 );

// Set material number for outflow1
//IndicatorCircle3D<T> outflow1(0.2388403,0.0900099,0.0343228, -1.5129,5.1039,-2.8431, 0.0058006 );
IndicatorCircle3D<T> outflow1( 0.2388403,0.0900099,0.0343228, 0.,-1.,0., 0.0058006 );
IndicatorCylinder3D<T> layerOutflow1( outflow1, 2.*converter.getPhysDeltaX() );
superGeometry.rename( 2,5,1,layerOutflow1 );

// Removes all not needed boundary voxels outside the surface
superGeometry.clean();
// Removes all not needed boundary voxels inside the surface
superGeometry.innerClean( 3 );
superGeometry.checkForErrors();

superGeometry.print();
clout << “Prepare Geometry … OK” << std::endl;
}

// Set up the geometry of the simulation
void prepareLattice( SuperLattice<T, DESCRIPTOR>& lattice,
UnitConverter<T,DESCRIPTOR> const& converter,
STLreader<T>& stlReader, SuperGeometry<T,3>& superGeometry )
{

OstreamManager clout( std::cout,”prepareLattice” );
clout << “Prepare Lattice …” << std::endl;

const T omega = converter.getLatticeRelaxationFrequency();

// material=1 –> bulk dynamics
lattice.defineDynamics<BulkDynamics>(superGeometry, 1);

if ( bouzidiOn ) {
// material=2 –> no dynamics + bouzidi zero velocity
setBouzidiBoundary<T,DESCRIPTOR>(lattice, superGeometry, 2, stlReader);
// material=3 –> no dynamics + bouzidi velocity (inflow)
setBouzidiBoundary<T,DESCRIPTOR,BouzidiVelocityPostProcessor>(lattice, superGeometry, 3, stlReader);
}
else {
// material=2 –> bounceBack dynamics
boundary::set<boundary::BounceBack>(lattice, superGeometry, 2);
// material=3 –> bulk dynamics + velocity (inflow)
lattice.defineDynamics<BulkDynamics>(superGeometry, 3);
boundary::set<boundary::InterpolatedVelocity>(lattice, superGeometry, 3);
}

// material=4,5 –> bulk dynamics + pressure (outflow)
lattice.defineDynamics<BulkDynamics>(superGeometry.getMaterialIndicator({4, 5}));
boundary::set<boundary::InterpolatedPressure>(lattice, superGeometry, 4);
boundary::set<boundary::InterpolatedPressure>(lattice, superGeometry, 5);

// Initial conditions
AnalyticalConst3D<T,T> rhoF( 1 );
std::vector<T> velocity( 3,T() );
AnalyticalConst3D<T,T> uF( velocity );

// Initialize all values of distribution functions to their local equilibrium
lattice.defineRhoU( superGeometry.getMaterialIndicator({1, 3, 4, 5}),rhoF,uF );
lattice.iniEquilibrium( superGeometry.getMaterialIndicator({1, 3, 4, 5}),rhoF,uF );

lattice.setParameter<descriptors::OMEGA>(omega);
lattice.setParameter<collision::LES::SMAGORINSKY>(T(0.1));
// Lattice initialize
lattice.initialize();

clout << “Prepare Lattice … OK” << std::endl;
}

// Generates a slowly increasing sinuidal inflow
void setBoundaryValues( SuperLattice<T, DESCRIPTOR>& sLattice,
UnitConverter<T,DESCRIPTOR> const& converter, std::size_t iT,
SuperGeometry<T,3>& superGeometry )
{

// No of time steps for smooth start-up
std::size_t iTperiod = converter.getLatticeTime( 0.5 );
std::size_t iTupdate = 50;

if ( iT%iTupdate == 0 ) {
// Smooth start curve, sinus
SinusStartScale<T,std::size_t> nSinusStartScale( iTperiod,converter.getCharLatticeVelocity() );

// Creates and sets the Poiseuille inflow profile using functors
std::size_t iTvec[1]= {iT};
T maxVelocity[1]= {T()};
nSinusStartScale( maxVelocity,iTvec );
CirclePoiseuille3D<T> velocity( superGeometry,3,maxVelocity[0], T() );

if ( bouzidiOn ) {
setBouzidiVelocity(sLattice, superGeometry, 3, velocity);
sLattice.setProcessingContext<Array<descriptors::BOUZIDI_VELOCITY>>(
ProcessingContext::Simulation);
}
else {
sLattice.defineU(superGeometry, 3, velocity);
sLattice.setProcessingContext<Array<momenta::FixedVelocityMomentumGeneric::VELOCITY>>(
ProcessingContext::Simulation);
}
}
}

// Computes flux at inflow and outflow
void getResults(SuperLattice<T, DESCRIPTOR>& sLattice,
const UnitConverter<T,DESCRIPTOR>& converter,
std::size_t iT,
SuperGeometry<T,3>& superGeometry,
util::Timer<T>& timer,
STLreader<T>& stlReader,
VTUsurfaceWriter<T>& vtuWriter)
{
OstreamManager clout( std::cout,”getResults” );

const std::size_t vtkIter = converter.getLatticeTime( .1 );
const std::size_t statIter = converter.getLatticeTime( .1 );

if ( iT==0 ) {
SuperVTMwriter3D<T> vtmWriter(“aorta3d”);
// Writes the geometry, cuboid no. and rank no. as vti file for visualization

SuperLatticeCuboid3D<T, DESCRIPTOR> cuboid( sLattice );
SuperLatticeRank3D<T, DESCRIPTOR> rank( sLattice );
vtmWriter.write( cuboid );
vtmWriter.write( rank );

vtmWriter.createMasterFile();
vtuWriter.createMasterFile();
}

// Writes the vtk files
if ( iT%vtkIter==0 ) {
sLattice.setProcessingContext(ProcessingContext::Evaluation);
sLattice.scheduleBackgroundOutputVTK([&,iT](auto task) {
SuperVTMwriter3D<T> vtmWriter(“aorta3d”);
SuperLatticePhysVelocity3D velocity(sLattice, converter);
SuperLatticePhysPressure3D pressure(sLattice, converter);
vtmWriter.addFunctor(velocity);
vtmWriter.addFunctor(pressure);
task(vtmWriter, iT);
});

// Write interpolated wall shear stress on the STL surface
{
SuperLatticeDensity3D densityF(sLattice);
AnalyticalFfromSuperF3D smoothDensityF(densityF);

SuperLatticeStress3D stressF(sLattice);
AnalyticalFfromSuperF3D smoothStressF(stressF);

PhysWallShearStressOnSurface3D<T,DESCRIPTOR> interpolatedWssF(converter, smoothDensityF, smoothStressF, stlReader);
interpolatedWssF.getName() = “interpolatedWss”;

vtuWriter.addFunctor(interpolatedWssF);

vtuWriter.write(iT);
}
}

// Writes output on the console
if ( iT%statIter==0 ) {
// Timer console output
timer.update( iT );
timer.printStep();

// Lattice statistics console output
sLattice.getStatistics().print( iT,converter.getPhysTime( iT ) );

// Flux at the inflow and outflow region
std::vector<int> materials = { 1, 3, 4, 5 };

IndicatorCircle3D<T> inflow( 0.218125,0.249987-2.*converter.getPhysDeltaX(),0.0234818, 0., -1.,0., 0.0112342+2*converter.getPhysDeltaX() );
SuperPlaneIntegralFluxVelocity3D<T> vFluxInflow( sLattice, converter, superGeometry, inflow, materials, BlockDataReductionMode::Discrete );
vFluxInflow.print( “inflow”,”ml/s” );
SuperPlaneIntegralFluxPressure3D<T> pFluxInflow( sLattice, converter, superGeometry, inflow, materials, BlockDataReductionMode::Discrete );
pFluxInflow.print( “inflow”,”N”,”mmHg” );

IndicatorCircle3D<T> outflow0( 0.2053696,0.0900099+2.*converter.getPhysDeltaX(),0.0346537, 0.,1.,0., 0.0054686+2*converter.getPhysDeltaX() );
SuperPlaneIntegralFluxVelocity3D<T> vFluxOutflow0( sLattice, converter, superGeometry, outflow0, materials, BlockDataReductionMode::Discrete );
vFluxOutflow0.print( “outflow0″,”ml/s” );
SuperPlaneIntegralFluxPressure3D<T> pFluxOutflow0( sLattice, converter, superGeometry, outflow0, materials, BlockDataReductionMode::Discrete );
pFluxOutflow0.print( “outflow0″,”N”,”mmHg” );

IndicatorCircle3D<T> outflow1( 0.2388403,0.0900099+2.*converter.getPhysDeltaX(),0.0343228, 0.,1.,0., 0.0058006+2*converter.getPhysDeltaX() );
SuperPlaneIntegralFluxVelocity3D<T> vFluxOutflow1( sLattice, converter, superGeometry, outflow1, materials, BlockDataReductionMode::Discrete );
vFluxOutflow1.print( “outflow1″,”ml/s” );
SuperPlaneIntegralFluxPressure3D<T> pFluxOutflow1( sLattice, converter, superGeometry, outflow1, materials, BlockDataReductionMode::Discrete );
pFluxOutflow1.print( “outflow1″,”N”,”mmHg” );

if ( bouzidiOn ) {
SuperLatticeYplus3D<T, DESCRIPTOR> yPlus( sLattice, converter, superGeometry, stlReader, 3 );
SuperMax3D<T> yPlusMaxF( yPlus, superGeometry, 1 );
int input[4]= {};
T yPlusMax[1];
yPlusMaxF( yPlusMax,input );
clout << “yPlusMax=” << yPlusMax[0] << std::endl;
}
}

if ( sLattice.getStatistics().getMaxU() > 0.3 ) {
clout << “PROBLEM uMax=” << sLattice.getStatistics().getMaxU() << std::endl;
std::exit(0);
}
}

int main( int argc, char* argv[] )
{
// === 1st Step: Initialization ===
initialize( &argc, &argv );
singleton::directories().setOutputDir( “./tmp/” );
OstreamManager clout( std::cout,”main” );
// display messages from every single mpi process
//clout.setMultiOutput(true);

const UnitConverter<T,DESCRIPTOR> converter(
(T) 0.02246/N, // physDeltaX: spacing between two lattice cells in __m__
(T) 0.02246/(M*N), // physDeltaT: time step in __s__
(T) 0.02246, // charPhysLength: reference length of simulation geometry
(T) 0.45, // charPhysVelocity: maximal/highest expected velocity during simulation in __m / s__
(T) 0.003/1055., // physViscosity: physical kinematic viscosity in __m^2 / s__
(T) 1055 // physDensity: physical density in __kg / m^3__
);
// Prints the converter log as console output
converter.print();
// Writes the converter log in a file
converter.write(“aorta3d”);

// === 2nd Step: Prepare Geometry ===

// Instantiation of the STLreader class
// file name, voxel size in meter, stl unit in meter, outer voxel no., inner voxel no.
STLreader<T> stlReader( “aorta3d.stl”, converter.getPhysDeltaX(), 0.001, olb::RayMode::FastRayZ, true );
IndicatorLayer3D<T> extendedDomain( stlReader, converter.getPhysDeltaX() );

// Instantiation of a cuboidDecomposition with weights
#ifdef PARALLEL_MODE_MPI
const int noOfCuboids = util::min(16*N, 8*singleton::mpi().getSize());
#else
const int noOfCuboids = 2;
#endif
CuboidDecomposition3D<T> cuboidDecomposition( extendedDomain, converter.getPhysDeltaX(), noOfCuboids, “volume” );
// Instantiation of a loadBalancer
HeuristicLoadBalancer<T> loadBalancer( cuboidDecomposition );

// Instantiation of a superGeometry
SuperGeometry<T,3> superGeometry( cuboidDecomposition, loadBalancer );

prepareGeometry( converter, extendedDomain, stlReader, superGeometry );

// === 3rd Step: Prepare Lattice ===
SuperLattice<T, DESCRIPTOR> sLattice( superGeometry );

util::Timer<T> timer1( converter.getLatticeTime( maxPhysT ), superGeometry.getStatistics().getNvoxel() );
timer1.start();

prepareLattice( sLattice, converter, stlReader, superGeometry );

VTUsurfaceWriter<T> vtuWriter(“surface”, cuboidDecomposition, loadBalancer);
vtuWriter.addSTL( stlReader );

timer1.stop();
timer1.printSummary();

// === 4th Step: Main Loop with Timer ===
clout << “starting simulation…” << std::endl;
util::Timer<T> timer( converter.getLatticeTime( maxPhysT ), superGeometry.getStatistics().getNvoxel() );
timer.start();

for ( std::size_t iT = 0; iT <= converter.getLatticeTime( maxPhysT ); iT++ ) {
// === 5th Step: Definition of Initial and Boundary Conditions ===
setBoundaryValues( sLattice, converter, iT, superGeometry );

// === 6th Step: Collide and Stream Execution ===
sLattice.collideAndStream();

// === 7th Step: Computation and Output of the Results ===
getResults( sLattice, converter, iT, superGeometry, timer, stlReader, vtuWriter );
}

timer.stop();
timer.printSummary();
}

Here is the output with GPU :

utilisateur@LABO-ACC:~/release-1.8.1/examples/turbulence/aorta3d$ make clean; make; ./aorta3d
rm -f tmp/*.* tmp/vtkData/*.* tmp/vtkData/data/*.* tmp/imageData/*.* tmp/imageData/data/*.* tmp/gnuplotData/*.* tmp/gnuplotData/data/*.*
rm -f aorta3d.o aorta3d.d aorta3d.wasm aorta3d.js aorta3d
make -C ../../../external
make[1] : on entre dans le répertoire « /home/utilisateur/release-1.8.1/external »
make -C zlib
make[2] : on entre dans le répertoire « /home/utilisateur/release-1.8.1/external/zlib »
make[2]: rien à faire pour « all ».
make[2] : on quitte le répertoire « /home/utilisateur/release-1.8.1/external/zlib »
cp zlib/build/libz.a lib/
make -C tinyxml2
make[2] : on entre dans le répertoire « /home/utilisateur/release-1.8.1/external/tinyxml2 »
make[2]: rien à faire pour « all ».
make[2] : on quitte le répertoire « /home/utilisateur/release-1.8.1/external/tinyxml2 »
cp tinyxml2/build/libtinyxml2.a lib/
make[1] : on quitte le répertoire « /home/utilisateur/release-1.8.1/external »
make -C ../../.. core
make[1] : on entre dans le répertoire « /home/utilisateur/release-1.8.1 »
make[1]: rien à faire pour « core ».
make[1] : on quitte le répertoire « /home/utilisateur/release-1.8.1 »
nvcc -O3 -std=c++20 –forward-unknown-to-host-compiler -pthread –forward-unknown-to-host-compiler -x cu -O3 -std=c++20 –generate-code=arch=compute_86,code=[compute_86,sm_86] –extended-lambda –expt-relaxed-constexpr -rdc=true -Xcudafe “–diag_suppress=implicit_return_from_non_void_function –display_error_number –diag_suppress=20014 –diag_suppress=20011” -DPLATFORM_CPU_SISD -DPLATFORM_GPU_CUDA -DDEFAULT_FLOATING_POINT_TYPE=double -I../../../src -I../../../external/zlib -I../../../external/tinyxml2 -c -o aorta3d.o aorta3d.cpp
../../../src/utilities/typeIndexedContainers.h(189): warning #20012-D: __device__ annotation is ignored on a function(“TypeIndexedTuple”) that is explicitly defaulted on its first declaration
TypeIndexedTuple() __attribute__((device)) __attribute__((host)) = default;
^

Remark: The warnings can be suppressed with “-diag-suppress <warning-number>”

../../../src/utilities/typeIndexedContainers.h(189): warning #20012-D: __host__ annotation is ignored on a function(“TypeIndexedTuple”) that is explicitly defaulted on its first declaration
TypeIndexedTuple() __attribute__((device)) __attribute__((host)) = default;
^

../../../src/utilities/typeIndexedContainers.h(190): warning #20012-D: __device__ annotation is ignored on a function(“TypeIndexedTuple”) that is explicitly defaulted on its first declaration
TypeIndexedTuple(TypeIndexedTuple&) __attribute__((device)) __attribute__((host)) = default;
^

../../../src/utilities/typeIndexedContainers.h(190): warning #20012-D: __host__ annotation is ignored on a function(“TypeIndexedTuple”) that is explicitly defaulted on its first declaration
TypeIndexedTuple(TypeIndexedTuple&) __attribute__((device)) __attribute__((host)) = default;
^

../../../src/utilities/typeIndexedContainers.h(191): warning #20012-D: __device__ annotation is ignored on a function(“TypeIndexedTuple”) that is explicitly defaulted on its first declaration
TypeIndexedTuple(TypeIndexedTuple&& rhs) __attribute__((device)) __attribute__((host)) = default;
^

../../../src/utilities/typeIndexedContainers.h(191): warning #20012-D: __host__ annotation is ignored on a function(“TypeIndexedTuple”) that is explicitly defaulted on its first declaration
TypeIndexedTuple(TypeIndexedTuple&& rhs) __attribute__((device)) __attribute__((host)) = default;
^

../../../src/io/stlReader.h(219): warning #611-D: overloaded virtual function “olb::IndicatorF3D<S>::distance [with S=T]” is only partially overridden in class “olb::STLreader<T>”
class STLreader : public IndicatorF3D<T> {
^
detected during instantiation of class “olb::STLreader<T> [with T=T]” at line 65 of aorta3d.cpp

../../../src/utilities/typeIndexedContainers.h(189): warning #20012-D: __device__ annotation is ignored on a function(“TypeIndexedTuple”) that is explicitly defaulted on its first declaration
TypeIndexedTuple() __attribute__((device)) __attribute__((host)) = default;
^

Remark: The warnings can be suppressed with “-diag-suppress <warning-number>”

../../../src/utilities/typeIndexedContainers.h(189): warning #20012-D: __host__ annotation is ignored on a function(“TypeIndexedTuple”) that is explicitly defaulted on its first declaration
TypeIndexedTuple() __attribute__((device)) __attribute__((host)) = default;
^

../../../src/utilities/typeIndexedContainers.h(190): warning #20012-D: __device__ annotation is ignored on a function(“TypeIndexedTuple”) that is explicitly defaulted on its first declaration
TypeIndexedTuple(TypeIndexedTuple&) __attribute__((device)) __attribute__((host)) = default;
^

../../../src/utilities/typeIndexedContainers.h(190): warning #20012-D: __host__ annotation is ignored on a function(“TypeIndexedTuple”) that is explicitly defaulted on its first declaration
TypeIndexedTuple(TypeIndexedTuple&) __attribute__((device)) __attribute__((host)) = default;
^

../../../src/utilities/typeIndexedContainers.h(191): warning #20012-D: __device__ annotation is ignored on a function(“TypeIndexedTuple”) that is explicitly defaulted on its first declaration
TypeIndexedTuple(TypeIndexedTuple&& rhs) __attribute__((device)) __attribute__((host)) = default;
^

../../../src/utilities/typeIndexedContainers.h(191): warning #20012-D: __host__ annotation is ignored on a function(“TypeIndexedTuple”) that is explicitly defaulted on its first declaration
TypeIndexedTuple(TypeIndexedTuple&& rhs) __attribute__((device)) __attribute__((host)) = default;
^

../../../src/io/stlReader.h(219): warning #611-D: overloaded virtual function “olb::IndicatorF3D<S>::distance [with S=T]” is only partially overridden in class “olb::STLreader<T>”
class STLreader : public IndicatorF3D<T> {
^
detected during instantiation of class “olb::STLreader<T> [with T=T]” at line 65 of aorta3d.cpp

../../../src/functors/analytical/indicator/indicatorF2D.h(121): warning #611-D: overloaded virtual function “olb::IndicatorF2D<S>::operator() [with S=T]” is only partially overridden in class “olb::IndicatorCircle2D<T>”
class IndicatorCircle2D : public IndicatorF2D<S> {
^
detected during:
instantiation of class “olb::IndicatorCircle2D<S> [with S=T]” at line 248 of ../../../src/functors/lattice/integral/superPlaneIntegralF3D.hh
instantiation of “olb::SuperPlaneIntegralF3D<T>::SuperPlaneIntegralF3D(olb::FunctorPtr<olb::SuperF3D<T, T>> &&, olb::SuperGeometry<T, 3U> &, const olb::IndicatorCircle3D<T> &, std::vector<int, std::allocator<int>>, olb::BlockDataReductionMode) [with T=T]” at line 221 of ../../../src/functors/lattice/integral/superPlaneIntegralFluxF3D.hh
instantiation of “olb::SuperPlaneIntegralFluxF3D<T, F>::SuperPlaneIntegralFluxF3D(olb::SuperLattice<T, DESCRIPTOR> &, const olb::UnitConverter<T, DESCRIPTOR> &, olb::SuperGeometry<T, 3U> &, olb::IndicatorCircle3D<T> &, std::vector<int, std::allocator<int>>, olb::BlockDataReductionMode) [with T=T, F=olb::SuperLatticePhysVelocity3D, DESCRIPTOR=DESCRIPTOR]” at line 164 of ../../../src/functors/lattice/integral/superPlaneIntegralFluxF3D.h

nvcc aorta3d.o -o aorta3d -lolbcore -L../../../external/lib -lpthread -lz -ltinyxml2 -lcuda -lcudadevrt -lcudart -L../../../build/lib
nvcc warning : Support for offline compilation for architectures prior to ‘<compute/sm/lto>_75’ will be removed in a future release (Use -Wno-deprecated-gpu-targets to suppress warning).
nvlink warning : SM Arch (‘sm_52’) not found in ‘aorta3d.o’
[ThreadPool] Sucessfully initialized, numThreads=1
[UnitConverter] —————– UnitConverter information —————–
[UnitConverter] — Parameters:
[UnitConverter] Resolution: N= 40
[UnitConverter] Lattice velocity: latticeU= 0.0225
[UnitConverter] Lattice relaxation frequency: omega= 1.99697
[UnitConverter] Lattice relaxation time: tau= 0.50076
[UnitConverter] Characteristical length(m): charL= 0.02246
[UnitConverter] Characteristical speed(m/s): charU= 0.45
[UnitConverter] Phys. kinematic viscosity(m^2/s): charNu= 2.8436e-06
[UnitConverter] Phys. density(kg/m^d): charRho= 1055
[UnitConverter] Characteristical pressure(N/m^2): charPressure= 0
[UnitConverter] Mach number: machNumber= 0.0389711
[UnitConverter] Reynolds number: reynoldsNumber= 3554.3
[UnitConverter] Knudsen number: knudsenNumber= 1.09645e-05
[UnitConverter] Characteristical CFL number: charCFLnumber= 0.0225
[UnitConverter]
[UnitConverter] — Conversion factors:
[UnitConverter] Voxel length(m): physDeltaX= 0.0005615
[UnitConverter] Time step(s): physDeltaT= 2.8075e-05
[UnitConverter] Velocity factor(m/s): physVelocity= 20
[UnitConverter] Density factor(kg/m^3): physDensity= 1055
[UnitConverter] Mass factor(kg): physMass= 1.86768e-07
[UnitConverter] Viscosity factor(m^2/s): physViscosity= 0.01123
[UnitConverter] Force factor(N): physForce= 0.133049
[UnitConverter] Pressure factor(N/m^2): physPressure= 422000
[UnitConverter] ————————————————————-
[UnitConverter] WARNING:
[UnitConverter] Potentially UNSTABLE combination of relaxation time (tau=0.50076)
[UnitConverter] and characteristical CFL number (lattice velocity) charCFLnumber=0.0225!
[UnitConverter] Potentially maximum characteristical CFL number (maxCharCFLnumber=0.00607715)
[UnitConverter] Actual characteristical CFL number (charCFLnumber=0.0225) > 0.00607715
[UnitConverter] Please reduce the the cell size or the time step size!
[UnitConverter] We recommend to use the cell size of 0.000151659 m and the time step size of 7.58294e-06 s.
[UnitConverter] ————————————————————-
[STLreader] Voxelizing …
[STLmesh] nTriangles=2654; maxDist2=0.000610779
[STLmesh] minPhysR(StlMesh)=(0.199901,0.0900099,0.0117236); maxPhysR(StlMesh)=(0.243584,0.249987,0.0398131)
[Octree] radius=0.143744; center=(0.221602,0.169858,0.025628)
[STLreader] voxelSize=0.0005615; stlSize=0.001
[STLreader] minPhysR(VoxelMesh)=(0.199984,0.0904058,0.0118712); maxPhysR(VoxelMesh)=(0.24322,0.249872,0.0393847)
[STLreader] Voxelizing … OK
[prepareGeometry] Prepare Geometry …
[SuperGeometry3D] cleaned 0 outer boundary voxel(s)
[SuperGeometry3D] cleaned 0 outer boundary voxel(s)
[SuperGeometry3D] cleaned 0 inner boundary voxel(s) of Type 3
[SuperGeometryStatistics3D] updated
[SuperGeometry3D] the model is correct!
[CuboidDecomposition] —Cuboid Structure Statistics—
[CuboidDecomposition] Number of Cuboids: 2
[CuboidDecomposition] Delta : 0.0005615
[CuboidDecomposition] Ratio (min): 0.293706
[CuboidDecomposition] (max): 3.40476
[CuboidDecomposition] Nodes (min): 252252
[CuboidDecomposition] (max): 557424
[CuboidDecomposition] Weight (min): 96042
[CuboidDecomposition] (max): 117790
[CuboidDecomposition] ——————————–
[SuperGeometryStatistics3D] materialNumber=0; count=595844; minPhysR=(0.199984,0.0898443,0.0113097); maxPhysR=(0.243781,0.250433,0.0399462)
[SuperGeometryStatistics3D] materialNumber=1; count=171218; minPhysR=(0.200546,0.0904058,0.0118712); maxPhysR=(0.24322,0.249872,0.0393847)
[SuperGeometryStatistics3D] materialNumber=2; count=41071; minPhysR=(0.199984,0.0898443,0.0113097); maxPhysR=(0.243781,0.250433,0.0399462)
[SuperGeometryStatistics3D] materialNumber=3; count=1059; minPhysR=(0.208407,0.250433,0.0124327); maxPhysR=(0.228059,0.250433,0.0332082)
[SuperGeometryStatistics3D] materialNumber=4; count=245; minPhysR=(0.200546,0.0898443,0.0298392); maxPhysR=(0.210653,0.0898443,0.0388232)
[SuperGeometryStatistics3D] materialNumber=5; count=239; minPhysR=(0.234236,0.0898443,0.0287162); maxPhysR=(0.24322,0.0898443,0.0388232)
[SuperGeometryStatistics3D] countTotal[1e6]=0.809676
[prepareGeometry] Prepare Geometry … OK
[prepareLattice] Prepare Lattice …
[prepareLattice] Prepare Lattice … OK
[Timer]
[Timer] —————-Summary:Timer—————-
[Timer] measured time (rt) : 0.420s
[Timer] measured time (cpu): 0.419s
[Timer] ———————————————
[main] starting simulation…
[Timer] step=0; percent=0; passedTime=0.232; remTime=16527; MLUPs=0
[LatticeStatistics] step=0; physT=0; maxCFL=1.49167e-154; avgEnergy=0; avgRho=1
[SuperPlaneIntegralFluxVelocity3D] regionName=inflow; regionSize[m^2]=0.000310868; volumetricFlowRate[ml/s]=0; meanVelocity[m/s]=0
[SuperPlaneIntegralFluxPressure3D] regionName=inflow; regionSize[m^2]=0.000310868; force[N]=0; meanPressure[mmHg]=0
[SuperPlaneIntegralFluxVelocity3D] regionName=outflow0; regionSize[m^2]=6.40023e-05; volumetricFlowRate[ml/s]=0; meanVelocity[m/s]=0
[SuperPlaneIntegralFluxPressure3D] regionName=outflow0; regionSize[m^2]=6.40023e-05; force[N]=0; meanPressure[mmHg]=0
[SuperPlaneIntegralFluxVelocity3D] regionName=outflow1; regionSize[m^2]=6.21106e-05; volumetricFlowRate[ml/s]=0; meanVelocity[m/s]=0
[SuperPlaneIntegralFluxPressure3D] regionName=outflow1; regionSize[m^2]=6.21106e-05; force[N]=0; meanPressure[mmHg]=0
[getResults] yPlusMax=2.22507e-308
[Timer] step=3562; percent=5.00014; passedTime=8.042; remTime=152.793; MLUPs=97.5124
[LatticeStatistics] step=3562; physT=0.100003; maxCFL=1.49167e-154; avgEnergy=0; avgRho=1
[SuperPlaneIntegralFluxVelocity3D] regionName=inflow; regionSize[m^2]=0.000310868; volumetricFlowRate[ml/s]=5.35481; meanVelocity[m/s]=0.0172253
[SuperPlaneIntegralFluxPressure3D] regionName=inflow; regionSize[m^2]=0.000310868; force[N]=3.21631; meanPressure[mmHg]=77.6032
[SuperPlaneIntegralFluxVelocity3D] regionName=outflow0; regionSize[m^2]=6.40023e-05; volumetricFlowRate[ml/s]=0; meanVelocity[m/s]=0
[SuperPlaneIntegralFluxPressure3D] regionName=outflow0; regionSize[m^2]=6.40023e-05; force[N]=0; meanPressure[mmHg]=0
[SuperPlaneIntegralFluxVelocity3D] regionName=outflow1; regionSize[m^2]=6.21106e-05; volumetricFlowRate[ml/s]=0; meanVelocity[m/s]=0
[SuperPlaneIntegralFluxPressure3D] regionName=outflow1; regionSize[m^2]=6.21106e-05; force[N]=0; meanPressure[mmHg]=0
[getResults] yPlusMax=7.14391
[Timer] step=7124; percent=10.0003; passedTime=15.87; remTime=142.826; MLUPs=97.3007
[LatticeStatistics] step=7124; physT=0.200006; maxCFL=1.49167e-154; avgEnergy=0; avgRho=1
[SuperPlaneIntegralFluxVelocity3D] regionName=inflow; regionSize[m^2]=0.000310868; volumetricFlowRate[ml/s]=12.8358; meanVelocity[m/s]=0.04129
[SuperPlaneIntegralFluxPressure3D] regionName=inflow; regionSize[m^2]=0.000310868; force[N]=24.4992; meanPressure[mmHg]=591.118
[SuperPlaneIntegralFluxVelocity3D] regionName=outflow0; regionSize[m^2]=6.40023e-05; volumetricFlowRate[ml/s]=0; meanVelocity[m/s]=0
[SuperPlaneIntegralFluxPressure3D] regionName=outflow0; regionSize[m^2]=6.40023e-05; force[N]=0; meanPressure[mmHg]=0
[SuperPlaneIntegralFluxVelocity3D] regionName=outflow1; regionSize[m^2]=6.21106e-05; volumetricFlowRate[ml/s]=0; meanVelocity[m/s]=0
[SuperPlaneIntegralFluxPressure3D] regionName=outflow1; regionSize[m^2]=6.21106e-05; force[N]=0; meanPressure[mmHg]=0
[getResults] yPlusMax=23.7621
[Timer] step=10686; percent=15.0004; passedTime=23.708; remTime=134.341; MLUPs=97.1765
[LatticeStatistics] step=10686; physT=0.300009; maxCFL=1.49167e-154; avgEnergy=0; avgRho=1
[SuperPlaneIntegralFluxVelocity3D] regionName=inflow; regionSize[m^2]=0.000310868; volumetricFlowRate[ml/s]=13.5503; meanVelocity[m/s]=0.0435886
[SuperPlaneIntegralFluxPressure3D] regionName=inflow; regionSize[m^2]=0.000310868; force[N]=75.1471; meanPressure[mmHg]=1813.15
[SuperPlaneIntegralFluxVelocity3D] regionName=outflow0; regionSize[m^2]=6.40023e-05; volumetricFlowRate[ml/s]=0; meanVelocity[m/s]=0
[SuperPlaneIntegralFluxPressure3D] regionName=outflow0; regionSize[m^2]=6.40023e-05; force[N]=0; meanPressure[mmHg]=0
[SuperPlaneIntegralFluxVelocity3D] regionName=outflow1; regionSize[m^2]=6.21106e-05; volumetricFlowRate[ml/s]=0; meanVelocity[m/s]=0
[SuperPlaneIntegralFluxPressure3D] regionName=outflow1; regionSize[m^2]=6.21106e-05; force[N]=0; meanPressure[mmHg]=0
[getResults] yPlusMax=126.305
[Timer] step=14248; percent=20.0006; passedTime=31.56; remTime=126.236; MLUPs=96.9909
[LatticeStatistics] step=14248; physT=0.400013; maxCFL=1.49167e-154; avgEnergy=0; avgRho=1
[SuperPlaneIntegralFluxVelocity3D] regionName=inflow; regionSize[m^2]=0.000310868; volumetricFlowRate[ml/s]=11.5981; meanVelocity[m/s]=0.0373087
[SuperPlaneIntegralFluxPressure3D] regionName=inflow; regionSize[m^2]=0.000310868; force[N]=155.24; meanPressure[mmHg]=3745.63
[SuperPlaneIntegralFluxVelocity3D] regionName=outflow0; regionSize[m^2]=6.40023e-05; volumetricFlowRate[ml/s]=0; meanVelocity[m/s]=0
[SuperPlaneIntegralFluxPressure3D] regionName=outflow0; regionSize[m^2]=6.40023e-05; force[N]=0; meanPressure[mmHg]=0
[SuperPlaneIntegralFluxVelocity3D] regionName=outflow1; regionSize[m^2]=6.21106e-05; volumetricFlowRate[ml/s]=0; meanVelocity[m/s]=0
[SuperPlaneIntegralFluxPressure3D] regionName=outflow1; regionSize[m^2]=6.21106e-05; force[N]=0; meanPressure[mmHg]=0
[getResults] yPlusMax=49.5797
[Timer] step=17810; percent=25.0007; passedTime=39.439; remTime=118.313; MLUPs=96.6708
[LatticeStatistics] step=17810; physT=0.500016; maxCFL=1.49167e-154; avgEnergy=0; avgRho=1
[SuperPlaneIntegralFluxVelocity3D] regionName=inflow; regionSize[m^2]=0.000310868; volumetricFlowRate[ml/s]=8.12449; meanVelocity[m/s]=0.0261348
[SuperPlaneIntegralFluxPressure3D] regionName=inflow; regionSize[m^2]=0.000310868; force[N]=253.61; meanPressure[mmHg]=6119.11
[SuperPlaneIntegralFluxVelocity3D] regionName=outflow0; regionSize[m^2]=6.40023e-05; volumetricFlowRate[ml/s]=0; meanVelocity[m/s]=0
[SuperPlaneIntegralFluxPressure3D] regionName=outflow0; regionSize[m^2]=6.40023e-05; force[N]=0; meanPressure[mmHg]=0
[SuperPlaneIntegralFluxVelocity3D] regionName=outflow1; regionSize[m^2]=6.21106e-05; volumetricFlowRate[ml/s]=0; meanVelocity[m/s]=0
[SuperPlaneIntegralFluxPressure3D] regionName=outflow1; regionSize[m^2]=6.21106e-05; force[N]=0; meanPressure[mmHg]=0
[getResults] yPlusMax=48.3407
[Timer] step=21372; percent=30.0008; passedTime=47.333; remTime=110.439; MLUPs=96.4749
[LatticeStatistics] step=21372; physT=0.600019; maxCFL=1.49167e-154; avgEnergy=0; avgRho=1
[SuperPlaneIntegralFluxVelocity3D] regionName=inflow; regionSize[m^2]=0.000310868; volumetricFlowRate[ml/s]=5.51686; meanVelocity[m/s]=0.0177466
[SuperPlaneIntegralFluxPressure3D] regionName=inflow; regionSize[m^2]=0.000310868; force[N]=352.107; meanPressure[mmHg]=8495.64
[SuperPlaneIntegralFluxVelocity3D] regionName=outflow0; regionSize[m^2]=6.40023e-05; volumetricFlowRate[ml/s]=0; meanVelocity[m/s]=0
[SuperPlaneIntegralFluxPressure3D] regionName=outflow0; regionSize[m^2]=6.40023e-05; force[N]=0; meanPressure[mmHg]=0
[SuperPlaneIntegralFluxVelocity3D] regionName=outflow1; regionSize[m^2]=6.21106e-05; volumetricFlowRate[ml/s]=0; meanVelocity[m/s]=0
[SuperPlaneIntegralFluxPressure3D] regionName=outflow1; regionSize[m^2]=6.21106e-05; force[N]=0; meanPressure[mmHg]=0
[getResults] yPlusMax=70.7212
[Timer] step=24934; percent=35.001; passedTime=55.232; remTime=102.569; MLUPs=96.4261
[LatticeStatistics] step=24934; physT=0.700022; maxCFL=1.49167e-154; avgEnergy=0; avgRho=1
[SuperPlaneIntegralFluxVelocity3D] regionName=inflow; regionSize[m^2]=0.000310868; volumetricFlowRate[ml/s]=3.34209; meanVelocity[m/s]=0.0107508
[SuperPlaneIntegralFluxPressure3D] regionName=inflow; regionSize[m^2]=0.000310868; force[N]=432.532; meanPressure[mmHg]=10436.1
[SuperPlaneIntegralFluxVelocity3D] regionName=outflow0; regionSize[m^2]=6.40023e-05; volumetricFlowRate[ml/s]=0; meanVelocity[m/s]=0
[SuperPlaneIntegralFluxPressure3D] regionName=outflow0; regionSize[m^2]=6.40023e-05; force[N]=0; meanPressure[mmHg]=0
[SuperPlaneIntegralFluxVelocity3D] regionName=outflow1; regionSize[m^2]=6.21106e-05; volumetricFlowRate[ml/s]=0; meanVelocity[m/s]=0
[SuperPlaneIntegralFluxPressure3D] regionName=outflow1; regionSize[m^2]=6.21106e-05; force[N]=0; meanPressure[mmHg]=0
[getResults] yPlusMax=70.9493
[Timer] step=28496; percent=40.0011; passedTime=63.095; remTime=94.6381; MLUPs=96.8552
[LatticeStatistics] step=28496; physT=0.800025; maxCFL=1.49167e-154; avgEnergy=0; avgRho=1
[SuperPlaneIntegralFluxVelocity3D] regionName=inflow; regionSize[m^2]=0.000310868; volumetricFlowRate[ml/s]=1.59319; meanVelocity[m/s]=0.00512497
[SuperPlaneIntegralFluxPressure3D] regionName=inflow; regionSize[m^2]=0.000310868; force[N]=483.591; meanPressure[mmHg]=11668.1
[SuperPlaneIntegralFluxVelocity3D] regionName=outflow0; regionSize[m^2]=6.40023e-05; volumetricFlowRate[ml/s]=0; meanVelocity[m/s]=0
[SuperPlaneIntegralFluxPressure3D] regionName=outflow0; regionSize[m^2]=6.40023e-05; force[N]=0; meanPressure[mmHg]=0
[SuperPlaneIntegralFluxVelocity3D] regionName=outflow1; regionSize[m^2]=6.21106e-05; volumetricFlowRate[ml/s]=0; meanVelocity[m/s]=0
[SuperPlaneIntegralFluxPressure3D] regionName=outflow1; regionSize[m^2]=6.21106e-05; force[N]=0; meanPressure[mmHg]=0
[getResults] yPlusMax=20.4464
[Timer] step=32058; percent=45.0013; passedTime=70.964; remTime=86.7294; MLUPs=96.7937
[LatticeStatistics] step=32058; physT=0.900028; maxCFL=1.49167e-154; avgEnergy=0; avgRho=1
[SuperPlaneIntegralFluxVelocity3D] regionName=inflow; regionSize[m^2]=0.000310868; volumetricFlowRate[ml/s]=0.4174; meanVelocity[m/s]=0.00134269
[SuperPlaneIntegralFluxPressure3D] regionName=inflow; regionSize[m^2]=0.000310868; force[N]=505.206; meanPressure[mmHg]=12189.6
[SuperPlaneIntegralFluxVelocity3D] regionName=outflow0; regionSize[m^2]=6.40023e-05; volumetricFlowRate[ml/s]=0; meanVelocity[m/s]=0
[SuperPlaneIntegralFluxPressure3D] regionName=outflow0; regionSize[m^2]=6.40023e-05; force[N]=0; meanPressure[mmHg]=0
[SuperPlaneIntegralFluxVelocity3D] regionName=outflow1; regionSize[m^2]=6.21106e-05; volumetricFlowRate[ml/s]=0; meanVelocity[m/s]=0
[SuperPlaneIntegralFluxPressure3D] regionName=outflow1; regionSize[m^2]=6.21106e-05; force[N]=0; meanPressure[mmHg]=0
[getResults] yPlusMax=20.5769
[Timer] step=35620; percent=50.0014; passedTime=78.814; remTime=78.8096; MLUPs=97.028
[LatticeStatistics] step=35620; physT=1.00003; maxCFL=1.49167e-154; avgEnergy=0; avgRho=1
[SuperPlaneIntegralFluxVelocity3D] regionName=inflow; regionSize[m^2]=0.000310868; volumetricFlowRate[ml/s]=-8.46732e-06; meanVelocity[m/s]=-2.72376e-08
[SuperPlaneIntegralFluxPressure3D] regionName=inflow; regionSize[m^2]=0.000310868; force[N]=508.549; meanPressure[mmHg]=12270.3
[SuperPlaneIntegralFluxVelocity3D] regionName=outflow0; regionSize[m^2]=6.40023e-05; volumetricFlowRate[ml/s]=0; meanVelocity[m/s]=0
[SuperPlaneIntegralFluxPressure3D] regionName=outflow0; regionSize[m^2]=6.40023e-05; force[N]=0; meanPressure[mmHg]=0
[SuperPlaneIntegralFluxVelocity3D] regionName=outflow1; regionSize[m^2]=6.21106e-05; volumetricFlowRate[ml/s]=0; meanVelocity[m/s]=0
[SuperPlaneIntegralFluxPressure3D] regionName=outflow1; regionSize[m^2]=6.21106e-05; force[N]=0; meanPressure[mmHg]=0
[getResults] yPlusMax=20.5963
[Timer] step=39182; percent=55.0015; passedTime=86.7; remTime=70.9319; MLUPs=96.5728
[LatticeStatistics] step=39182; physT=1.10003; maxCFL=1.49167e-154; avgEnergy=0; avgRho=1
[SuperPlaneIntegralFluxVelocity3D] regionName=inflow; regionSize[m^2]=0.000310868; volumetricFlowRate[ml/s]=0.406064; meanVelocity[m/s]=0.00130622
[SuperPlaneIntegralFluxPressure3D] regionName=inflow; regionSize[m^2]=0.000310868; force[N]=511.771; meanPressure[mmHg]=12348
[SuperPlaneIntegralFluxVelocity3D] regionName=outflow0; regionSize[m^2]=6.40023e-05; volumetricFlowRate[ml/s]=0; meanVelocity[m/s]=0
[SuperPlaneIntegralFluxPressure3D] regionName=outflow0; regionSize[m^2]=6.40023e-05; force[N]=0; meanPressure[mmHg]=0
[SuperPlaneIntegralFluxVelocity3D] regionName=outflow1; regionSize[m^2]=6.21106e-05; volumetricFlowRate[ml/s]=0; meanVelocity[m/s]=0
[SuperPlaneIntegralFluxPressure3D] regionName=outflow1; regionSize[m^2]=6.21106e-05; force[N]=0; meanPressure[mmHg]=0
[getResults] yPlusMax=20.6148
[Timer] step=42744; percent=60.0017; passedTime=94.599; remTime=63.0616; MLUPs=96.4139
[LatticeStatistics] step=42744; physT=1.20004; maxCFL=1.49167e-154; avgEnergy=0; avgRho=1
[SuperPlaneIntegralFluxVelocity3D] regionName=inflow; regionSize[m^2]=0.000310868; volumetricFlowRate[ml/s]=1.42205; meanVelocity[m/s]=0.00457443
[SuperPlaneIntegralFluxPressure3D] regionName=inflow; regionSize[m^2]=0.000310868; force[N]=533.068; meanPressure[mmHg]=12861.9
[SuperPlaneIntegralFluxVelocity3D] regionName=outflow0; regionSize[m^2]=6.40023e-05; volumetricFlowRate[ml/s]=0; meanVelocity[m/s]=0
[SuperPlaneIntegralFluxPressure3D] regionName=outflow0; regionSize[m^2]=6.40023e-05; force[N]=0; meanPressure[mmHg]=0
[SuperPlaneIntegralFluxVelocity3D] regionName=outflow1; regionSize[m^2]=6.21106e-05; volumetricFlowRate[ml/s]=0; meanVelocity[m/s]=0
[SuperPlaneIntegralFluxPressure3D] regionName=outflow1; regionSize[m^2]=6.21106e-05; force[N]=0; meanPressure[mmHg]=0
[getResults] yPlusMax=20.7326
[Timer] step=46306; percent=65.0018; passedTime=102.522; remTime=55.1997; MLUPs=96.134
[LatticeStatistics] step=46306; physT=1.30004; maxCFL=1.49167e-154; avgEnergy=0; avgRho=1
[SuperPlaneIntegralFluxVelocity3D] regionName=inflow; regionSize[m^2]=0.000310868; volumetricFlowRate[ml/s]=2.49626; meanVelocity[m/s]=0.00802995
[SuperPlaneIntegralFluxPressure3D] regionName=inflow; regionSize[m^2]=0.000310868; force[N]=583.735; meanPressure[mmHg]=14084.4
[SuperPlaneIntegralFluxVelocity3D] regionName=outflow0; regionSize[m^2]=6.40023e-05; volumetricFlowRate[ml/s]=0; meanVelocity[m/s]=0
[SuperPlaneIntegralFluxPressure3D] regionName=outflow0; regionSize[m^2]=6.40023e-05; force[N]=0; meanPressure[mmHg]=0
[SuperPlaneIntegralFluxVelocity3D] regionName=outflow1; regionSize[m^2]=6.21106e-05; volumetricFlowRate[ml/s]=0; meanVelocity[m/s]=0
[SuperPlaneIntegralFluxPressure3D] regionName=outflow1; regionSize[m^2]=6.21106e-05; force[N]=0; meanPressure[mmHg]=0
[getResults] yPlusMax=21.8774
[Timer] step=49868; percent=70.002; passedTime=110.432; remTime=47.3236; MLUPs=96.292
[LatticeStatistics] step=49868; physT=1.40004; maxCFL=1.49167e-154; avgEnergy=0; avgRho=1
[SuperPlaneIntegralFluxVelocity3D] regionName=inflow; regionSize[m^2]=0.000310868; volumetricFlowRate[ml/s]=3.0358; meanVelocity[m/s]=0.00976554
[SuperPlaneIntegralFluxPressure3D] regionName=inflow; regionSize[m^2]=0.000310868; force[N]=663.841; meanPressure[mmHg]=16017.2
[SuperPlaneIntegralFluxVelocity3D] regionName=outflow0; regionSize[m^2]=6.40023e-05; volumetricFlowRate[ml/s]=0; meanVelocity[m/s]=0
[SuperPlaneIntegralFluxPressure3D] regionName=outflow0; regionSize[m^2]=6.40023e-05; force[N]=0; meanPressure[mmHg]=0
[SuperPlaneIntegralFluxVelocity3D] regionName=outflow1; regionSize[m^2]=6.21106e-05; volumetricFlowRate[ml/s]=0; meanVelocity[m/s]=0
[SuperPlaneIntegralFluxPressure3D] regionName=outflow1; regionSize[m^2]=6.21106e-05; force[N]=0; meanPressure[mmHg]=0
[getResults] yPlusMax=70.2012
[Timer] step=53430; percent=75.0021; passedTime=118.364; remTime=39.4502; MLUPs=96.0249
[LatticeStatistics] step=53430; physT=1.50005; maxCFL=1.49167e-154; avgEnergy=0; avgRho=1
[SuperPlaneIntegralFluxVelocity3D] regionName=inflow; regionSize[m^2]=0.000310868; volumetricFlowRate[ml/s]=2.20282; meanVelocity[m/s]=0.00708603
[SuperPlaneIntegralFluxPressure3D] regionName=inflow; regionSize[m^2]=0.000310868; force[N]=762.216; meanPressure[mmHg]=18390.8
[SuperPlaneIntegralFluxVelocity3D] regionName=outflow0; regionSize[m^2]=6.40023e-05; volumetricFlowRate[ml/s]=0; meanVelocity[m/s]=0
[SuperPlaneIntegralFluxPressure3D] regionName=outflow0; regionSize[m^2]=6.40023e-05; force[N]=0; meanPressure[mmHg]=0
[SuperPlaneIntegralFluxVelocity3D] regionName=outflow1; regionSize[m^2]=6.21106e-05; volumetricFlowRate[ml/s]=0; meanVelocity[m/s]=0
[SuperPlaneIntegralFluxPressure3D] regionName=outflow1; regionSize[m^2]=6.21106e-05; force[N]=0; meanPressure[mmHg]=0
[getResults] yPlusMax=21.6485
[Timer] step=56992; percent=80.0022; passedTime=126.298; remTime=31.5701; MLUPs=95.9886
[LatticeStatistics] step=56992; physT=1.60005; maxCFL=1.49167e-154; avgEnergy=0; avgRho=1
[SuperPlaneIntegralFluxVelocity3D] regionName=inflow; regionSize[m^2]=0.000310868; volumetricFlowRate[ml/s]=2.40087; meanVelocity[m/s]=0.00772312
[SuperPlaneIntegralFluxPressure3D] regionName=inflow; regionSize[m^2]=0.000310868; force[N]=860.708; meanPressure[mmHg]=20767.2
[SuperPlaneIntegralFluxVelocity3D] regionName=outflow0; regionSize[m^2]=6.40023e-05; volumetricFlowRate[ml/s]=0; meanVelocity[m/s]=0
[SuperPlaneIntegralFluxPressure3D] regionName=outflow0; regionSize[m^2]=6.40023e-05; force[N]=0; meanPressure[mmHg]=0
[SuperPlaneIntegralFluxVelocity3D] regionName=outflow1; regionSize[m^2]=6.21106e-05; volumetricFlowRate[ml/s]=0; meanVelocity[m/s]=0
[SuperPlaneIntegralFluxPressure3D] regionName=outflow1; regionSize[m^2]=6.21106e-05; force[N]=0; meanPressure[mmHg]=0
[getResults] yPlusMax=21.9221
[Timer] step=60554; percent=85.0024; passedTime=134.257; remTime=23.688; MLUPs=95.6992
[LatticeStatistics] step=60554; physT=1.70005; maxCFL=1.49167e-154; avgEnergy=0; avgRho=1
[SuperPlaneIntegralFluxVelocity3D] regionName=inflow; regionSize[m^2]=0.000310868; volumetricFlowRate[ml/s]=1.54919; meanVelocity[m/s]=0.00498344
[SuperPlaneIntegralFluxPressure3D] regionName=inflow; regionSize[m^2]=0.000310868; force[N]=941.118; meanPressure[mmHg]=22707.3
[SuperPlaneIntegralFluxVelocity3D] regionName=outflow0; regionSize[m^2]=6.40023e-05; volumetricFlowRate[ml/s]=0; meanVelocity[m/s]=0
[SuperPlaneIntegralFluxPressure3D] regionName=outflow0; regionSize[m^2]=6.40023e-05; force[N]=0; meanPressure[mmHg]=0
[SuperPlaneIntegralFluxVelocity3D] regionName=outflow1; regionSize[m^2]=6.21106e-05; volumetricFlowRate[ml/s]=0; meanVelocity[m/s]=0
[SuperPlaneIntegralFluxPressure3D] regionName=outflow1; regionSize[m^2]=6.21106e-05; force[N]=0; meanPressure[mmHg]=0
[getResults] yPlusMax=22.1479
[Timer] step=64116; percent=90.0025; passedTime=142.197; remTime=15.7952; MLUPs=95.9282
[LatticeStatistics] step=64116; physT=1.80006; maxCFL=1.49167e-154; avgEnergy=0; avgRho=1
[SuperPlaneIntegralFluxVelocity3D] regionName=inflow; regionSize[m^2]=0.000310868; volumetricFlowRate[ml/s]=-0.217778; meanVelocity[m/s]=-0.000700546
[SuperPlaneIntegralFluxPressure3D] regionName=inflow; regionSize[m^2]=0.000310868; force[N]=992.16; meanPressure[mmHg]=23938.9
[SuperPlaneIntegralFluxVelocity3D] regionName=outflow0; regionSize[m^2]=6.40023e-05; volumetricFlowRate[ml/s]=0; meanVelocity[m/s]=0
[SuperPlaneIntegralFluxPressure3D] regionName=outflow0; regionSize[m^2]=6.40023e-05; force[N]=0; meanPressure[mmHg]=0
[SuperPlaneIntegralFluxVelocity3D] regionName=outflow1; regionSize[m^2]=6.21106e-05; volumetricFlowRate[ml/s]=0; meanVelocity[m/s]=0
[SuperPlaneIntegralFluxPressure3D] regionName=outflow1; regionSize[m^2]=6.21106e-05; force[N]=0; meanPressure[mmHg]=0
[getResults] yPlusMax=22.2753
[Timer] step=67678; percent=95.0027; passedTime=150.175; remTime=7.89951; MLUPs=95.4712
[LatticeStatistics] step=67678; physT=1.90006; maxCFL=1.49167e-154; avgEnergy=0; avgRho=1
[SuperPlaneIntegralFluxVelocity3D] regionName=inflow; regionSize[m^2]=0.000310868; volumetricFlowRate[ml/s]=0.239828; meanVelocity[m/s]=0.000771479
[SuperPlaneIntegralFluxPressure3D] regionName=inflow; regionSize[m^2]=0.000310868; force[N]=1013.76; meanPressure[mmHg]=24460.1
[SuperPlaneIntegralFluxVelocity3D] regionName=outflow0; regionSize[m^2]=6.40023e-05; volumetricFlowRate[ml/s]=0; meanVelocity[m/s]=0
[SuperPlaneIntegralFluxPressure3D] regionName=outflow0; regionSize[m^2]=6.40023e-05; force[N]=0; meanPressure[mmHg]=0
[SuperPlaneIntegralFluxVelocity3D] regionName=outflow1; regionSize[m^2]=6.21106e-05; volumetricFlowRate[ml/s]=0; meanVelocity[m/s]=0
[SuperPlaneIntegralFluxPressure3D] regionName=outflow1; regionSize[m^2]=6.21106e-05; force[N]=0; meanPressure[mmHg]=0
[getResults] yPlusMax=22.3259
[Timer]
[Timer] —————-Summary:Timer—————-
[Timer] measured time (rt) : 158.10s
[Timer] measured time (cpu): 163.815s
[Timer] average MLUPs : 91.587
[Timer] average MLUPps: 91.587
[Timer] ———————————————

And here is the output with CPU (the compilation of OpenLB has be done with the cpu_gcc_openmpi.mk file) :

utilisateur@LABO-ACC:~/release-1.8.1/examples/turbulence/aorta3d$ make clean; make; mpirun -np 32 ./aorta3d
rm -f tmp/*.* tmp/vtkData/*.* tmp/vtkData/data/*.* tmp/imageData/*.* tmp/imageData/data/*.* tmp/gnuplotData/*.* tmp/gnuplotData/data/*.*
rm -f aorta3d.o aorta3d.d aorta3d.wasm aorta3d.js aorta3d
make -C ../../../external
make[1] : on entre dans le répertoire « /home/utilisateur/release-1.8.1/external »
make -C zlib
make[2] : on entre dans le répertoire « /home/utilisateur/release-1.8.1/external/zlib »
make[2]: rien à faire pour « all ».
make[2] : on quitte le répertoire « /home/utilisateur/release-1.8.1/external/zlib »
cp zlib/build/libz.a lib/
make -C tinyxml2
make[2] : on entre dans le répertoire « /home/utilisateur/release-1.8.1/external/tinyxml2 »
make[2]: rien à faire pour « all ».
make[2] : on quitte le répertoire « /home/utilisateur/release-1.8.1/external/tinyxml2 »
cp tinyxml2/build/libtinyxml2.a lib/
make[1] : on quitte le répertoire « /home/utilisateur/release-1.8.1/external »
make -C ../../.. core
make[1] : on entre dans le répertoire « /home/utilisateur/release-1.8.1 »
make[1]: rien à faire pour « core ».
make[1] : on quitte le répertoire « /home/utilisateur/release-1.8.1 »
mpic++ -O3 -Wall -march=native -mtune=native -std=c++20 -pthread -DPARALLEL_MODE_MPI -DPLATFORM_CPU_SISD -DDEFAULT_FLOATING_POINT_TYPE=double -I../../../src -I../../../external/zlib -I../../../external/tinyxml2 -c -o aorta3d.o aorta3d.cpp
mpic++ aorta3d.o -o aorta3d -lolbcore -L../../../external/lib -lpthread -lz -ltinyxml2 -L../../../build/lib
Authorization required, but no authorization protocol specified

Authorization required, but no authorization protocol specified

[MpiManager] Sucessfully initialized, numThreads=32
[ThreadPool] Sucessfully initialized, numThreads=1
[UnitConverter] —————– UnitConverter information —————–
[UnitConverter] — Parameters:
[UnitConverter] Resolution: N= 40
[UnitConverter] Lattice velocity: latticeU= 0.0225
[UnitConverter] Lattice relaxation frequency: omega= 1.99697
[UnitConverter] Lattice relaxation time: tau= 0.50076
[UnitConverter] Characteristical length(m): charL= 0.02246
[UnitConverter] Characteristical speed(m/s): charU= 0.45
[UnitConverter] Phys. kinematic viscosity(m^2/s): charNu= 2.8436e-06
[UnitConverter] Phys. density(kg/m^d): charRho= 1055
[UnitConverter] Characteristical pressure(N/m^2): charPressure= 0
[UnitConverter] Mach number: machNumber= 0.0389711
[UnitConverter] Reynolds number: reynoldsNumber= 3554.3
[UnitConverter] Knudsen number: knudsenNumber= 1.09645e-05
[UnitConverter] Characteristical CFL number: charCFLnumber= 0.0225
[UnitConverter]
[UnitConverter] — Conversion factors:
[UnitConverter] Voxel length(m): physDeltaX= 0.0005615
[UnitConverter] Time step(s): physDeltaT= 2.8075e-05
[UnitConverter] Velocity factor(m/s): physVelocity= 20
[UnitConverter] Density factor(kg/m^3): physDensity= 1055
[UnitConverter] Mass factor(kg): physMass= 1.86768e-07
[UnitConverter] Viscosity factor(m^2/s): physViscosity= 0.01123
[UnitConverter] Force factor(N): physForce= 0.133049
[UnitConverter] Pressure factor(N/m^2): physPressure= 422000
[UnitConverter] ————————————————————-
[UnitConverter] WARNING:
[UnitConverter] Potentially UNSTABLE combination of relaxation time (tau=0.50076)
[UnitConverter] and characteristical CFL number (lattice velocity) charCFLnumber=0.0225!
[UnitConverter] Potentially maximum characteristical CFL number (maxCharCFLnumber=0.00607715)
[UnitConverter] Actual characteristical CFL number (charCFLnumber=0.0225) > 0.00607715
[UnitConverter] Please reduce the the cell size or the time step size!
[UnitConverter] We recommend to use the cell size of 0.000151659 m and the time step size of 7.58294e-06 s.
[UnitConverter] ————————————————————-
[STLreader] Voxelizing …
[STLmesh] nTriangles=2654; maxDist2=0.000610779
[STLmesh] minPhysR(StlMesh)=(0.199901,0.0900099,0.0117236); maxPhysR(StlMesh)=(0.243584,0.249987,0.0398131)
[Octree] radius=0.143744; center=(0.221602,0.169858,0.025628)
[STLreader] voxelSize=0.0005615; stlSize=0.001
[STLreader] minPhysR(VoxelMesh)=(0.199984,0.0904058,0.0118712); maxPhysR(VoxelMesh)=(0.24322,0.249872,0.0393847)
[STLreader] Voxelizing … OK
[prepareGeometry] Prepare Geometry …
[SuperGeometry3D] cleaned 0 outer boundary voxel(s)
[SuperGeometry3D] cleaned 0 outer boundary voxel(s)
[SuperGeometry3D] cleaned 0 inner boundary voxel(s) of Type 3
[SuperGeometryStatistics3D] updated
[SuperGeometry3D] the model is correct!
[CuboidDecomposition] —Cuboid Structure Statistics—
[CuboidDecomposition] Number of Cuboids: 256
[CuboidDecomposition] Delta : 0.0005615
[CuboidDecomposition] Ratio (min): 0.444444
[CuboidDecomposition] (max): 2.5
[CuboidDecomposition] Nodes (min): 360
[CuboidDecomposition] (max): 1800
[CuboidDecomposition] Weight (min): 277
[CuboidDecomposition] (max): 1800
[CuboidDecomposition] ——————————–
[SuperGeometryStatistics3D] materialNumber=0; count=60575; minPhysR=(0.199984,0.0898443,0.0113097); maxPhysR=(0.243781,0.250433,0.0399462)
[SuperGeometryStatistics3D] materialNumber=1; count=171218; minPhysR=(0.200546,0.0904058,0.0118712); maxPhysR=(0.24322,0.249872,0.0393847)
[SuperGeometryStatistics3D] materialNumber=2; count=41071; minPhysR=(0.199984,0.0898443,0.0113097); maxPhysR=(0.243781,0.250433,0.0399462)
[SuperGeometryStatistics3D] materialNumber=3; count=1059; minPhysR=(0.208407,0.250433,0.0124327); maxPhysR=(0.228059,0.250433,0.0332082)
[SuperGeometryStatistics3D] materialNumber=4; count=245; minPhysR=(0.200546,0.0898443,0.0298392); maxPhysR=(0.210653,0.0898443,0.0388232)
[SuperGeometryStatistics3D] materialNumber=5; count=239; minPhysR=(0.234236,0.0898443,0.0287162); maxPhysR=(0.24322,0.0898443,0.0388232)
[SuperGeometryStatistics3D] countTotal[1e6]=0.274407
[prepareGeometry] Prepare Geometry … OK
[prepareLattice] Prepare Lattice …
[prepareLattice] Prepare Lattice … OK
[Timer]
[Timer] —————-Summary:Timer—————-
[Timer] measured time (rt) : 0.103s
[Timer] measured time (cpu): 0.103s
[Timer] ———————————————
[main] starting simulation…
[Timer] step=0; percent=0; passedTime=0.111; remTime=7907.31; MLUPs=0
[LatticeStatistics] step=0; physT=0; maxCFL=1.49167e-154; avgEnergy=0; avgRho=1
[SuperPlaneIntegralFluxVelocity3D] regionName=inflow; regionSize[m^2]=0.000310868; volumetricFlowRate[ml/s]=0; meanVelocity[m/s]=0
[SuperPlaneIntegralFluxPressure3D] regionName=inflow; regionSize[m^2]=0.000310868; force[N]=0; meanPressure[mmHg]=0
[SuperPlaneIntegralFluxVelocity3D] regionName=outflow0; regionSize[m^2]=6.40023e-05; volumetricFlowRate[ml/s]=0; meanVelocity[m/s]=0
[SuperPlaneIntegralFluxPressure3D] regionName=outflow0; regionSize[m^2]=6.40023e-05; force[N]=0; meanPressure[mmHg]=0
[SuperPlaneIntegralFluxVelocity3D] regionName=outflow1; regionSize[m^2]=6.21106e-05; volumetricFlowRate[ml/s]=0; meanVelocity[m/s]=0
[SuperPlaneIntegralFluxPressure3D] regionName=outflow1; regionSize[m^2]=6.21106e-05; force[N]=0; meanPressure[mmHg]=0
[getResults] yPlusMax=2.22507e-308
[Timer] step=3562; percent=5.00014; passedTime=134.9; remTime=2563.02; MLUPs=5.65083
[LatticeStatistics] step=3562; physT=0.100003; maxCFL=0.0360234; avgEnergy=5.51886e-05; avgRho=1.00067
[SuperPlaneIntegralFluxVelocity3D] regionName=inflow; regionSize[m^2]=0.000310868; volumetricFlowRate[ml/s]=6.88058; meanVelocity[m/s]=0.0221334
[SuperPlaneIntegralFluxPressure3D] regionName=inflow; regionSize[m^2]=0.000310868; force[N]=0.0477156; meanPressure[mmHg]=1.15128
[SuperPlaneIntegralFluxVelocity3D] regionName=outflow0; regionSize[m^2]=6.40023e-05; volumetricFlowRate[ml/s]=-2.97519; meanVelocity[m/s]=-0.0464857
[SuperPlaneIntegralFluxPressure3D] regionName=outflow0; regionSize[m^2]=6.40023e-05; force[N]=0.000318598; meanPressure[mmHg]=0.0373375
[SuperPlaneIntegralFluxVelocity3D] regionName=outflow1; regionSize[m^2]=6.21106e-05; volumetricFlowRate[ml/s]=-2.84465; meanVelocity[m/s]=-0.0457997
[SuperPlaneIntegralFluxPressure3D] regionName=outflow1; regionSize[m^2]=6.21106e-05; force[N]=0.000319367; meanPressure[mmHg]=0.0385676
[getResults] yPlusMax=1.4707

I killed the simulation here because it is enough to see the difference with the outflow.

Thanks in advance and good luck !

[sfraniatte]

I forgot to mention that it’s only the outlet pressure condition that doesn’t work (the flow remains zero) with the MRT method.

[sfraniatte]

Hello,

I achieved a significant performance gain by changing the number of cuboids. I started from the aorta3d example, which used too many cuboids (8 per CPU core in parallel), and I greatly reduced this number (1 per core with 32 cores).

Hope this helps. Good luck!

Sylvain

[sfraniatte]

Hello,

I have solved my problem by looking in the file “stlReader.hh”. Then I have seen these lines :
template<typename T>
bool STLtriangle<T>::isPointInside(const PhysR<T,3>& pt) const
{
// tests with T=double and T=float show that the epsilon must be increased
const T epsilon = std::numeric_limits<BaseType<T>>::epsilon()*T(10);

const T beta = pt * uBeta + kBeta;
const T gamma = pt * uGamma + kGamma;

// check if approximately equal
if ( util::nearZero(norm(pt – (point[0].coords + beta*(point[1].coords-point[0].coords) + gamma*(point[2].coords-point[0].coords))), epsilon) ) {
const T alpha = T(1) – beta – gamma;
return (beta >= T(0) || util::nearZero(beta, epsilon))
&& (gamma >= T(0) || util::nearZero(gamma, epsilon))
&& (alpha >= T(0) || util::nearZero(alpha, epsilon));
}
return false;
}

There is a first solution which is change the FLOATING_POINT_TYPE in the file “config.mk” from double to float. But the calculation is slower. So, I am trying another solution which is to change the “epsilon” used by the STLreader in two files : “stlReader.hh” and “octree.hh”

[sfraniatte]

I do not know where is the problem exactly. If it is during the stlReader operation or rename operation. I have some indicator’s voxels which are wrong and the only solution seems to do not use GPU. I am sure that the problem comes from the GPU usage or the nvcc (whtih an nvidia card on Ampere architecture) usage to compile the code because I tested with gcc and mpirun on CPU whithout any problem.

Thank you for your time !

[sfraniatte]

Yes, I can. Here is the good .mk file :

# Example build config for OpenLB using CUDA on single GPU systems
#
# Tested using CUDA 11.4
#
# Usage:
# – Copy this file to OpenLB root as config.mk
# – Adjust CUDA_ARCH to match your specifc GPU
# – Run make clean; make
# – Switch to example directory, e.g. examples/laminar/cavity3dBenchmark
# – Run make
# – Start the simulation using ./cavity3d

CXX := nvcc
CC := nvcc

CXXFLAGS := -O3
CXXFLAGS += -std=c++17 –forward-unknown-to-host-compiler

PARALLEL_MODE := NONE

PLATFORMS := CPU_SISD GPU_CUDA

# for e.g. RTX 30* (Ampere), see table in rules.mk for other options
CUDA_ARCH := 86

FLOATING_POINT_TYPE := float

USE_EMBEDDED_DEPENDENCIES := ON

###########################################################################################
Here is the first one which does not work very well :
# OpenLB build configuration
#
# This file sets up the necessary build flags for compiling OpenLB with
# the GNU C++ compiler and sequential execution. For more complex setups
# edit this file or consult the example configs provided in config/.
#
# Basic usage:
# – Edit variables to fit desired configuration
# – Run make clean; make to clean up any previous artifacts and compile the dependencies
# – Switch to example directory, e.g. examples/laminar/poiseuille2d
# – Run make
# – Start the simulation using ./poiseuille2d

# Compiler to use for C++ files, change to mpic++ when using OpenMPI and GCC
#~ #parallel CPU ou hybrid
#~ CXX := mpic++
#GPU
CXX := nvcc

# Compiler to use for C files (used for emebedded dependencies)
#parallel CPU ou hybrid
#~ CC := gcc
#GPU
CC := nvcc

# Suggested optimized build flags for GCC, consult config/ for further examples
#parallel CPU ou hybrid
#~ CXXFLAGS := -O3 -Wall -march=native -mtune=native
#GPU
CXXFLAGS := -O3
CXXFLAGS += –forward-unknown-to-host-compiler
# Uncomment to add debug symbols and enable runtime asserts
#~ #CXXFLAGS += -g -DOLB_DEBUG

# OpenLB requires support for C++17
# works in:
# * gcc 9 or later (https://gcc.gnu.org/projects/cxx-status.html#cxx17)
# * icc 19.0 or later (https://software.intel.com/en-us/articles/c17-features-supported-by-intel-c-compiler)
# * clang 7 or later (https://clang.llvm.org/cxx_status.html#cxx17)
CXXFLAGS += -std=c++17

# optional linker flags
LDFLAGS :=

# Parallelization mode, must be one of: OFF, MPI, OMP, HYBRID
# Note that for MPI and HYBRID the compiler also needs to be adapted.
# See e.g. config/cpu_gcc_openmpi.mk
#parallel CPU
#~ PARALLEL_MODE := MPI
#GPU
PARALLEL_MODE := NONE
#~ #hybrid
#~ PARALLEL_MODE := HYBRID

# optional MPI and OpenMP flags
#parallel CPU
#~ MPIFLAGS :=
#~ OMPFLAGS := -fopenmp

# Options: CPU_SISD, CPU_SIMD, GPU_CUDA
# Both CPU_SIMD and GPU_CUDA require system-specific adjustment of compiler flags.
# See e.g. config/cpu_simd_intel_mpi.mk or config/gpu_only.mk for examples.
# CPU_SISD must always be present.
#parallel CPU
#~ PLATFORMS := CPU_SISD
#GPU
PLATFORMS := CPU_SISD GPU_CUDA
#hybrid
#~ PLATFORMS := CPU_SISD CPU_SIMD GPU_CUDA
#~ # Compiler to use for CUDA-enabled files
#~ CUDA_CXX := nvcc
#~ CUDA_CXXFLAGS := -O3 -std=c++17
#~ # Adjust to enable resolution of libcuda, libcudart, libcudadevrt
#~ CUDA_LDFLAGS := -L/run/libcuda/lib
#~ CUDA_LDFLAGS += -fopenmp
#~ #GPU ou hybrid
CUDA_ARCH := 86

# Fundamental arithmetic data type
# Common options are float or double
#parallel CPU
#~ FLOATING_POINT_TYPE := double
#GPU ou hybrid
FLOATING_POINT_TYPE := float

# Any entries are passed to the compiler as -DFEATURE_* declarations
# Used to enable some alternative code paths and dependencies
FEATURES :=

# Set to OFF if libz and tinyxml are provided by the system (optional)
USE_EMBEDDED_DEPENDENCIES := ON

###################################################################################
Also, I am trying to run my case on GPU but it is really too slow. The main difference with aorta example (which works well for me now) is the surface of the inlet (which is really bigger) and the fact that there are external edges and corners (the inlet has 5 faces). I am working on cleaning my code to have something as in the nozzle example (with stlReader). It can be that but I am not sure.

Thank you !

[sfraniatte]

Hello,

Thank you for your reply (which I’ve only just seen now because I forgot to check the notification option). In the meantime, I started working on implementing such a condition. That said, I feel like the approach you’re talking about complements the one I’ve coded. I’ll start testing my development tomorrow. In any case, I do plan to come to Marseille. Thanks again!

Best regards,

Sylvain

[Author]

Posts