[nipinl]

Hi Adrian,
Thank you very much for the suggestions.

  AnalyticalFfromSuperF3D<T,T> uAnalytical(coarseU, /*communicateToAll   =*/ false,  /*communicateOverlap =*/ true);

with using loadbalancer of coarse for decomposing fine worked !!
Best,
Nipin

For future readers, following was employed for decomposition:
Coarse:

IndicatorCuboid3D<T> cuboidCoarse(extendCoarse, originCoarse);
CuboidDecomposition3D<T> cuboidDecompositionCoarse(
    cuboidCoarse, converterCoarse.getPhysDeltaX(), noOfCuboids);

HeuristicLoadBalancer<T> loadBalancerCoarse(cuboidDecompositionCoarse);

SuperGeometry<T,3> superGeometryCoarse(
    cuboidDecompositionCoarse, loadBalancerCoarse, 4);

Fine:

IndicatorCuboid3D<T> cuboidFine(extendFine, originFine);
CuboidDecomposition3D<T> cuboidDecompositionFine(
    cuboidFine, converterFine.getPhysDeltaX(), noOfCuboids);
SuperGeometry<T,3> superGeometryFine(
    cuboidDecompositionFine, loadBalancerCoarse, 4)

[nipinl]

Image: https://www.dropbox.com/scl/fi/dcyze2b6b745c389vldld/serialCopy.png?rlkey=wobmt9paxqqc0awidnfannffn&st=agsktu3f&dl=0

[left (top and bottom) is the converged coarse run and right is the vtm file for fine lattice at time=0. ]

• This reply was modified 1 month, 4 weeks ago by nipinl. Reason: image
• This reply was modified 1 month, 4 weeks ago by nipinl. Reason: added description

[nipinl]

Hi Adrain,
First of all, thank you for helping me out here.
When run in serial, the current code works almost correct, just that it did not copy at x=xMax plane and at the edge (xMin,yMin) https://www.dropbox.com/scl/fi/dcyze2b6b745c389vldld/serialCopy.png?rlkey=wobmt9paxqqc0awidnfannffn&st=pjacrhww&dl=0

However, when using AnalyticalFfromSuperF3D<T,T> uAnalytical(coarseU, /*communicateToAll =*/ false, /*communicateOverlap =*/ true); for serial run, it fails citing “double free or corruption (!prev) tri0570:1320121] *** Process received signal *** ”

Regarding the decomposition, I use

CuboidDecomposition3D<T> cuboidDecompositionCoarse(
    cuboidCoarse, converterCoarse.getPhysDeltaX(), noOfCuboids);
CuboidDecomposition3D<T> cuboidDecompositionFine(
    cuboidFine, converterFine.getPhysDeltaX(), noOfCuboids);

I was wondering, how can we use same decomposition for both coarse and fine, since N and PhysDx are different?

[nipinl]

I made coarse and fine versions of superGeometry, SuperLattice based on resolutions Ncoarse and Nfine; and enforced same boundary conditions for both lattices separately. The simulation is planned as: Coarse run –> interpolate –> fine run; all in a single (cpp file). A helper function (provided below) is made to initialize fine lattice from Coarse.
However, I’m unsuccessful in carrying out the initialization. The provided version partially copies data to fine lattice. If I change “…uAnalytical(coarseU, false, false)” to “…uAnalytical(coarseU, true, true)” to enable communication, MPI throws following error (line “coarseU.getSuperStructure().communicate();” is commented out when false changes to true). I was wondering where exactly I’m making a mistake here.

ERROR:
[initializeFineFromCoarse] Interpolating coarse lattice onto fine lattice … [initializeFineFromCoarse] Defined rhoAnalytical … [tri-login02:00000] *** An error occurred in MPI_Bcast [tri-login02:00000] *** reported by process [2194538497,169] [tri-login02:00000] *** on communicator MPI_COMM_WORLD [tri-login02:00000] *** MPI_ERR_TRUNCATE: message truncated [tri-login02:00000] *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort, [tri-login02:00000] *** and MPI will try to terminate your MPI job as well) ————————————————————————– prterun has exited due to process rank 169 with PID 0 on node tri-login02 calling “abort”. This may have caused other processes in the application to be terminated by signals sent by prterun (as reported here).

// Interpolate a coarse simulation onto the current (fine) lattice
template <typename T, typename DESCRIPTOR>
void initializeFineFromCoarse(
  SuperLattice<T, DESCRIPTOR>& sLatticeFine,
  SuperGeometry<T,3>&          superGeometryFine,
  SuperLattice<T, DESCRIPTOR>& sLatticeCoarse,
  SuperGeometry<T,3>&          superGeometryCoarse)
{
  OstreamManager clout(std::cout, "initializeFineFromCoarse");

  SuperLatticeVelocity3D<T, DESCRIPTOR> coarseU(sLatticeCoarse);

  coarseU.getSuperStructure().communicate();

  AnalyticalFfromSuperF3D<T,T> uAnalytical(coarseU, false, false);

  AnalyticalConst3D<T,T> rhoAnalytical(1.);

  sLatticeFine.defineRhoU(superGeometryFine, 1, rhoAnalytical, uAnalytical);
  sLatticeFine.iniEquilibrium(superGeometryFine, 1, rhoAnalytical, uAnalytical);

  sLatticeFine.defineRhoU(superGeometryFine, 2, rhoAnalytical, uAnalytical);
  sLatticeFine.iniEquilibrium(superGeometryFine, 2, rhoAnalytical, uAnalytical);

  sLatticeFine.initialize();
}

[nipinl]

Hi Fedor,
As you suggested on the other thread, using DNS with ThirdOrderRLBdynamics solved the problem. Thanks again !!

[nipinl]

Hi Fedor,

I ran the pipe flow DNS using ThirdOrderRLBdynamics with D3Q19. It ran successfully for some time but eventually went to “nan” as the average density exceeded around 1.038.

However, when I used ThirdOrderRLBdynamics for my channel flow case, the re-laminarization issue was resolved, which was my main concern. I had originally attempted the nozzle case to address this issue.

Thank you for suggesting this!

Best,
Nipin

[nipinl]

Hi Fedor,
Thank you for the suggestion. I will try DNS with ThirdOrderRLBdynamics and D3Q19. I’ll update how it went.
Thanks again!!

Best,
Nipin

[nipinl]

Hi Fedor,
Thank you for the prompt reply. For DNS, I use BGK with D3Q27 descriptor. I was using RLB for only LES, which gives good results.Thanks for the suggestion regarding vortex inlet, I’ll use that. However, it is the significant lack of mass conservation in DNS (BGK) that worries me the most. This makes the simulation not useful.

Best,
Nipin

[nipinl]

Hi Bukreev,
Thank you for your reply and confirming the dynamics and BC. I did not use a higher lattice velocity because I wanted to be conservative on the Mach Number side. Ma correspond to the characteristic velocity is 0.044 and maximum velocity goes 1.3 times the characteristic velocity. That said, a slightly higher lattice velocity gives similar result.

[nipinl]

Velocity contours: Velocity Contours

LOG: Y+ 2.25 DNS :

`[UnitConverter] —————– UnitConverter information —————–
[UnitConverter] — Parameters:
[UnitConverter] Resolution: N= 200
[UnitConverter] Lattice velocity: latticeU= 0.0254542
[UnitConverter] Lattice relaxation frequency: omega= 1.99611
[UnitConverter] Lattice relaxation time: tau= 0.500973
[UnitConverter] Characteristical length(m): charL= 0.00199
[UnitConverter] Characteristical speed(m/s): charU= 7.88506
[UnitConverter] Phys. kinematic viscosity(m^2/s): charNu= 1e-06
[UnitConverter] Phys. density(kg/m^d): charRho= 1000
[UnitConverter] Characteristical pressure(N/m^2): charPressure= 0
[UnitConverter] Mach number: machNumber= 0.0440879
[UnitConverter] Reynolds number: reynoldsNumber= 15691.3
[UnitConverter] Knudsen number: knudsenNumber= 2.80971e-06
[UnitConverter]
[UnitConverter] — Conversion factors:
[UnitConverter] Voxel length(m): physDeltaX= 9.95e-06
[UnitConverter] Time step(s): physDeltaT= 3.21201e-08
[UnitConverter] Velocity factor(m/s): physVelocity= 309.775
[UnitConverter] Density factor(kg/m^3): physDensity= 1000
[UnitConverter] Mass factor(kg): physMass= 9.85075e-13
[UnitConverter] Viscosity factor(m^2/s): physViscosity= 0.00308226
[UnitConverter] Force factor(N): physForce= 0.00950033
[UnitConverter] Pressure factor(N/m^2): physPressure= 9.59605e+07
[UnitConverter] ————————————————————-
[main] ———————————————————————-
[main] Converge time(s): 0.266661
[main] Lattice converge time: 8301991
[main] Max. Phys. simulation time(s): 0.355548
[main] Max. Lattice simulation time: 11069321
[main] Frequency Statistics Save(Hz): 25
[main] Statistics save period(s): 0.04
[main] Lattice statistics save period: 1245326
[main] ———————————————————————-
[main] Channel height(m): 0.00199
[main] y+ value: 2.25004
[main] y+ value spacing: 4.4776

[nipinl]

Image

• This reply was modified 3 months, 1 week ago by nipinl. Reason: Added Image

[nipinl]

Sounds good!! Thanks again.

[nipinl]

Hi Mathias,
Thanks for the lead.I understand that the setPeriodicity() function in the cuboidGeometry class sets the periodicity.
Actually, I was looking for a stream-wise periodic boundary condition for channel/pipe flow with pressure gradient. One can achieve this with body force, but I read here that it is only applicable if the cross-section area is constant. Periodic bc with pressure gradient might include modification of the inlet density[Kim 2007]. I was wondering if this is already implemented in OpenLB?

Thanks!!
Best,
Nipin

[nipinl]

Hi,
I’ve commented out all calculations related to yPlus and then simulation is working fine.

Best,
Nipin

[nipinl]

Hi Adrian,
Thank you for your reply. I’m copying the backtrace here. I think I was vague in communicating the point at which simulation crashes. It is after stl file loading. Simulation works perfectly for coarser resolution;I tried N = 40 and 100 and it works. But it crashes for N>200.

Thank you for your time,
Best,
Nipin

Thread 1 “cylinder3d” received signal SIGABRT, Aborted.
0x0000155554d7e7ac in ?? () from /cvmfs/soft.computecanada.ca/gentoo/2023/x86-64-v3/usr/lib64/libc.so.6
#0 0x0000155554d7e7ac in ?? () from /cvmfs/soft.computecanada.ca/gentoo/2023/x86-64-v3/usr/lib64/libc.so.6
#1 0x0000155554d306f2 in raise () from /cvmfs/soft.computecanada.ca/gentoo/2023/x86-64-v3/usr/lib64/libc.so.6
#2 0x0000155554d1a4b2 in abort () from /cvmfs/soft.computecanada.ca/gentoo/2023/x86-64-v3/usr/lib64/libc.so.6
#3 0x0000155554d1a3d5 in ?? () from /cvmfs/soft.computecanada.ca/gentoo/2023/x86-64-v3/usr/lib64/libc.so.6
#4 0x0000155554d29662 in __assert_fail () from /cvmfs/soft.computecanada.ca/gentoo/2023/x86-64-v3/usr/lib64/libc.so.6
#5 0x000000000048ebc9 in olb::util::normalize<double> (a=std::vector of length 3, capacity 3 = {…}) at ../../../src/utilities/vectorHelpers.h:204
#6 0x00000000004a6d8b in olb::SuperLatticeYplus3D<double, olb::descriptors::D3Q19<olb::descriptors::FORCE> >::operator() (this=0x7fffffff4120, output=0x7fffffff3a30, input=0x7fffffff3a20) at ../../../src/functors/lattice/turbulentF3D.hh:77
#7 0x0000000000532b88 in olb::FunctorPtr<olb::SuperF3D<double, double> >::operator()<double*, int*> (this=0x7fffffff4360) at ../../../src/utilities/functorPtr.h:100
#8 olb::SuperMax3D<double, double>::operator() (this=this@entry=0x7fffffff4300, output=output@entry=0x7fffffff3e30, input=input@entry=0x7fffffff3d60) at ../../../src/functors/lattice/superMax3D.hh:88
#9 0x0000000000414fcb in getResults (stlReader=…, timer=…, superGeometry=warning: RTTI symbol not found for class ‘olb::SuperGeometry<double, 3u>’
…, iT=0, converter=…, sLattice=…) at cylinder3d.cpp:325
#10 main (argc=<optimized out>, argv=<optimized out>) at cylinder3d.cpp:466
#0 0x0000155554d7e7ac in ?? () from /cvmfs/soft.computecanada.ca/gentoo/2023/x86-64-v3/usr/lib64/libc.so.6
#1 0x0000155554d306f2 in raise () from /cvmfs/soft.computecanada.ca/gentoo/2023/x86-64-v3/usr/lib64/libc.so.6
#2 0x0000155554d1a4b2 in abort () from /cvmfs/soft.computecanada.ca/gentoo/2023/x86-64-v3/usr/lib64/libc.so.6
#3 0x0000155554d1a3d5 in ?? () from /cvmfs/soft.computecanada.ca/gentoo/2023/x86-64-v3/usr/lib64/libc.so.6
#4 0x0000155554d29662 in __assert_fail () from /cvmfs/soft.computecanada.ca/gentoo/2023/x86-64-v3/usr/lib64/libc.so.6
#5 0x000000000048ebc9 in olb::util::normalize<double> (a=std::vector of length 3, capacity 3 = {…}) at ../../../src/utilities/vectorHelpers.h:204
#6 0x00000000004a6d8b in olb::SuperLatticeYplus3D<double, olb::descriptors::D3Q19<olb::descriptors::FORCE> >::operator() (this=0x7fffffff4120, output=0x7fffffff3a30, input=0x7fffffff3a20) at ../../../src/functors/lattice/turbulentF3D.hh:77
#7 0x0000000000532b88 in olb::FunctorPtr<olb::SuperF3D<double, double> >::operator()<double*, int*> (this=0x7fffffff4360) at ../../../src/utilities/functorPtr.h:100
#8 olb::SuperMax3D<double, double>::operator() (this=this@entry=0x7fffffff4300, output=output@entry=0x7fffffff3e30, input=input@entry=0x7fffffff3d60) at ../../../src/functors/lattice/superMax3D.hh:88
#9 0x0000000000414fcb in getResults (stlReader=…, timer=…, superGeometry=warning: RTTI symbol not found for class ‘olb::SuperGeometry<double, 3u>’
…, iT=0, converter=…, sLattice=…) at cylinder3d.cpp:325
#10 main (argc=<optimized out>, argv=<optimized out>) at cylinder3d.cpp:466

[Author]

Posts