[jdm]

Thanks!
I have set OMP_PROC_BIND=close and OMP_PLACES=cores as you suggested, otherwise I used the same configuration as previously.

[jdm]

Hi again,

I have been able to run a quick benchmark using the nozzle3d example on OpenLB 1.4.

As you can see, the maximum speedup using OMP is about x7.7. Surprisingly, this maximum speedup with OMP is achieved using only 17 threads out of 144. adding more than 17 threads increases the runtime.
On the contrary, the maximum speedup using MPI if about x26 and is achieved using 72 processes (for 72 cores).

I hope this can useful for you!
jdm

[jdm]

Hi Adrian,

I’ll keep you updated as soon as I can try running OpenLB examples with OMP on the cluster!

Changing the angle brackets for quotation marks solved the issue. My simulation using OpenLB as a shared library (so outside of OpenLB folder) is working good now.
In this simulation, I encapsulate the OpenLB simulation inside an other simulation framework (the BioDynaMo framework).

In brief, I run the main (BioDynaMo) simulation that will call OpenLB every x time steps. Information from the BioDynaMo simulation are used to construct, initialise and run the OpenLB simulation. Results from The OpenLB simulation are then transmitted to BioDynaMo in order to update it, run it a number of time steps before calling OpenLB again. This loop is repeated several times to complete the main BioDynaMo simulation.
We run the whole process inside a docker container for better control of the environment and for repeatability.

I use the precompiled mode of OpenLB for the shared library, yes.

[jdm]

I can not post the error message text directly as it is perceived as ‘A potentially unsafe operation has been detected in your request to this site’ by Wordfence.
Here it is as a picture

Sorry for the repost
Jdm

[jdm]

Hi Adrian,

Thanks a lot for your detailed answer.

I updated to OpenLB 1.4 on my small system (6 cores / 6 threads) and noticed an improvement when using OMP. When running the nozzel3D example on 1.3 the speedup was around x1.4 while it is now around x2.9 when using 1.4.
I couldn’t try yet on the bigger system (72 cores / 144 threads) unfortunately, but will try as soon as possible. As using OMP on this big system was increasing the run time (taking 3 times longer), it will be very interesting to see if the 1.4 version improve the situation.

With version 1.3, when running the nozzel3D example with MPI, the speedup was about x19 on my big system (72 cores).

However, when trying to build my own simulation with olb 1.4, I now have a compilation error

In my own simulation, OpenLB is compiled as a shared library (using the -fPIC cxx flag), and the simulation code is outside of the olb-1.4r0 folder. This wasn’t a problem with OpenLB 1.3, but I guess it is now due to the relative address used to include functors3D.h in particleOperations3D.h.
Do you know how I can solve this issue?

Best,
jdm

[jdm]

Dear all,

Thanks Stephan and Adrian for your answer.

– I used the OMP mode, not the Hybrid one, indeed.

– here is my OpenMP environment:

OPENMP DISPLAY ENVIRONMENT BEGIN
  _OPENMP = '201511'
  OMP_DYNAMIC = 'FALSE'
  OMP_NESTED = 'FALSE'
  OMP_NUM_THREADS = '144'
  OMP_SCHEDULE = 'DYNAMIC'
  OMP_PROC_BIND = 'TRUE'
  OMP_PLACES = '{0},{72},{1},{73},{2},{74},{3},{75},{4},{76},{5},{77},{6},{78},{7},{79},{8},{80},{9},{81},{10},{82},{11},{83},{12},{84},{13},{85},{14},{86},{15},{87},{16},{88},{17},{89},{18},{90},{19},{91},{20},{92},{21},{93},{22},{94},{23},{95},{24},{96},{25},{97},{26},{98},{27},{99},{28},{100},{29},{101},{30},{102},{31},{103},{32},{104},{33},{105},{34},{106},{35},{107},{36},{108},{37},{109},{38},{110},{39},{111},{40},{112},{41},{113},{42},{114},{43},{115},{44},{116},{45},{117},{46},{118},{47},{119},{48},{120},{49},{121},{50},{122},{51},{123},{52},{124},{53},{125},{54},{126},{55},{127},{56},{128},{57},{129},{58},{130},{59},{131},{60},{132},{61},{133},{62},{134},{63},{135},{64},{136},{65},{137},{66},{138},{67},{139},{68},{140},{69},{141},{70},{142},{71},{143}'
  OMP_STACKSIZE = '0'
  OMP_WAIT_POLICY = 'PASSIVE'
  OMP_THREAD_LIMIT = '4294967295'
  OMP_MAX_ACTIVE_LEVELS = '2147483647'
  OMP_CANCELLATION = 'FALSE'
  OMP_DEFAULT_DEVICE = '0'
  OMP_MAX_TASK_PRIORITY = '0'
  OMP_DISPLAY_AFFINITY = 'FALSE'
  OMP_AFFINITY_FORMAT = 'level %L thread %i affinity %A'
OPENMP DISPLAY ENVIRONMENT END

– Unfortunately, I can not easily use MPI on my actual simulation so I would like to avoid anything else than OMP.
I also tried OMP on a much more modest computer (6 cores / 6 threads) and noticed only moderate speedup (~1.4, with all cores running at 100%).

Best,
jdm

[jdm]

Dear all,

Thanks to your help I have been able to build my showcase.
I am still missing one functionality though, which is to retrieve the velocity at one point in space. Is there a method like Get3DVelocity({x, y, z})?

I also encountered a problem when running an OpenLB example with OMP.
On a 74 cores / 144 threads computer (centos, Red Hat 4.8.5), the nozzle3d example took 3 times longer with OMP enabled than without OMP, while still using all the 144 threads at 100% during the whole simulation.
Has anybody ever faced this issue? How can it be solved?
I tried using MPI and the runtime was 19 times quicked using MPI than no parallel mode.
However, I unfortunately can not (easily) use MPI for my own simulation, which is why I would like to use OMP.

Thanks in advance for your help!
jdm

[jdm]

Hi both,

Thanks to your help I have been able to solve a couple of issues. The geometry is now correct, and the cylinder used as the inlet is created at the right position.
I also changed the reference length of simulation geometry (charPhysLength is set to 0.01) as my StL file is in cm and not m.

However, running the simulation with the cylinder as an inlet now raises a seg fault.
it is raised during the setBoundaryValues function, when trying to set:

lattice.iniEquilibrium( superGeometry, 3, rhoF, uSol );

here is the seg fault if needed:

#6  0x00007ff2e04da070 in olb::Dynamics<double, olb::descriptors::D3Q19<> >::iniEquilibrium(olb::Cell<double, olb::descriptors::D3Q19<> >&, double, double const*) () from /home/jdm/git/closed_space/build/libclosed_space.so
#7  0x00007ff2e062aa76 in olb::BlockLatticeStructure3D<double, olb::descriptors::D3Q19<> >::iniEquilibrium(olb::BlockIndicatorF3D<double>&, olb::AnalyticalF3D<double, double>&, olb::AnalyticalF3D<double, double>&) () from /home/jdm/git/closed_space/build/libclosed_space.so
#8  0x00007ff2e04e264a in olb::SuperLattice3D<double, olb::descriptors::D3Q19<> >::iniEquilibrium(olb::FunctorPtr<olb::SuperIndicatorF3D<double> >&&, olb::AnalyticalF3D<double, double>&, olb::AnalyticalF3D<double, double>&) () from /home/jdm/git/closed_space/build/libclosed_space.so
#9  0x00007ff2e04e3d98 in olb::SuperLattice3D<double, olb::descriptors::D3Q19<> >::iniEquilibrium(olb::SuperGeometry3D<double>&, int, olb::AnalyticalF3D<double, double>&, olb::AnalyticalF3D<double, double>&) () from /home/jdm/git/closed_space/build/libclosed_space.so
#10 0x00007ff2e0499fcd in openlb_sim::setBoundaryValues(olb::UnitConverter<double, olb::descriptors::D3Q19<> > const&, olb::SuperLattice3D<double, olb::descriptors::D3Q19<> >&, olb::SuperGeometry3D<double>&, int) () from /home/jdm/git/closed_space/build/libclosed_space.so
d

Both “prepare geometry” and “prepare Lattice” return OK before the seg fault.
Parameters are taken from the nozzle3d examples, and weren’t raising a seg fault before (when the diffusion of the inlet was happening in the inside of the sphere).
Changing the charPhysLength and stlReader back to 1 (so in meter) doesn’t have an impact.
As said at the begining of this post, the cylinder used as the inlet is correctly placed in front of the sphere. I tried to place it further away, at mid distance between the sphere and the wall as shown in the figure, with similar results.
(in this figure, the cylinder used as inlet is displayed with a material set as 0 instead of 3 only for visualisation purpose)

Does anybody has an idea what could be causing this issue?

Thanks again for your help,
jdm

[jdm]

Hi Stephan,

Thanks for your advise.
This is a slice I obtained when I visualise the geometry using the vtk file, instead of the stl used as an input:

<br />site png<br />

If I understand correctly, could it be that the geometry is inverted, and that diffusion is possible only inside the sphere and the walls?
If this is the case, how can I invert the materials? Setting the inside of the sphere and the walls as solid material, and allowing diffusion in the space between the sphere and the walls?
at the moment, after importing the stl file, I got

superGeometry.rename( 0, 2, indicator );
superGeometry.rename( 2, 1, stlReader );
superGeometry.clean();

which I believe is setting the materials for the stl file.

Thanks for your help!

• This reply was modified 3 years, 5 months ago by jdm.

[jdm]

Dear Stephan,

Thanks for your quick answer.
Here is a screenshot of the simulation corresponding to the second case (inlet positioned on the left, at {-12.5,0,0}, and thus spreading inside the sphere instead of outside) and showing the velocity.

I re-used many pieces of code from the examples nozzle3d and aorta3d.
What could be done if the issue is coming from the indicator not matching the grid?

Thanks again for your time!

• This reply was modified 3 years, 5 months ago by jdm.
• This reply was modified 3 years, 5 months ago by jdm.
• This reply was modified 3 years, 5 months ago by jdm.

[Author]

Posts