[Adrian]

Sure, fur functions that are not yet converted to using FunctorPtr arguments such as rename you can dereference the shared_ptr using the “*” operator:

`
superGeometry.rename(4,1,*unified_box);
`

[Adrian]

This is an oversight in the documentation. Functor and indicator arithmetic use std::shared_ptr since version 1.3. See section 10.3.2 of the userguide for an introduction.

Basically you can write any indicator as

`
std::shared_ptr<IndicatorF2D<T>> box1 = std::make_shared<IndicatorCuboid2D<T>>(extend1, origin1);
std::shared_ptr<IndicatorF2D<T>> box2 = std::make_shared<IndicatorCuboid2D<T>>(extend2, origin2);
auto unified_box = box1 + box2;
`

and so on. This also works for functors.

[Adrian]

Seems to be an interesting application! Thank you for expanding on it. Feel free to contact me if you want to talk about possibilities for collaboration in the integration between your application and OpenLB.

The scaling plot doesn’t look good for OpenMP at all. I’ll take a closer look on my own system.

Did you change anything wrt. the OpenMP configuration you posted previously?

[Adrian]

OpenMP taking three times longer should not happen in any case, probably the thread configuration is faulty there. I am interested in any update on this.

As for the compilation error: There shouldn’t be angle brackets there. You can try to replace them with quotation marks.

Better support for using OpenLB in separate applications outside of its folder are on our todo list. What is your setup there? Do you use OpenLB’s precompiled mode for the shared library?

[Adrian]

Ok, some things you could try:

– Set OMP_NUM_THREADS to 72 to match your physical core count
– Set OMP_PROC_BIND=close and OMP_PLACES=cores
– Try pinning the threads to the cores using likwid-pin [1]

What kind of performance are you observing when using MPI? (Should be reported in the CLI output as MLUPs)

I suspect you are using release 1.3? Our latest release [2] which was published shortly after your initial post changes the entire core data structure and the way propagation is performed, it might be worth to try if this improves your performance results. I have not performed detailed benchmarks using only OpenMP with this latest version but the implementation in this area was simplified significantly.

[1]: https://github.com/RRZE-HPC/likwid/wiki/Likwid-Pin
[2]: https://www.openlb.net/news/openlb-release-1-4-available-for-download/

[Adrian]

Some questions on the OpenMP issue:

– Did you use the OMP or the HYBRID compilation mode? (I assume OMP?)
– Which if any external environment variables were set up for OpenMP? Especially concerning thread pinning?
– This is probably a dual socket system? If so the results might improve when using HYBRID mode with one MPI process per socket and OpenMP only socket-local [1].

As a sidenote: Using more OpenMP threads than there are physical cores didn’t improve LBM performance on any system / software that I have tested.

[1]: Both binding MPI processes to the sockets and OpenMP threads to the cores

[Adrian]

Happy to help.

As a sidenote: Your solution is basically the same as what is done in my patch – *(channel-step) is just a shorthand for wrapping the subtracted indicators in an identity.

[Adrian]

Thanks for your detailed report!

We are aware of this problem and it will be fixed in our next release.
See the following patch for a solution:

`
Date: Fri, 5 Jun 2020 16:54:38 +0200
Subject: [PATCH] Fix bstep2d material number setup

_Implicit_ geometry modelling by approximating the step shape using the
cuboids themselves only works if we have enought cuboids to start with.

The material numbers were set correctly in e.g. olb 1.2. It probably got
lost somewhere along the introduction of managed functor arithmetic.
—
examples/laminar/bstep2d/bstep2d.cpp | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)

diff –git a/examples/laminar/bstep2d/bstep2d.cpp b/examples/laminar/bstep2d/bstep2d.cpp
index f6359164e..83faa3093 100644
— a/examples/laminar/bstep2d/bstep2d.cpp
+++ b/examples/laminar/bstep2d/bstep2d.cpp
@@ -85,7 +85,7 @@ SuperGeometry2D<T> prepareGeometry( UnitConverter<T,DESCRIPTOR> const& converter
SuperGeometry2D<T> superGeometry( *cuboidGeometry, *loadBalancer, 2 );

// material numbers from zero to 2 inside geometry defined by indicator
– superGeometry.rename( 0,2 );
+ superGeometry.rename( 0,2, *(channel-step) );
superGeometry.rename( 2,1,1,1 );

Vector<T,2> extendBC( 0,ly0 );
—
2.25.4
`

It works for larger cuboid numbers as the block decomposition can then approximate the step geometry as the unification of multiple cuboids. For single cuboid setups we have to use material numbers.

[Adrian]

I suspect that you did not change the CXX variable to icpc -D__aligned__=ignored. This would explain why you still get g++ errors. For reference see the contents of a config.mk file modified to use ICC:

#CXX := g++
CXX := icpc -D__aligned__=ignored
#CXX := mpiCC
#CXX := mpic++

CC := icc # necessary for zlib, for Intel use icc

#OPTIM := -O3 -Wall -march=native -mtune=native # for gcc
OPTIM := -O3 -Wall -xHost # for Intel compiler
#OPTIM := -O3 -Wall -xHost -ipo # optional for Intel compiler
DEBUG := -g -Wall -DOLB_DEBUG

CXXFLAGS := $(OPTIM)
#CXXFLAGS := $(DEBUG)

# compilation requires support for C++14
# works in:
# * gcc 5 or later (https://gcc.gnu.org/projects/cxx-status.html#cxx14)
# * icc 17.0 or later (https://software.intel.com/en-us/articles/c14-features-supported-by-intel-c-compiler)
# * clang 3.4 or later (https://clang.llvm.org/cxx_status.html#cxx14)
CXXFLAGS += -std=c++14

#ARPRG := ar
ARPRG := xiar # mandatory for intel compiler

LDFLAGS :=

PARALLEL_MODE := OFF
#PARALLEL_MODE := MPI
#PARALLEL_MODE := OMP
#PARALLEL_MODE := HYBRID

MPIFLAGS :=
OMPFLAGS := -fopenmp

BUILDTYPE := precompiled
#BUILDTYPE := generic

I hope that this will solve the problem.

[Adrian]

Your initial post in this thread indicated that you get a runtime exception upon executing your program. i.e. the problem in this case is not the build system.

On the topic of build systems: OpenLB uses a plain Makefile for building both the library and any applications / examples and we can not support any custom setup on your side, especially considering that we do not know how exactly you configured your CMakeLists.txt file. Any IDE that supports CMake should have no problem coping with plain Makefiles.

On the topic of your initial problem: Sthavishtha already replied that the problem seems to be where you place the STL file / from which working directory you execute the program. I agree with this assessment. Did you try to switch to the cylinder3d example directory in a terminal and manually execute the executable from there?

[Adrian]

Functors such as BlockReduction2D2D are function objects, i.e. they implement C++’s function call operator operator().

You can look up documentation for BlockReduction2D2D and any other of OpenLB’s classes using DoxyGen (Documentation -> Developer Guide in the navigation, I seem to be unable to post links here). There one can see that for this particular class the function call operator is inherited from the BlockDataF2D functor.

Your guess at how your goal can be accomplished in 3D is correct. You can use BlockReduction3D2D to reduce a given super functor on arbitrary hyperplanes.

[Adrian]

Indeed, I mistook BlockData2D for the functor BlockDataF2D when looking up the available methods for BlockReduction2D2D.

You can use the operator:

SuperLatticePhysVelocity2D<T,DESCRIPTOR> velocityF(sLattice, converter);
BlockReduction2D2D<T> velocityPlane(velocityF, 600, BlockDataSyncMode::ReduceOnly);
int latticeR[2] { };
for (latticeR[0]=0; latticeR[0] < velocityPlane.getBlockStructure().getNx(); ++latticeR[0]) {
  for (latticeR[1]=0; latticeR[1] < velocityPlane.getBlockStructure().getNy(); ++latticeR[1]) {
    T vel[2] { };
    velocityPlane(vel, latticeR);
    myfile << latticeR[0] << " " << latticeR[1] << " "
           << std::setprecision(9)
           << vel[0] << " "
           << vel[1] << std::endl;  
  }
}

[Adrian]

One straight forward way of extracting such data from a super lattice into a single block is to use BlockReduction2D2D.

This is actually used in laminar/cavity2d/parallel and most other examples for extracting image data in getResults.

Something along these lines should do what you want:

  SuperLatticePhysVelocity2D<T,DESCRIPTOR> velocity(sLattice, converter);
  BlockReduction2D2D<T> velocityPlane(velocityF, 600, BlockDataSyncMode::ReduceOnly);
  for (int iX=0; iX < velocityPlane.getNx(); ++iX) {
    for (int iY=0; iY < velocityPlane.getNy(); ++iY) {
      myfile << iX << " " << iY << " "
             << std::setprecision(9)
             << velocityPlane.get(iX,iY,0) << " "
             << velocityPlane.get(iX,iY,1) << std::endl;  
    }
  }

[Adrian]

This is the patch that will be included in the next release:

diff –git a/src/functors/lattice/indicator/blockIndicatorF2D.hh b/src/functors/lattice/indicator/blockIndicatorF2D.hh
index 8d01a6b03c8cea37d1d9ec05033d1bda4a616efa..f52a48a3ec4b2b966e6341e186d253fe73bc453c 100644
— a/src/functors/lattice/indicator/blockIndicatorF2D.hh
+++ b/src/functors/lattice/indicator/blockIndicatorF2D.hh
@@ -124,7 +124,7 @@ bool BlockIndicatorMaterial2D<T>::operator() (bool output[], const int input[])
// of BlockGeometry2D<T>::get to avoid resetting geometry
// statistics:
const BlockGeometryStructure2D<T>& blockGeometry = this->_blockGeometryStructure;
– const int current = blockGeometry.get(input[0], input[1]);
+ const int current = blockGeometry.getMaterial(input[0], input[1]);
output[0] = std::any_of(_materials.cbegin(),
_materials.cend(),
[current](int material) { return current == material; });
diff –git a/src/functors/lattice/indicator/blockIndicatorF3D.hh b/src/functors/lattice/indicator/blockIndicatorF3D.hh
index dcf2f5fc0e7658c487d671a02322eea51669378f..60c00322c4f79dacf7cf88ab0687ee746f65bfb8 100644
— a/src/functors/lattice/indicator/blockIndicatorF3D.hh
+++ b/src/functors/lattice/indicator/blockIndicatorF3D.hh
@@ -136,7 +136,7 @@ bool BlockIndicatorMaterial3D<T>::operator() (bool output[], const int input[])
// of BlockGeometry3D<T>::get to avoid resetting geometry
// statistics:
const BlockGeometryStructure3D<T>& blockGeometry = this->_blockGeometryStructure;
– const int current = blockGeometry.get(input[0], input[1], input[2]);
+ const int current = blockGeometry.getMaterial(input[0], input[1], input[2]);
output[0] = std::any_of(_materials.cbegin(),
_materials.cend(),
[current](int material) { return current == material; });

i.e. change BlockGeometry(2,3)D::get calls to BlockGeometry(2,3)D::getMaterial for the block indicators.

[Author]

Posts