Compiling OpenLB in parallel and running it with a set number of threads

[d3rioe]

New user of OpenLB here. I’ve successfully installed and ran some of the included examples that are included in the source code, compiled using OpenMPI and ran them in parallel. The problem I have is that I would like to choose how many threads to use when I compile and run it, but it always runs on all of the processors for my computer.

This is pretty obvious since the very first line of output after starting a simulation is:

[OmpManager] Sucessfully initialized, numThreads=32

Since I have a 32 core processor. However since I’m also running other simulations on the computer as well, I’d like to limit the number of threads to something smaller.

The problem is that I can’t find anywhere where I can do that. config.mk in the head directory of course has the lines where you choose serial, MPI, or OpenMPI:

#PARALLEL_MODE   := OFF
#PARALLEL_MODE   := MPI
PARALLEL_MODE   := OMP

But I can’t find anything to set the number of threads to a number of my choosing or anything like that.

[Julius]

Hello d3rioe,

I think in the user manual page 38, “2.11 Lesson 11: Run Your Programs on a Parallel Machine” has your answer.
“To run the program in parallel, use the command mpirun -np 2 ./cavity2d. Here -np 2 specifies the number of processors to be used”
Use the flag --use-hwthread-cpus to enable threads in addition to individual cores.

Regards, Julius

[d3rioe]

Thanks for the reply. I should have mentioned that I was using mpirun. For example I just ran bstep2d with the following command:

$ /usr/lib64/openmpi/bin/mpirun -np 4 --use-hwthread-cpus ./bstep2d

However the first line of output is:

[OmpManager] Sucessfully initialized, numThreads=32

Similarly when I ran it without the –use-hwthread-cpus option:

$ /usr/lib64/openmpi/bin/mpirun -np 4 ./bstep2d
[OmpManager] Sucessfully initialized, numThreads=32

I’m pretty sure that Openmpi is fine, I use it for running the other simulations that are running on my system without any issues.

• This reply was modified 4 years, 5 months ago by d3rioe.

[Julius]

Hello d3rioe,

In my config.mk I have PARALLEL_MODE := MPI and CXX := mpic++
Running that with the mpirun command allows me to change the number of threads.
I also have as a first message: [MpiManager] Sucessfully initialized, numThreads=15 for me. Maybe the manager is an issue?

Regards, Julius

[d3rioe]

OK, thanks for pointing that out. I had missed the CXX := mpic++ option instead of the default CXX := g++. After recompiling it with that option, running it with mpirun -np 4 ./bstep2d ran on 4 cores like it was supposed to.

Thanks for the help!

d3rioe

• This reply was modified 4 years, 5 months ago by d3rioe.

[Author]

Posts