[achodankar]

Hello Adrian,

I retracted back to the original code for the number of cuboids, however, the number of cuboids is set to one after running the code.

#ifdef PARALLEL_MODE_MPI
const int noOfCuboids = 2*singleton::mpi().getSize();
#else // ifdef PARALLEL_MODE_MPI
const int noOfCuboids = 1;
#endif // ifdef PARALLEL_MODE_MPI

Also, the command #SBATCH –tasks-per-node=8 was added to the SLURM script. It didn’t make any difference.

I observe that the issue stems from the mpi process. There is something not syncing between the mpi process or ranks with the Slurm script. I would appreciate your feedback.

Thank you.

Yours sincerely,

Abhijeet C.

[achodankar]

Hello Adrian,

How exactly did you launch the application and how did you assign each process a single GPU?
1) I ran the make file.
2) Ran the slurm script

Did I miss on anything?

[achodankar]

Hello Adrian,
I did use the config/gpu_openmpi.mk example config.

I used this SLURM script:

#!/bin/bash
#SBATCH –job-name=run1
#SBATCH –output=run1.out
#SBATCH –mail-type=ALL
#SBATCH –partition=gpu
#SBATCH –nodes=1
#SBATCH –gpus-per-node=a100:8
#SBATCH –mem=50gb
#SBATCH –time=5-00:00:00
#SBATCH –get-user-env

CUDA_VISIBLE_DEVICES_SETTING=(“0” “0” “0,1” “0,1,2” “0,1,2,3” “0,1,2,3,4” “0,1,2,3,4,5” “0,1,2,3,4,5,6” “0,1,2,3,4,5,6,7” “0,1,2,3,4,5,6,7,8” “0” )

srun –mpi=pmix_v3 bash -c export CUDA_VISIBLE_DEVICES=${OMPI_COMM_WORLD_LOCAL_RANK}; ./poiseuille3d

————————————————————————————————-

The mpirun command doesn’t work on the cluster for me. I also tried by setting the no of cuboids to 8. Also, I tried using the cuda_visible device settings. Also, I used this line srun –mpi=pmix_v3 export env CUDA_VISIBLE_DEVICES=${CUDA_VISIBLE_DEVICES_SETTING[$gpus-per-node]}; ./poiseuille3d instead of the previous one. However, none of the cases worked out for me.

I was adviced to use cudasetdevice command with the rank. How should I implement it? Is it the right approach to fix this issue?

I would really appreciate your help to resolve this issue.

Thank you.

Yours sincerely,

Abhijeet C.

[achodankar]

Hello Adrian,
I get this error:
poiseuille3d.cpp:468:32: error: use of deleted function ‘olb::SuperF3D<double, double>::SuperF3D(const olb::SuperF3D<double, double>&)’
468 | olb::SuperF3D<T,T> f(velocity);
| ^
In file included from ../../../src/core/superLattice.hh:37,
from ../../../src/boundary/setLocalVelocityBoundary3D.h:34,
from ../../../src/boundary/boundary3D.h:36,
from ../../../src/olb3D.h:2,
from poiseuille3d.cpp:31:
../../../src/functors/lattice/superBaseF3D.h:56:7: note: ‘olb::SuperF3D<double, double>::SuperF3D(const olb::SuperF3D<double, double>&)’ is implicitly deleted because the default definition would be ill-formed:
56 | class SuperF3D : public GenericF<W,int> {
| ^~~~~~~~
../../../src/functors/lattice/superBaseF3D.h:56:7: error: use of deleted function ‘olb::GenericF<T, S>::GenericF(const olb::GenericF<T, S>&) [with T = double; S = int]’
In file included from ../../../src/functors/analytical/indicator/indicatorBaseF3D.h:31,
from ../../../src/functors/analytical/indicator/indicatorF3D.h:28,
from ../../../src/geometry/cuboid3D.h:37,
from ../../../src/geometry/cuboidGeometry3D.h:39,
from ../../../src/communication/superStructure.h:31,
from ../../../src/core/superData.h:37,
from ../../../src/core/core.h:28,
from ../../../src/core/core3D.h:28,
from ../../../src/olb3D.h:1,
from poiseuille3d.cpp:31:
../../../src/functors/genericF.h:61:3: note: declared here
61 | GenericF(const GenericF&) = delete;
| ^~~~~~~~
make: *** [../../../default.mk:31: poiseuille3d.o] Error 1

Thank you.

Yours sincerely,

Abhijeet C

[achodankar]

Hello Adrian,
This explanation helps a lot. Thank you. I really appreciate your help.

Thank you.

Yours sincerely,

Abhijeet C.

[achodankar]

Hello Adrian,
Here is the error:

mpic++ -DPARALLEL_MODE_MPI -O3 -Wall -march=native -mtune=native -std=c++14 -DTIXML_USE_STL -I./src -I./src/ -I./src/external -I./src/external/zlib -c src/communication/loadBalancer.cpp -o buil
d/generic/obj/src/communication/loadBalancer.o
In file included from /home/applications/spack/applications/gcc-4.8.5/gcc-8.2.0-sxbf4jq6ghmoybsjlpqz2dm2qbbxzfyn/include/c++/8.2.0/x86_64-pc-linux-gnu/bits/c++config.h:508,
from /home/applications/spack/applications/gcc-4.8.5/gcc-8.2.0-sxbf4jq6ghmoybsjlpqz2dm2qbbxzfyn/include/c++/8.2.0/bits/stl_algobase.h:59,
from /home/applications/spack/applications/gcc-4.8.5/gcc-8.2.0-sxbf4jq6ghmoybsjlpqz2dm2qbbxzfyn/include/c++/8.2.0/vector:60,
from ./src/communication/loadBalancer.h:28,
from src/communication/loadBalancer.cpp:25:
/home/applications/spack/applications/gcc-4.8.5/gcc-8.2.0-sxbf4jq6ghmoybsjlpqz2dm2qbbxzfyn/include/c++/8.2.0/x86_64-pc-linux-gnu/bits/os_defines.h:39:10: fatal error: features.h: No such file or director
y
#include <features.h>
^~~~~~~~~~~~
compilation terminated.
make[2]: *** [build/generic/obj/src/communication/loadBalancer.o] Error 1

Thank you.

Yours sincerely,

Abhijeet C.

[achodankar]

Hello Adrian,
Thank you for your prompt response and explanation. You mentioned earlier that we need to assign each cuboid per process. So, Will the number of cuboids be equivalent to the number of the processes(i.e tasks) defined in the SLURM script? When you say that it is problem dependent, does it mean that it is geometry dependent? If the geometry is complex, do we need to define more number of cuboids? Did the group conduct any testing to define the range of the number of cuboids to be used in regard to the geometry?

Thank you.

Yours sincerely,

Abhijeet C.

[achodankar]

Hello Adrian,
Openlb runs properly when I run it on the login server. I use the make and mpirun commands, and it spits out the results. However, when I run the batch script, it starts giving me errors related to loadbalancer. If I include make cleanbuild in the script, then the error changes accordingly.

I am unable to post the batch script since the forum is blocking me, when I try to post it.

Thank you.

Abhijeet C.

[achodankar]

Hello Adrian,
Thank you very much for your prompt response. This is really helpful.

Thank you.

Yours sincerely,

Abhijeet C.

[achodankar]

Thank you Adrian!!!

[achodankar]

Hello Adrian,
I have set the entry nodes(3) for first cross-section and exit nodes(4) for last cross-section, while, the rest of others are fluid nodes(1). There are no wall nodes. The problem just got resolved now. It came to my notice that I forgot to set the periodicity directions to true or false, and missing that caused the error. I fail to understand how it is related to the periodicity conditions. I would really appreciate if you could explain the relation to it.

Thank you.

Yours sincerely,

Abhijeet C.

[achodankar]

Hello Developers,
The error shows up when I try to set the velocity and density values for the entry and exit nodes

sLattice.defineU( superGeometry, 4, uF1);

sLattice.defineRho( superGeometry, 3, rhoF1);

Thank you.

Yours sincerely,

Abhijeet C.

[achodankar]

Hello Developers,
I realized that the geometry setup is perfect. However, the above error appears during lattice initialization. What are the possible ways to fix this issue?

I would really appreciate your help.

Thank you.

Yours sincerely,

Abhijeet C.

[achodankar]

Hello Mathias,
Thank you. I will look into the code and implement it.

Yours sincerely,

Abhijeet C.

[achodankar]

Hello Mathias,
Thank you very much. I will try out those examples. I really appreciate your help.

Thank you.

Yours sincerely,

Abhijeet C.

[Author]

Posts