multi-block approach

[lbmcoding]

Hi developers,
I find that the problem are all solved on a homogeneous lattice at present. Is there a multi-block approach in the OpenLB? Thans very much.

[mathias]

We have a multi-block approach which is called superLattice. It holds several blockLattices of the same resolution.

Best
Mathias

[hungryfish]

What is the reason for if the Multi-block inspector in Paraview only shows me the hierarchy for Block 0? The file (data set) which was loaded is tmp/vtkData/name.pvd.

[jan]

Could you provide more information on your problem? Which example do you consider, how does your config.mk look like, etc?

My first guess would be that you’re only setting one cuboid in your simulation setup.

[hungryfish]

Sorry, its only preliminary:
I modified the code borrowed by ‘cavity3d’ which is belonging to the set of GPU-enabled examples in OpenLB 1.5 (page 141). If I ran the code on a CPU-cluster, the expected multi-block dataset (e.g. 0-125) is shown as expected in paraview, else, on GPU the block hierarchy only consists of block 0.

[Adrian]

Ok – as the VTK Output logic is independent of the specific platform the block lattices are processed on, I suspect that there is an issue with how you enabled and use the GPU support.

Did you compile using both the GPU_CUDA platform and MPI? (e.g. following config/gpu_openmpi.mk)

How did you launch the code on the cluster? Each GPU needs to be assigned to one process, one example is provided in config/gpu_openmpi.mk. However the way to do this is quite specific to each cluster.

(I suspect that the results of the single GPU block 0 encompass the entire lid driven cavity? This would basically guarantee that the issue is missing MPI support / wrong startup)

[hungryfish]

The run itself on a single NVIDIA A100 works fine.
Compilation for ‘gpu_only’, ! without MPI !, CUDA_ARCH := 80, PARALLEL_MODE := NONE

prepareGeometry] Prepare Geometry … OK
[prepareLattice] Prepare Lattice …
[prepareLattice] Prepare Lattice … OK
[main] starting simulation…

[Timer] step=7755; percent=1.10059; passedTime=458.053; remTime=41160.6; MLUPs=3508.08
[LatticeStatistics] step=7755; t=1.10059; uMax=1.67154; avEnergy=-nan; avRho=-nan

Wed Jun 29 17:04:26 2022
+—————————————————————————–+
| NVIDIA-SMI 515.48.07 Driver Version: 515.48.07 CUDA Version: 11.7 |
|——————————-+———————-+———————-+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 NVIDIA A100-SXM… Off | 00000000:0E:00.0 Off | 0 |
| N/A 42C P0 240W / 400W | 64895MiB / 81920MiB | 100% Default |
| | | Disabled |
+——————————-+———————-+———————-+
….
+—————————————————————————–+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
| 0 N/A N/A 23303 C ./sphereFlow 64893MiB |
+—————————————————————————–+

You said ‘the VTK Output logic is independent of the specific platform’ so I will try to search further ..

[Adrian]

Good to hear that the single GPU run works (although I am worried about the NaN values for average energy and density in the log – this should not happen so there may be a separate issue)

The reason for only ever getting a single block with this config is given by PARALLEL_MODE := NONE. i.e. MPI is not enabled. The same way that MPI is required for multi-CPU / node execution it is also required for multi-GPU execution.

[hungryfish]

‘The reason for only ever getting a single block with this config is given by PARALLEL_MODE := NONE. i.e. MPI is not enabled.’

This basically means that the GPU-only mode is useless for my application. The multi-GPU approach via MPI is then to be aimed for only.

PS
I could clarify the cause of the NaN values. The example is adapted from ‘cylinder3d’ ( correction above)

[Adrian]

This basically means that the GPU-only mode is useless for my application. The multi-GPU approach via MPI is then to be aimed for only.

It depends: If you want to distribute the simulation over multiple GPUs then yes, GPU-mode without parallel mode MPI can’t be used. However if you have e.g. a workstation with a single GPU you can still use multiple blocks (simply set the block count in the cuboid decomposition as normally) without requiring MPI.

As OpenLB applications that are compiled both with GPU support and parallel mode MPI can also be executed as a single process on a single GPU there is nothing lost by always including MPI and GPU support.

I’ll check the results for cylinder3d on a my systems to confirm that the statistics are calculated correctly there and come back to you.

[Author]

Posts