Manipulating populations with postprocessors on GPU
- This topic has 11 replies, 2 voices, and was last updated 2 weeks, 4 days ago by .
-
Posts
-
Dear OpenLB Community,
I have been working on creating a shear periodic (Lees-Edwards) boundary condition postprocessor in order to run homogeneous shear turbulence (HST) simulations, largely following the work of Peng, Wu, Wang, and Li (2023). I have been able to successfully run simulations on a single CPU core but I am running into issues running with MPI and on a single GPU.
Regarding parallelization, the shear periodic boundary condition requires retrieving cell data across the domain, outside of the block lattice of the cell being operated on. It makes sense that there would be problems here without addressing the communication of the cell data across blocks.
For using the postprocessor on a GPU, I am experiencing a different problem. There are several steps in the postprocessing algorithm that require obtaining and manipulating the populations at a given cell. For example, one step involves linearly interpolating the populations at a point in space between two cells with the following function:
template <typename CELL, typename V = typename CELL::value_t, typename DESCRIPTOR = typename CELL::descriptor_t> void interpolateDF(CELL& cell, Vector<V,DESCRIPTOR::d> distance, V f_tilde[DESCRIPTOR::q]) any_platform{ Vector<V,DESCRIPTOR::d> point1; for( int iD = 0; iD < DESCRIPTOR::d; iD++ ) { point1[iD] = util::floor(distance[iD]); } V q = distance[0] - point1[0]; Vector<V,DESCRIPTOR::d> point2 = point1; point2[0] += 1; auto f_x1 = cell.neighbor(point1); auto f_x2 = cell.neighbor(point2); for(int iPop = 0; iPop < DESCRIPTOR::q; iPop++ ) { f_tilde[iPop] = f_x1[iPop]*(V{1}-q) + f_x2[iPop]*q; } };The code will compile successfully for a single GPU, but running it results in a segmentation fault due to OpenLB deciding to use
olb::cpu::Cell<T,DESCRIPTOR>as the cell object. I suspect this is because I am using theoperator[]forCellwhich is not a part ofolb::gpu::cuda::Cell<T,DESCRIPTOR>. I’ve noticed thatolb::gpu::cuda::DataOnlyCell<T,DESCRIPTOR>does have the functionality of theoperator[], but I am unsure of how to utilize that. There is also a later step in the algorithm that involves setting the value of the populations at a neighboring cell which looks likecell.neighbor(-1 * normal)[iPop] = f_tilde[iPop] - f_eq_xm[iPop] + f_eq_xt[iPop];which I suspect has a similar problem as the above function.At the moment, I am more so interested in getting the postprocessor to work on a single GPU rather than parallelization. Any help on the GPU implementation would be greatly appreciated. I can include the entire postprocessor script if necessary.
Thank You,
jmctigheHow do you compile exactly? If the
cpu::Cellis used this is not running on GPU. Square bracket access to populations is definitely implemented for the GPU cells. The way the transparent execution works is that the operatorapplyis instantiated for the platform-specific cells. They all share the same outer functionality just implemented differently.The code you posted should work on GPU (assuming it was called correctly in the
apply(can you also show the code of this or ideally your entire post processor?)Right now I suspect that it is just a compilation issue.
Thanks for these additional details and good to know that the same problem occurs on CPU. Looking at your debug output I suspect that the issue is a out-of-bounds / undefined access to a neighbor cell (leading to the most likely invalid cell index of ~2e9, especially considering the index of the original cell). Looking at your interpolation function I can not exclude the possibility that very far / out of bounds neighbors are accessed. For performance reasons the index validity is not checked by default leading to the segfault we observe.
What is the max range of neighbors accessed from a cell? Did you consider that the overlap needs to be large enough to cover these accesses (if the post processor is executed at the outermost non-overlap cells). The issue might also be faulty initialization of your extent fields.
Side note: It seems to me that e.g.
DOMAIN_EXTENTcould be a parameter instead of a per-cell field. Using parameters where possible will help reduce your memory load.Did you confirm that the data is set up how you expect e.g. by viewing the fields in Paraview?
For the GPU case: Did you switch the processing context s.t. the data is synced to GPU before you execute the post processor?
Overlap communication is not designed to encompass the entire domain. You will need custom code to realize such a complex scheme. E.g. for our FSI integrals we implemented custom
PerBlockscoped post processors with CPU / GPU specific implementations.Can you explain what you want to implement in more detail? I do not quite get why you need to access cells at the opposite side of the domain (unless it is for periodicity which is a different question and doesn’t require the overlap to span the entire domain).
I see, this is interesting!
An efficient general implementation will require custom MPI communication I think, executing parts on CPU won’t change the fundamental problem of the data not being available where it needs to be.
For an intermediary step: How large of a simulation domain / how many cells do you currently plan to use this on? If you can e.g. be sure that this will fit on a single node or even single GPU (which can fit a lot of cells nowadays) the implementation will be much simpler.
- You must be logged in to reply to this topic.