[Danial.Khazaeipoul]

For anyone interested, if you encounter an issue where custom tasks added via addCustomTask<stage::> in SuperLattice are not executed, it’s because the SuperLattice<T, DESCRIPTOR>::collideAndStream() method only calls executeCustomTasks(PostStream()), meaning tasks scheduled for other stages are not automatically executed.

To resolve this, you need to explicitly invoke executeCustomTasks() for your specific stage. You can do this by modifying collideAndStream() to include the call for your stage or by calling it manually from anywhere appropriate in your application using a SuperLattice object. For example:

executeCustomTasks(PreCollide())

• This reply was modified 4 months, 1 week ago by Danial.Khazaeipoul.

[Danial.Khazaeipoul]

What do you mean by “their tasks”?

The order in which post processors are added (meaning the calls to addPostProcessor) does not matter for the execution. The execution is ordered first by stage and then by priority. Post processors (or “tasks”) that share the same stage and priority are executed in parallel, platform permitting.

I understand that, for instance, the freeSurfacePostProcessor3D includes five individual post-processors, each of which must define a priority and implement the apply method. I just wanted to confirm that my understanding of your previous answer was correct. In doing so, I used the terms task and post-processor interchangeably.

Thank you for your clear and detailed explanation.

[Danial.Khazaeipoul]

Just to confirm that I understood you clearly, the different post processors do not interleave their tasks based on the order they were added; rather, the tasks are globally ordered by their stage (here, PostStream) and then by their declared priority. Am I correct?

[Danial.Khazaeipoul]

Hi Adrian,

Thank you! I tested it and didn’t encounter any obvious issues. Nonetheless, I followed your suggestion and switched back to using separate fields for each variable. I have a question about how priority works in custom tasks to improve my understanding of OpenLB’s core. Consider the following scenario:

(1) freeSurfacePostProcessor3D schedules its custom tasks at the PostStream stage, consisting of five local stages with assigned priorities (e.g., 1, 2, 3, 4, 5).

(2) Another post-processor schedules its own custom tasks at the PostStream stage with different local stages and assigned priorities (e.g., 1, 2, 3, 4).

What would be the expected execution order of these tasks? How are priorities handled when multiple post-processors add tasks at the same stage?

[Danial.Khazaeipoul]

Hello Adrian,

When defining a global data structure in OpenLB, is it possible to use a custom type such as one given below with the TYPED_FIELD_BASE:

struct BubbleInfo {
  CellID bubbleID = CellID(0);
  double volume = 0.0;
};

struct BUBBLE_FIELD : public descriptors::TYPED_FIELD_BASE<BubbleInfo,1> {};

[Danial.Khazaeipoul]

Yes, exactly (this is of course quite ugly but it will work).

FYI, I implemented a cleaner method to replace this and now one can check whether a cell is a padding or not directly from the Cell interface. However, this required me to modify the src/core files to introduce a new function inside the BlockStructureD class that retrieves a local LatticR<D> from a linearized CellID.

FYI, I implemented a more streamlined method to replace this, allowing direct checks on whether a cell is padding via the Cell interface. To achieve this, I updated the src/core files to include a new function in the BlockStructureD class, which retrieves a local LatticeR<D> from a linearized CellID.

  /// Get D-dimensional LatticeR from 1D cell ID
  LatticeR<D> getLatticeR(CellID iCell) const
  {
    if constexpr (D == 3) {
      std::int32_t iX = iCell / _projection[0];
      std::int32_t remainder = iCell % _projection[0];
      std::int32_t iY = remainder / _projection[1];
      std::int32_t iZ = remainder % _projection[1];
      return LatticeR<D>(iX - _padding, iY - _padding, iZ - _padding);
    }
    else {
      std::int32_t iX = iCell / _projection[0];
      std::int32_t iY = iCell % _projection[0];
      return LatticeR<D>(iX - _padding, iY - _padding);
    }
  };

  /// Return whether iCell is valid
  bool isPadding(CellID iCell) const
  {
    return isPadding(getLatticeR(iCell));
  };

For more details, please refer to the “feature-CCL” branch in my repository, specifically commits 2ce2d4fc and b28e5305.

[Danial.Khazaeipoul]

Thank you for the assistance Adrian.

Right now the most straight forward way would be to mark the overlap cells with a dedicated field.

Are you suggesting that I define a new dedicated field, where for instance overlap cells are assigned a value of “1” and all other cells are assigned a value of “0”? This field could then be used to distinguish overlap cells from regular cells.

If you need more control on the post processor placement right now I can probably also extract you a patch.

That would be greatly appreciated, as it would help me gain a better understanding of the code. Additionally, it could provide me with more control over the process.

[Danial.Khazaeipoul]

Hello Adrian,

Can you confirm that the “apply” operators used in the post-processors operate only on the internal nodes? As I understand it, these operators cannot act directly on overlap nodes. The values of overlap nodes are updated only when communication is triggered using the communicate() method. However, it is still possible to access the values of an overlap node, such as when an internal node has an overlap node as its neighbor.

Is it possible to determine whether a neighbor of an internal node is an overlap node? I have found a temporary workaround to address my issue, as shown in the picture below. However, a cleaner and safer approach would be preferable as I need to avoid using the overlap nodes values until ids are finalized.

After the temporary fix is applied, as you can see, the connected regions on each block have their own unique ID.

[Danial.Khazaeipoul]

Dear Adrian,

I believe I’ve identified the issue with the CCL implementation when MPI is enabled. Thank you for your insights so far in working toward a robust solution. Since you’ve already reviewed the code, you’re familiar with how cell IDs are used to assign bubble IDs and to navigate between cells in each kernel to access their bubble IDs.

I’ll do my best to clearly explain the problem. The CCL algorithm assumes that cell IDs follow a consistent order in each direction within a block, which works fine for a single block. However, when MPI is enabled and the domain is divided into multiple blocks, this consistency breaks down for neighboring cells located in different blocks. As a result, the bubble ID assignment becomes incorrect for blocks other than Block 0. In the worst-case scenario, this can lead to illegal memory access, especially since blocks may have an unequal number of cells, see the below picture.

When the orange cell in Block #2 is being processed, it accesses the bubble ID of the red cell, as the red cell is a valid neighbor. However, since the red cell belongs to a different block, its cell ID does not follow the block-local order of cell IDs in Block #2. When the orange cell adopts the bubble ID of the red cell, it may incorrectly reference a cell within Block #2, as indicated by the blue arrow. This can result in either assigning an incorrect bubble ID to the orange cell or, in the worst-case scenario, causing illegal memory access. This happens because Block #2 may not contain a cell with the cell ID from Block #0.

Is it possible to defer the automatic communication between blocks for a specific global data structure until it is explicitly triggered when needed?

Regards,
Danial

• This reply was modified 7 months ago by Danial.Khazaeipoul.

[Danial.Khazaeipoul]

Hello Adrian,

Is my understanding correct regarding the use of a block-specific parameter passed to an operator? Suppose I have an offset vector, synchronized across all processors, which records the number of cells in each preceding block, shown below.

struct CELL_COUNT : public descriptors::TYPED_FIELD_BASE<std::size_t,1> { };
std::vector<std::size_t> offset(_sLattice.getCuboidGeometry().getNc(), 0);

The size of the offset vector matches the number of global cuboids. For a domain with 4 blocks, the resulting offset vector might look like {0, 431433, 855297, 1279161}, as an example. Given this setup, will the following method yield the expected results for the CELL_COUNT, as shown in the image?

for (int iC = 0; iC < _sLattice.getLoadBalancer().size(); ++iC)
{
  auto& block = _sLattice.getBlock(iC);
  block.template setParameter<FreeSurface::CELL_COUNT>(offset[_sLattice.getLoadBalancer().glob(iC)]);
}

Please note that the image is only for demonstration purposes. When an operator processes a cell, will calling the parameter use the correct value of CELL_COUNT based on the block in which the cell resides?

• This reply was modified 7 months, 2 weeks ago by Danial.Khazaeipoul.
• This reply was modified 7 months, 2 weeks ago by Danial.Khazaeipoul.
• This reply was modified 7 months, 2 weeks ago by Danial.Khazaeipoul.

[Danial.Khazaeipoul]

Using the cell interface, is there any method that can be used to identify the global cuboid number to which the current cell belongs?

[Danial.Khazaeipoul]

Just to confirm my understanding, by

if you use the maximum global block size

Are you referring to the fact that cuboids may not have an equal number of associated cells, and that using the maximum block size can prevent overlapping global cell IDs?

[Danial.Khazaeipoul]

Thank you, Mathias, for the explanation. The code I’ve implemented works well in a single partition setup where localCellID == globalCellID. However, in configurations with multiple partitions or blocks, localCellID != globalCellID, leading to non-unique IDs.

For local cell IDs, which are of type CellID or std::uint32_t, there’s an advantage in CCL algorithms as these IDs are unique and remain consistent throughout the simulation. Because of this, I’m looking for a way to generate global cell IDs with similar characteristics. For example, I am thinking about a method like the following which creates a unique global CellID for each cell:

std::uint32_t globalCellID = blockID × cellsPerBlock + localCellID

• This reply was modified 8 months, 2 weeks ago by Danial.Khazaeipoul.

[Danial.Khazaeipoul]

Dear Adrian,

I believe I’m encountering an issue with the cell IDs returned by the getCellId method when the domain is divided into separate partitions. Does this method return a cell ID that is local to a specific block?

After running a few tests, it appears that the method indeed returns a cell ID that is local within each block. Is there an existing method that can provide a global cell ID, unique to each cell across the entire domain, regardless of partitioning?

[Danial.Khazaeipoul]

FYI, I have modified getComponentPointer to be a public member rather than protected in the (1) core/columnVector.h, (2) core/vector.h, (3) core/fieldArrayD.h, and (4) core/platform/gpu/cuda/context.hh. With these modifications, the atomicMin function can now be used as below:

auto nextID = atomicMin(nextCell.template getFieldPointer<FreeSurface::TEST_ID>().getComponentPointer(0), nbrID);

With the mentioned modifications and the use of atomicMin operation, the code now compiles and runs correctly, just as it did with the non-atomic version. I can guess that you prefer not to modify the core/ in this way as it might introduce memory-safety issues with underlying data now exposed through public methods. However, I am not sure what would be a better alternative to be able to access raw pointers with in an apply operator without modifying the core/ as above.

• This reply was modified 8 months, 3 weeks ago by Danial.Khazaeipoul.

[Author]

Posts