Implementing the Cumulant Collision Operator

[jmctighe]

Dear OpenLB Community,

I am attempting to modify the channel3d example to make use of the cumulant collision operator, while maintaining the application of the channel forcing and LES turbulence modeling. When I set DESCRIPTOR and BulkDynamics to the following:

using DESCRIPTOR = D3Q27<tag::CUM,FORCE,TENSOR,VELOCITY,POROSITY,VELOCITY2,AVERAGE_VELOCITY>;
using BulkDynamics = CUMdynamics<T,DESCRIPTOR>;

and then later, when initializing the lattice, I set the bulk dynamics using

sLattice.defineDynamics<BulkDynamics>(superGeometry, 1);

I experience the following error during compilation:

In file included from ../../../src/dynamics/cumulantDynamics.h:28:
../../../src/dynamics/collisionCUM.h:313:19: error: static assertion failed due to requirement 'D2Q9<>::d == 3': Cumulant Dynamics only implemented in D3Q27
  313 |     static_assert(DESCRIPTOR::d==3 && DESCRIPTOR::q == 27, "Cumulant Dynamics only implemented in D3Q27");

Even though the descriptor is set to D3Q27, the code seems to think it is D2Q9. Could this be a potential bug? To look at a simplified example, without body forcing or LES, I also attempted to modify the cavity3d example in a similar way and received the same error.

Despite the encountered error are there any additional considerations I need to make when trying to apply the cumulant collision operator to the channel3d problem? Is the current implementation of the cumulant operator in OpenLB compatible with body forcing and LES turbulence modeling?

Thanks!

[FBukreev]

Hello,

the cumulant dynamics was only a prototype, which was tested only by taylor green vortex example. If you want to simulate the channel case it is better to use the standard HRR collision with wall model.

[jmctighe]

Hi FBukreev,

Thank you for the clarification. I was also wondering about the SuperLatticeTimeAveraged3D and SuperLatticeTimeAveragedCrossCorrelationF3D functors.

For the turbulent channel flow, I would like to perform first and second order statistics on the velocity field data over ~200,000 consecutive time steps with a grid that consists of ~25M voxels. I am using the “gpu_openmpi.mk” configuration with two NVIDIA H100 GPUs.

When I use the time averaged functors, the simulation becomes severely bottlenecked, since the functors are running on the CPU. Are there any work arounds for this? Is it possible to carry out these operations on the GPU using the current framework? Greatly appreciate your help!

-jmctighe

[FBukreev]

In the actual realease of openLB the averaging happens already on GPU and you dont need any separate functor.

[jmctighe]

Perhaps there is something wrong on my end then… When running the simulation with the same descriptor, collision operator, and wall-model parameters as in the provided channel3d example, the performance is about 390 MLUPs before starting the SuperLatticeTimeAveraged3D calculations (performed using .addEnsemble()). Once the calculations begin, the performance dramatically drops to under 2.5 MLUPs, the volatile GPU utilization goes to zero on average for both GPUs, and the CPU utilization goes to 100% for the two CPUs in use.

Could this be at all related to the compilation configuration? I can provide the contents of my config.mk file and the simulation setup if you would like.

• This reply was modified 4 months, 4 weeks ago by jmctighe. Reason: typo

[FBukreev]

In the last release these functors are not used in general, check the actual release.

[jmctighe]

The release is 1.8.1. I did notice that these functors are not used in any of the provided examples. Should I be using something different to collect the field statistics? I know the averages can be obtained with the AVERAGE_VELOCITY field, but I am particularly interested in the variance and covariance terms.

[FBukreev]

ok, for that we have not implemented a gpu-capable postprocessor. You can how the averaging is performed now and try to extend it with the required statistics evaluation.

[jmctighe]

OK, got it. Just so I know I am looking in the right place, the averaging is being performed using either the TrackAverageVelocity or StoreAndTrackAverageVelocity structs located in src/dynamics/collisionModifiers.h, correct? And then I should be able to write a postprocessor with a similar structure that operates on a new field, i.e. RMS_VELOCITY for rms data? but this post-processor does not have to be defined within the COLLISION namespace, or should I use the collision namespace to make use of parameters such as LATTICE_TIME? I have limited experience with C++, so this is all a bit of a learning process for me.

Thanks again for all your help.

[FBukreev]

yes, you understand correctly. LATTICE_TIME is already existing, you can define RMS_VELOCITY in the descriptors/fields.h

[jmctighe]

FBukreev,

I’ve written the following postprocessor for computing the full covariance matrix of velocity, making use of the already existing fields LATTICE_TIME and AVERAGE_VELOCITY:

struct TrackCovarVelocity {
  static constexpr OperatorScope scope = OperatorScope::PerCellWithParameters;

  int getPriority() const {
    return std::numeric_limits<int>::max();
  }
 
  using parameters = meta::list<descriptors::LATTICE_TIME>;

  template <typename CELL,typename PARAMETERS>
  void apply(CELL& cell, PARAMETERS& parameters) any_platform {
    using V = typename CELL::value_t;
    using DESCRIPTOR = typename CELL::descriptor_t;

    int iT = parameters.template get<descriptors::LATTICE_TIME>();
    auto uCovar = cell.template getField<COVARIANCE_VELOCITY>();
    if (iT > 1) { 
      auto uAvg = cell.template getField<AVERAGE_VELOCITY>();

      V u[DESCRIPTOR::d] {};
      cell.computeU(u);
      
      int k;
      for (int i=0; i < DESCRIPTOR::d; i++){
        for (int j = 0; j < DESCRIPTOR::d; j++){
          k = j + i * DESCRIPTOR::d;
         uCovar[k] = (uCovar[k]*(iT - 1) + (u[i] - uAvg[i]) * (u[j] - uAvg[j])*(iT/(iT-1))) / iT;
        }
      }     
    }
    else {
      for(int i=0; i<(DESCRIPTOR::d * DESCRIPTOR::d); i++){
        uCovar[i] = 0.;
      }
    }
    cell.template setField<COVARIANCE_VELOCITY>(uCovar);

  }
  
  
};

The postprocessor is working as expected. Thank you again for all your help!

Best,

jmctighe

[Author]

Posts