Regarding Particle Checkpoints

[Rookie]

Dear OpenLB developers,

I want to ask about two questions regarding “check point”:

1.In bstep3d, I have added the following code segment, and after running it, output files like “bstep3d.checkpoint_rank7_size8.dat” are generated. However, when I execute “mpirun -np 8 ./bstep3d”, it still starts the simulation anew. Do I need to add any additional commands? I couldn’t find anything in the User Guide or the forum.

// Saves lattice data
if ( iT%( saveIter/2 )==0 && iT>0 ) {
clout << “Checkpointing the system at t=” << iT << std::endl;
sLattice.save( “bstep3d.checkpoint” );
// The data can be reloaded using
sLattice.load(“bstep3d.checkpoint”);
}

2.I have already implemented continuation calculations similar to bifurcation3d, where particle calculations are done after fluid calculations. I’m wondering if it’s possible to continue calculations for fluid-particle coupling. That is, after pausing the program, can fluid and particles continue simulation together? Or is there a way to continue calculations for particles?

Best regards,
Rookie

[jan]

Dear Rookie,

1.
The mentioned code should save and then immediately load the saved fluid data. However, that seems unnecessary, because you could just keep using the data that is already stored in the SuperLattice. Instead, you could load the solutions if they exist and then skip the fluid calculations:

if ( !( superLattice.load(  “bstep3d.checkpoint”  ) ) ) {
  // Calculate fluid solution

  // And save afterwards
  sLattice.save( “bstep3d.checkpoint” );
}
else {
  // Do something with the saved fluid data
}

The if condition is important here because it ensures that the fluid calculations are only performed if the checkpoint data doesn’t exist.
In your case, if you’d like to always load the fluid solution and then do the same calculations again from that point, for example to divide long simulations into shorter segments, this should work:

sLattice.load(“bstep3d.checkpoint”);
// Calculate fluid solution (main loop)
// And save afterwards
sLattice.save( “bstep3d.checkpoint” );
// Terminate program and restart

2.
Of course, you can also do the fluid calculation at the same time. To do this you just have to call superLattice.collideAndStream(); again (and maybe also setBoundaryValues( superLattice, converter, iT, fluidMaxPhysT, superGeometry ); if necessary). Note, however, that without back-coupling the fluid velocity doesn’t really change anymore, so it seems unnecessary to keep calculating it. Only if you add some dynamic boundary conditions it might be necessary.

Best regards,
Jan

• This reply was modified 2 weeks ago by jan.

[Rookie]

Dear Jan,

Thank you again for your response. Currently, my simulation involves first running the fluid simulation until it fully develops, and then introducing particles. I have already implemented the reverse coupling of particles to the fluid. Since the simulation time is quite long, I need to add checkpoints to prevent situations like unexpected shutdowns, ensuring that the simulation can resume from where it left off. Regarding your first question, I still haven’t found a solution. I did notice that simply uncommenting and running the checkpoint in bstep3d doesn’t successfully load and continue the simulation. I understand the importance of the if statement you emphasized, but I’ve been struggling to figure out how to incorporate it these past few days. I saw someone on the forum questioning the last time step called in the checkpoint file（https://www.openlb.net/forum/topic/checkpointing-how-to-update-it-when-loading-a-checkpoint-file/）, which raised my own doubts. Shouldn’t continuing the simulation after loading the checkpoint file mean continuing from this time step? Figuring out how to retrieve this time step seems crucial to solving the problem. I’m not sure if my modifications will achieve my goal.

int main( int argc, char* argv[] )
{

// === 1st Step: Initialization ===
olbInit( &argc, &argv );
singleton::directories().setOutputDir( “./tmp3/” );
OstreamManager clout( std::cout,”main” );
// display messages from every single mpi process
//clout.setMultiOutput(true);

UnitConverter<T,DESCRIPTOR> converter(
(T) 1./N, // physDeltaX: spacing between two lattice cells in __m__
(T) 1./(M*N), // physDeltaT: time step in __s__
(T) 1., // charPhysLength: reference length of simulation geometry
(T) 1., // charPhysVelocity: maximal/highest expected velocity during simulation in __m / s__
(T) 1./100., // physViscosity: physical kinematic viscosity in __m^2 / s__
(T) 1. // physDensity: physical density in __kg / m^3__
);

const int saveIter = converter.getLatticeTime( 1. );

// Prints the converter log as console output
converter.print();
// Writes the converter log in a file
converter.write(“bstep3d”);

// === 2nd Step: Prepare Geometry ===
Vector<T,3> extend( lx0, ly0, lz0 );
Vector<T,3> origin;
IndicatorCuboid3D<T> cuboid( extend, origin );

// Instantiation of a cuboidGeometry with weights
#ifdef PARALLEL_MODE_MPI
const int noOfCuboids = singleton::mpi().getSize();
#else
const int noOfCuboids = 7;
#endif
CuboidGeometry3D<T> cuboidGeometry( cuboid, converter.getConversionFactorLength(), noOfCuboids );

// Instantiation of a loadBalancer
HeuristicLoadBalancer<T> loadBalancer( cuboidGeometry );

// Instantiation of a superGeometry
SuperGeometry<T,3> superGeometry( cuboidGeometry, loadBalancer );

prepareGeometry( converter, superGeometry );

// === 3rd Step: Prepare Lattice ===
SuperLattice<T, DESCRIPTOR> sLattice( superGeometry );

//prepareLattice and set boundaryConditions
prepareLattice( converter, sLattice, superGeometry );

// === 4th Step: Main Loop with Timer ===
clout << “starting simulation…” << std::endl;
util::Timer<T> timer( converter.getLatticeTime( maxPhysT ), superGeometry.getStatistics().getNvoxel() );
timer.start();

// Set up persistent measuring functors for result extraction
SuperLatticePhysVelocity3D<T, DESCRIPTOR> velocity( sLattice, converter );
SuperEuklidNorm3D<T> normVel( velocity );

BlockReduction3D2D<T> planeReduction(
normVel,
Hyperplane3D<T>().centeredIn(cuboidGeometry.getMotherCuboid()).normalTo({0,0,1}),
600,
BlockDataSyncMode::ReduceOnly);
std::size_t iT = 0;
if (!(sLattice.load(“bstep3d.checkpoint”)))
{
for (; iT < converter.getLatticeTime( maxPhysT ); ++iT ) {

// === 5th Step: Definition of Initial and Boundary Conditions ===
setBoundaryValues( converter, sLattice, iT, superGeometry );

// === 6th Step: Collide and Stream Execution ===
sLattice.collideAndStream();

// === 7th Step: Computation and Output of the Results ===
getResults( sLattice, converter, planeReduction, iT, superGeometry, timer );
if ( iT%( saveIter/2 )==0 && iT>0 ) {
clout << “Checkpointing the system at t=” << iT << std::endl;
sLattice.save( “bstep3d.checkpoint” );
}
}
}
else
{
for (iT = LAST_TIME_STEP_IN_THE_CHECKPOINT_FILE + 1; iT < converter.getLatticeTime( maxPhysT ); ++iT ) {

// === 5th Step: Definition of Initial and Boundary Conditions ===
setBoundaryValues( converter, sLattice, iT, superGeometry );

// === 6th Step: Collide and Stream Execution ===
sLattice.collideAndStream();

// === 7th Step: Computation and Output of the Results ===
getResults( sLattice, converter, planeReduction, iT, superGeometry, timer );
if ( iT%( saveIter/2 )==0 && iT>0 ) {
clout << “Checkpointing the system at t=” << iT << std::endl;
sLattice.save( “bstep3d.checkpoint” );
}
}
}
timer.stop();
timer.printSummary();
}

Regarding the second issue, what I want to address is the saving and loading of SuperParticleSystem3D<T, PARTICLE> supParticleSystem(superGeometry);, because I’ve seen someone trying to implement the serialization of superGeometry（https://www.openlb.net/forum/topic/check-point-using-gpu/）, which is also achievable.I’ll be able to calculate the coupling of fluid and particles simultaneously when I stop the program and restart the computation.

Best regards,
Rookie

[jan]

Dear Rookie,

Loading the saved data will indeed restart the simulation at the first time step, but with the saved fluid data. If you want to continue from the stopped time step, I’d suggest to write this information additionally to another file and then read the time step from this file when loading the fluid data. Then use it to set iT accordingly. See [1] for an introduction to file input/output, if necessary.

Unfortunately, there is currently no implemented method for saving and loading the particle data. You could do it manually, for example by writing data (global particle ID, position, velocity, force, …) to files and loading them later, creating new particles and updating their data with the input from the file.

Best regards,
Jan

[1]: https://cplusplus.com/doc/tutorial/files/

[Rookie]

I followed your advice to output and read iT, and successfully implemented the continuation of the fluid simulation. I will try to write checkpoints for particles as well. Thank you for always providing me with helpful assistance. I hope the next version of OpenLB can improve this part of the functionality. Wish you all the best! I’ve placed the code below to help others solve similar problems in the future.
if (!(sLattice.load(“bstep3d.checkpoint”)))
{
for (; iT < converter.getLatticeTime( maxPhysT ); ++iT ) {

// === 5th Step: Definition of Initial and Boundary Conditions ===
setBoundaryValues( converter, sLattice, iT, superGeometry );

// === 6th Step: Collide and Stream Execution ===
sLattice.collideAndStream();

// === 7th Step: Computation and Output of the Results ===
getResults( sLattice, converter, planeReduction, iT, superGeometry, timer );
if ( iT%( saveIter/2 )==0 && iT>0 ) {
clout << “Checkpointing the system at t=” << iT << std::endl;
sLattice.save( “bstep3d.checkpoint” );
std::ofstream checkpointFile(“bstep3d_iT.txt”);
checkpointFile << iT;
checkpointFile.close();
}
}
}
else
{
std::ifstream checkpointFile(“bstep3d_iT.txt”);
if (checkpointFile.is_open()) {
checkpointFile >> iT;
checkpointFile.close();
} else {
iT = -1;
}
for (++iT; iT < converter.getLatticeTime( maxPhysT ); ++iT ) {

// === 5th Step: Definition of Initial and Boundary Conditions ===
setBoundaryValues( converter, sLattice, iT, superGeometry );

// === 6th Step: Collide and Stream Execution ===
sLattice.collideAndStream();

// === 7th Step: Computation and Output of the Results ===
getResults( sLattice, converter, planeReduction, iT, superGeometry, timer );
if ( iT%( saveIter/2 )==0 && iT>0 ) {
clout << “Checkpointing the system at t=” << iT << std::endl;
sLattice.save( “bstep3d.checkpoint” );
std::ofstream checkpointFile(“bstep3d_iT.txt”);
checkpointFile << iT;
checkpointFile.close();
}
}
}
timer.stop();
timer.printSummary();
}

[jan]

Dear Rookie,

I am glad to hear that it works now and thank you very much for providing the working source code.

Best regards,
Jan

[Author]

Posts