Skip to content

Regarding Particle Checkpoints

OpenLB – Open Source Lattice Boltzmann Code Forums on OpenLB General Topics Regarding Particle Checkpoints

Viewing 6 posts - 1 through 6 (of 6 total)
  • Author
    Posts
  • #8455
    Rookie
    Participant

    Dear OpenLB developers,

    I want to ask about two questions regarding “check point”:

    1.In bstep3d, I have added the following code segment, and after running it, output files like “bstep3d.checkpoint_rank7_size8.dat” are generated. However, when I execute “mpirun -np 8 ./bstep3d”, it still starts the simulation anew. Do I need to add any additional commands? I couldn’t find anything in the User Guide or the forum.

    // Saves lattice data
    if ( iT%( saveIter/2 )==0 && iT>0 ) {
    clout << “Checkpointing the system at t=” << iT << std::endl;
    sLattice.save( “bstep3d.checkpoint” );
    // The data can be reloaded using
    sLattice.load(“bstep3d.checkpoint”);
    }

    2.I have already implemented continuation calculations similar to bifurcation3d, where particle calculations are done after fluid calculations. I’m wondering if it’s possible to continue calculations for fluid-particle coupling. That is, after pausing the program, can fluid and particles continue simulation together? Or is there a way to continue calculations for particles?

    Best regards,
    Rookie

    #8466
    jan
    Participant

    Dear Rookie,

    1.
    The mentioned code should save and then immediately load the saved fluid data. However, that seems unnecessary, because you could just keep using the data that is already stored in the SuperLattice. Instead, you could load the solutions if they exist and then skip the fluid calculations:

    
    if ( !( superLattice.load(  “bstep3d.checkpoint”  ) ) ) {
      // Calculate fluid solution
    
      // And save afterwards
      sLattice.save( “bstep3d.checkpoint” );
    }
    else {
      // Do something with the saved fluid data
    }
    

    The if condition is important here because it ensures that the fluid calculations are only performed if the checkpoint data doesn’t exist.
    In your case, if you’d like to always load the fluid solution and then do the same calculations again from that point, for example to divide long simulations into shorter segments, this should work:

    
    sLattice.load(“bstep3d.checkpoint”);
    // Calculate fluid solution (main loop)
    // And save afterwards
    sLattice.save( “bstep3d.checkpoint” );
    // Terminate program and restart
    

    2.
    Of course, you can also do the fluid calculation at the same time. To do this you just have to call superLattice.collideAndStream(); again (and maybe also setBoundaryValues( superLattice, converter, iT, fluidMaxPhysT, superGeometry ); if necessary). Note, however, that without back-coupling the fluid velocity doesn’t really change anymore, so it seems unnecessary to keep calculating it. Only if you add some dynamic boundary conditions it might be necessary.

    Best regards,
    Jan

    • This reply was modified 1 month ago by jan.
    #8471
    Rookie
    Participant

    Dear Jan,

    Thank you again for your response. Currently, my simulation involves first running the fluid simulation until it fully develops, and then introducing particles. I have already implemented the reverse coupling of particles to the fluid. Since the simulation time is quite long, I need to add checkpoints to prevent situations like unexpected shutdowns, ensuring that the simulation can resume from where it left off. Regarding your first question, I still haven’t found a solution. I did notice that simply uncommenting and running the checkpoint in bstep3d doesn’t successfully load and continue the simulation. I understand the importance of the if statement you emphasized, but I’ve been struggling to figure out how to incorporate it these past few days. I saw someone on the forum questioning the last time step called in the checkpoint file(https://www.openlb.net/forum/topic/checkpointing-how-to-update-it-when-loading-a-checkpoint-file/), which raised my own doubts. Shouldn’t continuing the simulation after loading the checkpoint file mean continuing from this time step? Figuring out how to retrieve this time step seems crucial to solving the problem. I’m not sure if my modifications will achieve my goal.

    int main( int argc, char* argv[] )
    {

    // === 1st Step: Initialization ===
    olbInit( &argc, &argv );
    singleton::directories().setOutputDir( “./tmp3/” );
    OstreamManager clout( std::cout,”main” );
    // display messages from every single mpi process
    //clout.setMultiOutput(true);

    UnitConverter<T,DESCRIPTOR> converter(
    (T) 1./N, // physDeltaX: spacing between two lattice cells in __m__
    (T) 1./(M*N), // physDeltaT: time step in __s__
    (T) 1., // charPhysLength: reference length of simulation geometry
    (T) 1., // charPhysVelocity: maximal/highest expected velocity during simulation in __m / s__
    (T) 1./100., // physViscosity: physical kinematic viscosity in __m^2 / s__
    (T) 1. // physDensity: physical density in __kg / m^3__
    );

    const int saveIter = converter.getLatticeTime( 1. );

    // Prints the converter log as console output
    converter.print();
    // Writes the converter log in a file
    converter.write(“bstep3d”);

    // === 2nd Step: Prepare Geometry ===
    Vector<T,3> extend( lx0, ly0, lz0 );
    Vector<T,3> origin;
    IndicatorCuboid3D<T> cuboid( extend, origin );

    // Instantiation of a cuboidGeometry with weights
    #ifdef PARALLEL_MODE_MPI
    const int noOfCuboids = singleton::mpi().getSize();
    #else
    const int noOfCuboids = 7;
    #endif
    CuboidGeometry3D<T> cuboidGeometry( cuboid, converter.getConversionFactorLength(), noOfCuboids );

    // Instantiation of a loadBalancer
    HeuristicLoadBalancer<T> loadBalancer( cuboidGeometry );

    // Instantiation of a superGeometry
    SuperGeometry<T,3> superGeometry( cuboidGeometry, loadBalancer );

    prepareGeometry( converter, superGeometry );

    // === 3rd Step: Prepare Lattice ===
    SuperLattice<T, DESCRIPTOR> sLattice( superGeometry );

    //prepareLattice and set boundaryConditions
    prepareLattice( converter, sLattice, superGeometry );

    // === 4th Step: Main Loop with Timer ===
    clout << “starting simulation…” << std::endl;
    util::Timer<T> timer( converter.getLatticeTime( maxPhysT ), superGeometry.getStatistics().getNvoxel() );
    timer.start();

    // Set up persistent measuring functors for result extraction
    SuperLatticePhysVelocity3D<T, DESCRIPTOR> velocity( sLattice, converter );
    SuperEuklidNorm3D<T> normVel( velocity );

    BlockReduction3D2D<T> planeReduction(
    normVel,
    Hyperplane3D<T>().centeredIn(cuboidGeometry.getMotherCuboid()).normalTo({0,0,1}),
    600,
    BlockDataSyncMode::ReduceOnly);
    std::size_t iT = 0;
    if (!(sLattice.load(“bstep3d.checkpoint”)))
    {
    for (; iT < converter.getLatticeTime( maxPhysT ); ++iT ) {

    // === 5th Step: Definition of Initial and Boundary Conditions ===
    setBoundaryValues( converter, sLattice, iT, superGeometry );

    // === 6th Step: Collide and Stream Execution ===
    sLattice.collideAndStream();

    // === 7th Step: Computation and Output of the Results ===
    getResults( sLattice, converter, planeReduction, iT, superGeometry, timer );
    if ( iT%( saveIter/2 )==0 && iT>0 ) {
    clout << “Checkpointing the system at t=” << iT << std::endl;
    sLattice.save( “bstep3d.checkpoint” );
    }
    }
    }
    else
    {
    for (iT = LAST_TIME_STEP_IN_THE_CHECKPOINT_FILE + 1; iT < converter.getLatticeTime( maxPhysT ); ++iT ) {

    // === 5th Step: Definition of Initial and Boundary Conditions ===
    setBoundaryValues( converter, sLattice, iT, superGeometry );

    // === 6th Step: Collide and Stream Execution ===
    sLattice.collideAndStream();

    // === 7th Step: Computation and Output of the Results ===
    getResults( sLattice, converter, planeReduction, iT, superGeometry, timer );
    if ( iT%( saveIter/2 )==0 && iT>0 ) {
    clout << “Checkpointing the system at t=” << iT << std::endl;
    sLattice.save( “bstep3d.checkpoint” );
    }
    }
    }
    timer.stop();
    timer.printSummary();
    }

    Regarding the second issue, what I want to address is the saving and loading of SuperParticleSystem3D<T, PARTICLE> supParticleSystem(superGeometry);, because I’ve seen someone trying to implement the serialization of superGeometry(https://www.openlb.net/forum/topic/check-point-using-gpu/), which is also achievable.I’ll be able to calculate the coupling of fluid and particles simultaneously when I stop the program and restart the computation.

    Best regards,
    Rookie

    #8475
    jan
    Participant

    Dear Rookie,

    Loading the saved data will indeed restart the simulation at the first time step, but with the saved fluid data. If you want to continue from the stopped time step, I’d suggest to write this information additionally to another file and then read the time step from this file when loading the fluid data. Then use it to set iT accordingly. See [1] for an introduction to file input/output, if necessary.

    Unfortunately, there is currently no implemented method for saving and loading the particle data. You could do it manually, for example by writing data (global particle ID, position, velocity, force, …) to files and loading them later, creating new particles and updating their data with the input from the file.

    Best regards,
    Jan

    [1]: https://cplusplus.com/doc/tutorial/files/

    #8477
    Rookie
    Participant

    I followed your advice to output and read iT, and successfully implemented the continuation of the fluid simulation. I will try to write checkpoints for particles as well. Thank you for always providing me with helpful assistance. I hope the next version of OpenLB can improve this part of the functionality. Wish you all the best! I’ve placed the code below to help others solve similar problems in the future.
    if (!(sLattice.load(“bstep3d.checkpoint”)))
    {
    for (; iT < converter.getLatticeTime( maxPhysT ); ++iT ) {

    // === 5th Step: Definition of Initial and Boundary Conditions ===
    setBoundaryValues( converter, sLattice, iT, superGeometry );

    // === 6th Step: Collide and Stream Execution ===
    sLattice.collideAndStream();

    // === 7th Step: Computation and Output of the Results ===
    getResults( sLattice, converter, planeReduction, iT, superGeometry, timer );
    if ( iT%( saveIter/2 )==0 && iT>0 ) {
    clout << “Checkpointing the system at t=” << iT << std::endl;
    sLattice.save( “bstep3d.checkpoint” );
    std::ofstream checkpointFile(“bstep3d_iT.txt”);
    checkpointFile << iT;
    checkpointFile.close();
    }
    }
    }
    else
    {
    std::ifstream checkpointFile(“bstep3d_iT.txt”);
    if (checkpointFile.is_open()) {
    checkpointFile >> iT;
    checkpointFile.close();
    } else {
    iT = -1;
    }
    for (++iT; iT < converter.getLatticeTime( maxPhysT ); ++iT ) {

    // === 5th Step: Definition of Initial and Boundary Conditions ===
    setBoundaryValues( converter, sLattice, iT, superGeometry );

    // === 6th Step: Collide and Stream Execution ===
    sLattice.collideAndStream();

    // === 7th Step: Computation and Output of the Results ===
    getResults( sLattice, converter, planeReduction, iT, superGeometry, timer );
    if ( iT%( saveIter/2 )==0 && iT>0 ) {
    clout << “Checkpointing the system at t=” << iT << std::endl;
    sLattice.save( “bstep3d.checkpoint” );
    std::ofstream checkpointFile(“bstep3d_iT.txt”);
    checkpointFile << iT;
    checkpointFile.close();
    }
    }
    }
    timer.stop();
    timer.printSummary();
    }

    #8497
    jan
    Participant

    Dear Rookie,

    I am glad to hear that it works now and thank you very much for providing the working source code.

    Best regards,
    Jan

Viewing 6 posts - 1 through 6 (of 6 total)
  • You must be logged in to reply to this topic.