Parallelization usig MPI
- This topic has 4 replies, 2 voices, and was last updated 5 months ago by .
-
Posts
-
Hi all,
I have been trying to set up an OpenLB simulation in our university HPC system and wanted to use MPI to parallelize it. The code compiles correctly but when I run it using mpirun, the output looks as if the whole code is being executed by individual threads i.e., if I have 72 threads running, I would get 72 individual converter print output in my log file and so on. I am providing the config and code below, and could someone tell me what I am missing out?config.mk
CXX := mpic++ CXXFLAGS := -O3 -Wall -march=native -mtune=native CXXFLAGS += -std=c++20 LDFLAGS := PARALLEL_MODE := MPI #default is OFF MPIFLAGS := OMPFLAGS := -fopenmp PLATFORMS := CPU_SISD FLOATING_POINT_TYPE := double FEATURES := USE_EMBEDDED_DEPENDENCIES := ONSimulation Code (only the relevant parts)
#include <olb.h> using namespace olb; using namespace olb::descriptors; using T = FLOATING_POINT_TYPE; using DESCRIPTOR = D3Q27<descriptors::FORCE, FreeSurface::MASS, FreeSurface::EPSILON, FreeSurface::CELL_TYPE, FreeSurface::CELL_FLAGS, FreeSurface::TEMP_MASS_EXCHANGE, FreeSurface::PREVIOUS_VELOCITY>; using BulkDynamics = SmagorinskyBGKdynamics<T,DESCRIPTOR>; // Rest of the functions and helper structs int main(int argc, char* argv[]){ initialize(argc, argv); OstreamManager clout(std::cout, "main"); SimVolHelper c; UnitConverterFromResolutionAndRelaxationTime<T, DESCRIPTOR> const converter( int {c.N}, // resolution: number of voxels per charPhysL (T) c.latticeRelaxationTime, // latticeRelaxationTime: relaxation time, have to be greater than 0.5! (T) c.charPhysLength, // charPhysLength: reference length of simulation geometry (T) c.charPhysVel, // charPhysVelocity: maximal/highest expected velocity during simulation in __m / s__ (T) c.viscosity, // physViscosity: physical kinematic viscosity in __m^2 / s__ (T) c.density // physDensity: physical density in __kg / m^3__ ); converter.print(); STLreader<T> simvol( "openlb_m.stl", converter.getPhysDeltaX(), 1, olb::RayMode::FastRayZ, true); IndicatorLayer3D<T> extendedDomain( simvol, converter.getPhysDeltaX() ); simvol.print(); #ifdef PARALLEL_MODE_MPI const int numCuboids = util::min(16, 8*singleton::mpi().getSize()); CuboidDecomposition3D<T> cuboidDecomposition( extendedDomain, converter.getPhysDeltaX(), numCuboids*singleton::mpi().getSize() ); #else const int numCuboids = 1; //for serial execution CuboidDecomposition3D<T> cuboidDecomposition( extendedDomain, converter.getPhysDeltaX(), numCuboids ); #endif // Instantiation of a loadBalancer HeuristicLoadBalancer<T> loadBalancer( cuboidDecomposition ); // Instantiation of a superGeometry SuperGeometry<T,3> superGeometry( cuboidDecomposition, loadBalancer ); prepareGeometry( converter, extendedDomain, simvol, superGeometry ); SuperLattice<T, DESCRIPTOR> sLattice(superGeometry); prepareLattice(converter, sLattice, superGeometry, c, simvol); FreeSurface3DSetup<T,DESCRIPTOR> free_surface_setup{sLattice}; free_surface_setup.addPostProcessor(); // Set variables from freeSurfaceHelpers.h T surface_tension_coefficient_factor = std::pow(converter.getConversionFactorTime(),2)/ (c.density * std::pow(converter.getPhysDeltaX(),3)); sLattice.setParameter<FreeSurface::DROP_ISOLATED_CELLS>(true); sLattice.setParameter<FreeSurface::TRANSITION>(c.transitionThreshold); sLattice.setParameter<FreeSurface::LONELY_THRESHOLD>(c.lonelyThreshold); sLattice.setParameter<FreeSurface::HAS_SURFACE_TENSION>(c.has_surface_tension); sLattice.setParameter<FreeSurface::SURFACE_TENSION_PARAMETER>(surface_tension_coefficient_factor * c.surface_tension_coefficient); // === 4th Step: Main Loop with Timer === clout << "starting simulation..." << std::endl; util::Timer<T> timer( converter.getLatticeTime( c.physTime ), superGeometry.getStatistics().getNvoxel() ); timer.start(); setInitialValues(sLattice, superGeometry, converter); for ( std::size_t iT = 0; iT < converter.getLatticeTime( c.physTime ); ++iT ) { setBoundaryValues(sLattice, superGeometry, converter, iT); getResults( sLattice, converter, iT, superGeometry, timer ); sLattice.collideAndStream(); } timer.stop(); timer.printSummary(); return 0; }Thanks in advance!
The usual source of this behavior is that MPI was not enabled in the config. As this is done in your case the next best explanation is that you did not recompile. Did you execute a full rebuild after the config change?
In principle you can just stick to using
g++as the backend compiler formpic++. Of course, on Intel CPUs you get quite nice speedups using Intel’s C++ compilers. The new error indicates that the ICC version is too old and / or pulls in a old standard library, specifically it doesn’t support C++20.Which cluster are you using? We also have some example configs in the
config/directory.
- You must be logged in to reply to this topic.