Multi-GPU with grid refinement in OpenLB 1.8-1
- This topic has 3 replies, 2 voices, and was last updated 2 days, 20 hours ago by .
Viewing 4 posts - 1 through 4 (of 4 total)
-
Posts
-
Hi Team,
I’m running multi-GPU tests in OpenLB 1.8-1 and wanted to confirm whether grid refinement is officially supported on multiple GPUs, and to ask for help with a crash I’m seeing.
Works: /olb-1.8-1-release/examples/laminar/cylinder3d runs successfully on two H200 GPUs.
Crashes:
/olb-1.8-1-release/examples/gridRefinement/cylinder2d/
/olb-1.8-1-release/examples/gridRefinement/sphere3d/
Both fail on two H200 GPUs at the coupler construction:auto coarseToFine = refinement::lagrava::makeCoarseToFineCoupler( sLatticeLevel0, sGeometryLevel0, sLatticeLevel1, sGeometryLevel1);Build & runtime environment
OpenLB: 1.8-1 (release)
GCC: 11.4.0 (Ubuntu 22.04)
Open MPI: 4.1.6
UCX: 1.17.0
GPUs: NVIDIA H200 (CUDA arch 90)
config.mk highlights:
CXX := mpic++
CC := gcc# CXXFLAGS := -O0 -g -Wall -std=c++20 #(Debug run used)
CXXFLAGS := -O3 -Wall -march=native -mtune=native
CXXFLAGS += -std=c++20PARALLEL_MODE := MPI
PLATFORMS := CPU_SISD GPU_CUDA
CUDA_CXX := nvcc
CUDA_CXXFLAGS := -O3 -std=c++20
CUDA_LDFLAGS := -L/run/opengl-driver/lib
CUDA_ARCH := 90FLOATING_POINT_TYPE := float
USE_EMBEDDED_DEPENDENCIES := ON
Question:
Does the released OpenLB 1.8-1 support multi-GPU parallel runs with grid refinement?
If yes, are there known limitations or configuration requirements for refinement::lagrava::makeCoarseToFineCoupler on multi-GPU?
Any guidance on fixes or patches would be much appreciated.
Backtrace (gridRefinement/sphere3d/ on two H200 GPUs):
Loading openmpi4/gcc/4.1.6-cuda12.6 Loading requirement: cuda/12.6.2 --------------- 2 GPU are running--------------- [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". [New Thread 0x1555504a8000 (LWP 388159)] [New Thread 0x1555504a8000 (LWP 388158)] [New Thread 0x15554bfff000 (LWP 388161)] [New Thread 0x15554bfff000 (LWP 388160)] [New Thread 0x15554b5fe000 (LWP 388163)] [New Thread 0x15554b5fe000 (LWP 388162)] [New Thread 0x15554821f000 (LWP 388165)] [New Thread 0x15554821f000 (LWP 388164)] [Detaching after vfork from child process 388168] [Detaching after vfork from child process 388167] [Detaching after vfork from child process 388169] [Detaching after vfork from child process 388170] [MpiManager] Sucessfully initialized, numThreads=2 [New Thread 0x1552fb9ec000 (LWP 388320)] [ThreadPool] Sucessfully initialized, numThreads=1 [New Thread 0x155303b5b000 (LWP 388321)] [GPU_CUDA:0] Found 2 CUDA devices but only one can be used per MPI process. [UnitConverter] ----------------- UnitConverter information ----------------- [UnitConverter] -- Parameters: [UnitConverter] Resolution: N= 11 [UnitConverter] Lattice velocity: latticeU= 0.0151515 [UnitConverter] Lattice relaxation frequency: omega= 1.9802 [UnitConverter] Lattice relaxation time: tau= 0.505 [UnitConverter] Characteristical length(m): charL= 0.2 [UnitConverter] Characteristical speed(m/s): charU= 0.2 [UnitConverter] Phys. kinematic viscosity(m^2/s): charNu= 0.0004 [UnitConverter] Phys. density(kg/m^d): charRho= 1 [UnitConverter] Characteristical pressure(N/m^2): charPressure= 0 [UnitConverter] Mach number: machNumber= 0.0262432 [UnitConverter] Reynolds number: reynoldsNumber= 100 [UnitConverter] Knudsen number: knudsenNumber= 0.000262432 [UnitConverter] Characteristical CFL number: charCFLnumber= 0.0151515 [UnitConverter] [UnitConverter] -- Conversion factors: [UnitConverter] Voxel length(m): physDeltaX= 0.0181818 [UnitConverter] Time step(s): physDeltaT= 0.00137741 [UnitConverter] Velocity factor(m/s): physVelocity= 13.2 [UnitConverter] Density factor(kg/m^3): physDensity= 1 [UnitConverter] Mass factor(kg): physMass= 6.01052e-06 [UnitConverter] Viscosity factor(m^2/s): physViscosity= 0.24 [UnitConverter] Force factor(N): physForce= 0.0576001 [UnitConverter] Pressure factor(N/m^2): physPressure= 174.24 [UnitConverter] ------------------------------------------------------------- [GPU_CUDA:1] Found 2 CUDA devices but only one can be used per MPI process. [cuboid] --------Cuboid Details---------- [cuboid] Corner: [-0.0181818 -0.0181818 -0.0181818] [cuboid] Delta: 0.0181818 [cuboid] Perimeter: 1.4281 [cuboid] Volume: 0.113599 [cuboid] Extent: [21 30 30] [cuboid] Nodes at Perimeter: 4002 [cuboid] Nodes in Volume: 18900 [cuboid] Nodes in Indicator: 18900 [cuboid] Other Corner: [0.345455 0.509091 0.509091] [cuboid] -------------------------------- [cuboid] --------Cuboid Details---------- [cuboid] Corner: [0.363636 -0.0181818 -0.0181818] [cuboid] Delta: 0.0181818 [cuboid] Perimeter: 3.3719 [cuboid] Volume: 0.378663 [cuboid] Extent: [70 30 30] [cuboid] Nodes at Perimeter: 9686 [cuboid] Nodes in Volume: 63000 [cuboid] Nodes in Indicator: 63000 [cuboid] Other Corner: [1.61818 0.509091 0.509091] [cuboid] -------------------------------- [cuboid] --------Cuboid Details---------- [cuboid] Corner: [1.63636 -0.0181818 -0.0181818] [cuboid] Delta: 0.0181818 [cuboid] Perimeter: 2.53884 [cuboid] Volume: 0.265064 [cuboid] Extent: [49 30 30] [cuboid] Nodes at Perimeter: 7250 [cuboid] Nodes in Volume: 44100 [cuboid] Nodes in Indicator: 44100 [cuboid] Other Corner: [2.50909 0.509091 0.509091] [cuboid] -------------------------------- [cuboid] --------Cuboid Details---------- [cuboid] Corner: [0.363636 -0.0181818 -0.0181818] [cuboid] Delta: 0.0181818 [cuboid] Perimeter: 1.58678 [cuboid] Volume: 0.0378663 [cuboid] Extent: [70 3 30] [cuboid] Nodes at Perimeter: 4394 [cuboid] Nodes in Volume: 6300 [cuboid] Nodes in Indicator: 6300 [cuboid] Other Corner: [1.61818 0.0181818 0.509091] [cuboid] -------------------------------- [cuboid] --------Cuboid Details---------- [cuboid] Corner: [0.363636 0.0363636 -0.0181818] [cuboid] Delta: 0.0181818 [cuboid] Perimeter: 2.97521 [cuboid] Volume: 0.30293 [cuboid] Extent: [70 24 30] [cuboid] Nodes at Perimeter: 8510 [cuboid] Nodes in Volume: 50400 [cuboid] Nodes in Indicator: 50400 [cuboid] Other Corner: [1.61818 0.454545 0.509091] [cuboid] -------------------------------- [cuboid] --------Cuboid Details---------- [cuboid] Corner: [0.363636 0.472727 -0.0181818] [cuboid] Delta: 0.0181818 [cuboid] Perimeter: 1.58678 [cuboid] Volume: 0.0378663 [cuboid] Extent: [70 3 30] [cuboid] Nodes at Perimeter: 4394 [cuboid] Nodes in Volume: 6300 [cuboid] Nodes in Indicator: 6300 [cuboid] Other Corner: [1.61818 0.509091 0.509091] [cuboid] -------------------------------- [cuboid] --------Cuboid Details---------- [cuboid] Corner: [0.363636 0.0363636 -0.0181818] [cuboid] Delta: 0.0181818 [cuboid] Perimeter: 1.29719 [cuboid] Volume: 0.030293 [cuboid] Extent: [70 24 3] [cuboid] Nodes at Perimeter: 3542 [cuboid] Nodes in Volume: 5040 [cuboid] Nodes in Indicator: 5040 [cuboid] Other Corner: [1.61818 0.454545 0.0181818] [cuboid] -------------------------------- [cuboid] --------Cuboid Details---------- [cuboid] Corner: [0.363636 0.0363636 0.0363636] [cuboid] Delta: 0.0181818 [cuboid] Perimeter: 2.60231 [cuboid] Volume: 0.242344 [cuboid] Extent: [70 24 24] [cuboid] Nodes at Perimeter: 7406 [cuboid] Nodes in Volume: 40320 [cuboid] Nodes in Indicator: 40320 [cuboid] Other Corner: [1.61818 0.454545 0.454545] [cuboid] -------------------------------- [cuboid] --------Cuboid Details---------- [cuboid] Corner: [0.363636 0.0363636 0.472727] [cuboid] Delta: 0.0181818 [cuboid] Perimeter: 1.29719 [cuboid] Volume: 0.030293 [cuboid] Extent: [70 24 3] [cuboid] Nodes at Perimeter: 3542 [cuboid] Nodes in Volume: 5040 [cuboid] Nodes in Indicator: 5040 [cuboid] Other Corner: [1.61818 0.454545 0.509091] [cuboid] -------------------------------- [cuboid] --------Cuboid Details---------- [cuboid] Corner: [0.363636 0.0363636 0.0363636] [cuboid] Delta: 0.0181818 [cuboid] Perimeter: 1.49157 [cuboid] Volume: 0.121172 [cuboid] Extent: [35 24 24] [cuboid] Nodes at Perimeter: 4186 [cuboid] Nodes in Volume: 20160 [cuboid] Nodes in Indicator: 20160 [cuboid] Other Corner: [0.981818 0.454545 0.454545] [cuboid] -------------------------------- [cuboid] --------Cuboid Details---------- [cuboid] Corner: [1 0.0363636 0.0363636] [cuboid] Delta: 0.0181818 [cuboid] Perimeter: 1.49157 [cuboid] Volume: 0.121172 [cuboid] Extent: [35 24 24] [cuboid] Nodes at Perimeter: 4186 [cuboid] Nodes in Volume: 20160 [cuboid] Nodes in Indicator: 20160 [cuboid] Other Corner: [1.61818 0.454545 0.454545] [cuboid] -------------------------------- [cuboid] --------Cuboid Details---------- [cuboid] Corner: [1 0.0363636 0.0363636] [cuboid] Delta: 0.0181818 [cuboid] Perimeter: 1.02347 [cuboid] Volume: 0.060586 [cuboid] Extent: [35 12 24] [cuboid] Nodes at Perimeter: 2818 [cuboid] Nodes in Volume: 10080 [cuboid] Nodes in Indicator: 10080 [cuboid] Other Corner: [1.61818 0.236364 0.454545] [cuboid] -------------------------------- [cuboid] --------Cuboid Details---------- [cuboid] Corner: [1 0.254545 0.0363636] [cuboid] Delta: 0.0181818 [cuboid] Perimeter: 1.02347 [cuboid] Volume: 0.060586 [cuboid] Extent: [35 12 24] [cuboid] Nodes at Perimeter: 2818 [cuboid] Nodes in Volume: 10080 [cuboid] Nodes in Indicator: 10080 [cuboid] Other Corner: [1.61818 0.454545 0.454545] [cuboid] -------------------------------- [CuboidDecomposition] ---Cuboid Structure Statistics--- [CuboidDecomposition] Number of Cuboids: 9 [CuboidDecomposition] Delta : 0.0181818 [CuboidDecomposition] Ratio (min): 0.0428571 [CuboidDecomposition] (max): 23.3333 [CuboidDecomposition] Nodes (min): 5040 [CuboidDecomposition] (max): 44100 [CuboidDecomposition] Weight (min): 5040 [CuboidDecomposition] (max): 44100 [CuboidDecomposition] -------------------------------- [CuboidDecomposition] ---Cuboid Structure Statistics--- [CuboidDecomposition] Number of Cuboids: 3 [CuboidDecomposition] Delta : 0.00909091 [CuboidDecomposition] Ratio (min): 0.489362 [CuboidDecomposition] (max): 3 [CuboidDecomposition] Nodes (min): 74589 [CuboidDecomposition] (max): 154630 [CuboidDecomposition] Weight (min): 80640 [CuboidDecomposition] (max): 161280 [CuboidDecomposition] -------------------------------- [prepareGeometry] Prepare Geometry ... [SuperGeometry3D] cleaned 0 outer boundary voxel(s) [SuperGeometry3D] cleaned 0 outer boundary voxel(s) [SuperGeometryStatistics3D] updated [SuperGeometry3D] the model is correct! [CuboidDecomposition] ---Cuboid Structure Statistics--- [CuboidDecomposition] Number of Cuboids: 9 [CuboidDecomposition] Delta : 0.0181818 [CuboidDecomposition] Ratio (min): 0.0428571 [CuboidDecomposition] (max): 23.3333 [CuboidDecomposition] Nodes (min): 5040 [CuboidDecomposition] (max): 44100 [CuboidDecomposition] Weight (min): 5040 [CuboidDecomposition] (max): 329420 [CuboidDecomposition] -------------------------------- [SuperGeometryStatistics3D] materialNumber=1; count=108192; minPhysR=(0,0,0); maxPhysR=(2.49091,0.490909,0.490909) [SuperGeometryStatistics3D] materialNumber=2; count=16240; minPhysR=(-0.0181818,-0.0181818,-0.0181818); maxPhysR=(2.50909,0.509091,0.509091) [SuperGeometryStatistics3D] materialNumber=3; count=784; minPhysR=(-0.0181818,0,0); maxPhysR=(-0.0181818,0.490909,0.490909) [SuperGeometryStatistics3D] materialNumber=4; count=784; minPhysR=(2.50909,0,0); maxPhysR=(2.50909,0.490909,0.490909) [SuperGeometryStatistics3D] countTotal[1e6]=0.126 [prepareGeometry] Prepare Geometry ... OK [New Thread 0x1552fabcd000 (LWP 388325)] [New Thread 0x155303383000 (LWP 388326)] [prepareLattice] Prepare Lattice ... [RefinedLoadBalancer:1] Matching fineC=1 to coarseC=7 on rank=1 [RefinedLoadBalancer:1] Matching fineC=2 to coarseC=8 on rank=1 [prepareLattice] Prepare Lattice ... OK [RefinedLoadBalancer:0] Matching fineC=0 to coarseC=6 on rank=0 [UnitConverter] ----------------- UnitConverter information ----------------- [UnitConverter] -- Parameters: [UnitConverter] Resolution: N= 22 [UnitConverter] Lattice velocity: latticeU= 0.0151515 [UnitConverter] Lattice relaxation frequency: omega= 1.96078 [UnitConverter] Lattice relaxation time: tau= 0.51 [UnitConverter] Characteristical length(m): charL= 0.2 [UnitConverter] Characteristical speed(m/s): charU= 0.2 [UnitConverter] Phys. kinematic viscosity(m^2/s): charNu= 0.0004 [UnitConverter] Phys. density(kg/m^d): charRho= 1 [UnitConverter] Characteristical pressure(N/m^2): charPressure= 0 [UnitConverter] Mach number: machNumber= 0.0262432 [UnitConverter] Reynolds number: reynoldsNumber= 100 [UnitConverter] Knudsen number: knudsenNumber= 0.000262432 [UnitConverter] Characteristical CFL number: charCFLnumber= 0.0151515 [UnitConverter] [UnitConverter] -- Conversion factors: [UnitConverter] Voxel length(m): physDeltaX= 0.00909091 [UnitConverter] Time step(s): physDeltaT= 0.000688704 [UnitConverter] Velocity factor(m/s): physVelocity= 13.2 [UnitConverter] Density factor(kg/m^3): physDensity= 1 [UnitConverter] Mass factor(kg): physMass= 7.51315e-07 [UnitConverter] Viscosity factor(m^2/s): physViscosity= 0.12 [UnitConverter] Force factor(N): physForce= 0.0144 [UnitConverter] Pressure factor(N/m^2): physPressure= 174.24 [UnitConverter] ------------------------------------------------------------- [prepareGeometryFine] Prepare Geometry ... [SuperGeometry3D] cleaned 3568 outer boundary voxel(s) [SuperGeometryStatistics3D] updated [SuperGeometry3D] error! [CuboidDecomposition] ---Cuboid Structure Statistics--- [CuboidDecomposition] Number of Cuboids: 3 [CuboidDecomposition] Delta : 0.00909091 [CuboidDecomposition] Ratio (min): 0.489362 [CuboidDecomposition] (max): 3 [CuboidDecomposition] Nodes (min): 74589 [CuboidDecomposition] (max): 154630 [CuboidDecomposition] Weight (min): 80640 [CuboidDecomposition] (max): 161280 [CuboidDecomposition] -------------------------------- [SuperGeometryStatistics3D] materialNumber=0; count=3568; minPhysR=(0.463636,0.163636,0.163636); maxPhysR=(0.636364,0.336364,0.336364) [SuperGeometryStatistics3D] materialNumber=1; count=301435; minPhysR=(0.363636,0.0363636,0.0363636); maxPhysR=(1.61818,0.454545,0.454545) [SuperGeometryStatistics3D] materialNumber=2; count=2048; minPhysR=(0.454545,0.154545,0.154545); maxPhysR=(0.645455,0.345455,0.345455) [SuperGeometryStatistics3D] countTotal[1e6]=0.307051 [prepareGeometryFine] Prepare Geometry ... OK [prepareLattice] Prepare Lattice ... [prepareLattice] Prepare Lattice ... OK Thread 1 "sphere3d" received signal SIGSEGV, Segmentation fault. 0x0000555555698142 in std::vector<olb::Cuboid<float, 3u>, std::allocator<olb::Cuboid<float, 3u> > >::operator[] (this=0x28, __n=1731567920) at /usr/include/c++/11/bits/stl_vector.h:1046 1046 return *(this->_M_impl._M_start + __n); #0 0x0000555555698142 in std::vector<olb::Cuboid<float, 3u>, std::allocator<olb::Cuboid<float, 3u> > >::operator[] (this=0x28, __n=1731567920) at /usr/include/c++/11/bits/stl_vector.h:1046 No locals. #1 0x0000555555682355 in olb::CuboidDecomposition<float, 3u>::get (this=0x0, iC=1731567920) at /home/users/astar/ihpc/zhaox/olb-release_multiGPU_openmpi_mixed/src/geometry/cuboidDecomposition.hh:143 No locals. #2 0x00005555556cb81f in olb::BlockCommunicationNeighborhood<float, 3u>::requestCell (this=0x55556735d5f8, latticeR=...) at /home/users/astar/ihpc/zhaox/olb-release_multiGPU_openmpi_mixed/src/communication/blockCommunicationNeighborhood.hh:69 cuboid = @0x55555576cc02: {_origin = {<olb::ScalarVector<float, 3, olb::Vector<float, 3> >> = {<olb::GenericVector<float, 3, olb::Vector<float, 3> >> = {<No data fields>}, <No data fields>}, _data = {_M_elems = {2.40370637e-29, 219112.469, -2.41820512e-33}}}, _extent = {<olb::ScalarVector<int, 3, olb::Vector<int, 3> >> = {<olb::GenericVector<int, 3, olb::Vector<int, 3> >> = {<No data fields>}, <No data fields>}, _data = {_M_elems = {-1991710595, -1991712651, -1958160299}}}, _delta = 7.13655101e-13, _weight = 16736936732031449157} paddedBlock = {_core = {<olb::ScalarVector<int, 3, olb::Vector<int, 3> >> = {<olb::GenericVector<int, 3, olb::Vector<int, 3> >> = {<No data fields>}, <No data fields>}, _data = {_M_elems = {2, 0, -28980}}}, _size = {<olb::ScalarVector<int, 3, olb::Vector<int, 3> >> = {<olb::GenericVector<int, 3, olb::Vector<int, 3> >> = {<No data fields>}, <No data fields>}, _data = {_M_elems = {32767, -29088, 32767}}}, _projection = {<olb::ScalarVector<int, 3, olb::Vector<int, 3> >> = {<olb::GenericVector<int, 3, olb::Vector<int, 3> >> = {<No data fields>}, <No data fields>}, _data = {_M_elems = {1432841789, 21845, 2}}}, _padding = 0} #3 0x00005555556a0381 in olb::SuperCommunicator<float, olb::SuperLattice<float, olb::descriptors::D3Q19<> > >::requestCell(olb::Vector<int, 4u>) (this=0x5555674b07c0, latticeR=...) at /home/users/astar/ihpc/zhaox/olb-release_multiGPU_openmpi_mixed/src/communication/superCommunicator.hh:151 No locals. #4 0x0000555555683e0f in olb::refinement::lagrava::makeCoarseToFineCoupler<float, olb::descriptors::D3Q19<> >(olb::SuperLattice<float, olb::descriptors::D3Q19<> >&, olb::SuperGeometry<float, olb::descriptors::D3Q19<>::d>&, olb::SuperLattice<float, olb::descriptors::D3Q19<> >&, olb::SuperGeometry<float, olb::descriptors::D3Q19<>::d>&)::{lambda(olb::Vector<int, 3u>)#1}::operator()(olb::Vector<int, 3u>) const (__closure=0x7fffffff8fd0, fineLatticeR=...) at /home/users/astar/ihpc/zhaox/olb-release_multiGPU_openmpi_mixed/src/refinement/algorithm/lagrava.h:304 coarseLatticeR = {<olb::ScalarVector<int, 4, olb::Vector<int, 4> >> = {<olb::GenericVector<int, 4, olb::Vector<int, 4> >> = {<No data fields>}, <No data fields>}, _data = {_M_elems = {7, -1, 0, 0}}} coarseToFineCommunicatorCoarse = @0x5555674b07c0: {_super = @0x7fffffffa170, _tagCoordinator = {_loadBalancer = @0x7fffffff9470, _tags = std::map with 0 elements}, _neighborhoodComm = 0x5555674b08a0, _communicatorComm = 0x555567357ac0, _blockNeighborhoods = std::vector of length 3, capacity 4 = {std::unique_ptr<olb::BlockCommunicationNeighborhood<float, 3>> = {get() = 0x555567359d20}, std::unique_ptr<olb::BlockCommunicationNeighborhood<float, 3>> = {get() = 0x55556735c4f0}, std::unique_ptr<olb::BlockCommunicationNeighborhood<float, 3>> = {get() = 0x5555674b7a60}}, _blockCommunicators = std::vector of length 0, capacity 0, _fieldsRequested = std::vector of length 1, capacity 1 = {{_M_target = 0x555555f22380 <typeinfo for olb::descriptors::POPULATION>}}, _remoteCuboidNeighborhood = std::set with 0 elements, _enabled = false, _ready = false} warning: RTTI symbol not found for class 'olb::RefinedLoadBalancer<float, 3u>' loadBalancerFine = @0x7fffffff95c0: {<olb::LoadBalancer<float>> = {<olb::BufferSerializable> = {<olb::Serializable> = {_vptr.Serializable = 0x555555f18268 <vtable for olb::RefinedLoadBalancer<float, 3u>+16>}, _dataBuffer = std::vector of length 0, capacity 0, _sizeBuffer = std::vector of length 0, capacity 0}, _size = 2, _loc = std::map with 3 elements = {[0] = 0, [1] = 0, [2] = 1}, _glob = std::vector of length 2, capacity 2 = {1, 2}, _rank = std::map with 3 elements = {[0] = 0, [1] = 1, [2] = 1}, _platform = std::map with 2 elements = {[1] = olb::Platform::GPU_CUDA, [2] = olb::Platform::GPU_CUDA}, _doOutput = std::map with 0 elements}, _coarseLoc = std::map with 2 elements = {[0] = 1, [1] = 2}} loadBalancerCoarse = @0x7fffffff9470: {<olb::BufferSerializable> = {<olb::Serializable> = {_vptr.Serializable = 0x555555f182c8 <vtable for olb::HeuristicLoadBalancer<float>+16>}, _dataBuffer = std::vector of length 0, capacity 0, _sizeBuffer = std::vector of length 0, capacity 0}, _size = 3, _loc = std::map with 3 elements = {[1] = 0, [7] = 1, [8] = 2}, _glob = std::vector of length 3, capacity 4 = {1, 7, 8}, _rank = std::map with 9 elements = {[0] = 0, [1] = 1, [2] = 0, [3] = 0, [4] = 0, [5] = 0, [6] = 0, [7] = 1, [8] = 1}, _platform = std::map with 0 elements, _doOutput = std::map with 0 elements} coarseToFine = std::unique_ptr<olb::SuperLatticeRefinement<float, olb::descriptors::D3Q19<> >> = {get() = 0x5555671da080} fBlock = @0x555561e6c0d0: {<olb::BlockStructureD<3>> = {_core = {<olb::ScalarVector<int, 3, olb::Vector<int, 3> >> = {<olb::GenericVector<int, 3, olb::Vector<int, 3> >> = {<No data fields>}, <No data fields>}, _data = {_M_elems = {69, 24, 47}}}, _size = {<olb::ScalarVector<int, 3, olb::Vector<int, 3> >> = {<olb::GenericVector<int, 3, olb::Vector<int, 3> >> = {<No data fields>}, <No data fields>}, _data = {_M_elems = {75, 30, 53}}}, _projection = {<olb::ScalarVector<int, 3, olb::Vector<int, 3> >> = {<olb::GenericVector<int, 3, olb::Vector<int, 3> >> = {<No data fields>}, <No data fields>}, _data = {_M_elems = {1590, 53, 1}}}, _padding = 3}, <olb::Serializable> = {_vptr.Serializable = 0x555555f17db0 <vtable for olb::ConcreteBlockLattice<float, olb::descriptors::D3Q19<>, (olb::Platform)2>+16>}, _platform = olb::Platform::GPU_CUDA, _statisticsEnabled = true, _statistics = 0x5555561a39b0, _introspectable = true} iC = @0x7fffffff904c: 0 warning: RTTI symbol not found for class 'olb::SuperGeometry<float, 3u>' c2fFrontierI = @0x7fffffff90f0: {<olb::SuperIndicatorF3D<float>> = {<olb::SuperF3D<float, bool>> = {<olb::GenericF<bool, int>> = {_vptr.GenericF = 0x555555f17b68 <vtable for olb::SuperIndicatorDomainFrontierDistanceF<float, olb::descriptors::D3Q19<> >+16>, _name = "", _n = 1, _m = 4, _ptrCalcC = std::shared_ptr<olb::GenericF<bool, int>> (empty) = {get() = 0x0}}, _superStructure = warning: RTTI symbol not found for class 'olb::SuperGeometry<float, 3u>' @0x7fffffffb240, _blockF = std::vector of length 2, capacity 2 = {std::unique_ptr<olb::BlockF3D<bool>> = {get() = 0x5555671b53d0}, std::unique_ptr<olb::BlockF3D<bool>> = {get() = 0x5555671d62d0}}}, _superGeometry = @0x7fffffffb240, _cachedData = std::unique_ptr<olb::SuperData<3, float, bool>> = {get() = 0x0}}, <No data fields>} #5 0x00005555556a0450 in olb::BlockStructureD<3u>::forSpatialLocations<olb::refinement::lagrava::makeCoarseToFineCoupler<float, olb::descriptors::D3Q19<> >(olb::SuperLattice<float, olb::descriptors::D3Q19<> >&, olb::SuperGeometry<float, olb::descriptors::D3Q19<>::d>&, olb::SuperLattice<float, olb::descriptors::D3Q19<> >&, olb::SuperGeometry<float, olb::descriptors::D3Q19<>::d>&)::{lambda(olb::Vector<int, 3u>)#1}>(olb::refinement::lagrava::makeCoarseToFineCoupler<float, olb::descriptors::D3Q19<> >(olb::SuperLattice<float, olb::descriptors::D3Q19<> >&, olb::SuperGeometry<float, olb::descriptors::D3Q19<>::d>&, olb::SuperLattice<float, olb::descriptors::D3Q19<> >&, olb::SuperGeometry<float, olb::descriptors::D3Q19<>::d>&)::{lambda(olb::Vector<int, 3u>)#1}) const (this=0x555561e6c0d8, f=...) at /home/users/astar/ihpc/zhaox/olb-release_multiGPU_openmpi_mixed/src/core/blockStructure.h:245 iZ = 0 iY = 0 iX = -2 #6 0x0000555555685957 in olb::refinement::lagrava::makeCoarseToFineCoupler<float, olb::descriptors::D3Q19<> >(olb::SuperLattice<float, olb::descriptors::D3Q19<> >&, olb::SuperGeometry<float, olb::descriptors::D3Q19<>::d>&, olb::SuperLattice<float, olb::descriptors::D3Q19<> >&, olb::SuperGeometry<float, olb::descriptors::D3Q19<>::d>&) (sLatticeCoarse=..., sGeometryCoarse=warning: RTTI symbol not found for class 'olb::SuperGeometry<float, 3u>' ..., sLatticeFine=..., sGeometryFine=warning: RTTI symbol not found for class 'olb::SuperGeometry<float, 3u>' ...) at /home/users/astar/ihpc/zhaox/olb-release_multiGPU_openmpi_mixed/src/refinement/algorithm/lagrava.h:292 fBlock = @0x555561e6c0d0: {<olb::BlockStructureD<3>> = {_core = {<olb::ScalarVector<int, 3, olb::Vector<int, 3> >> = {<olb::GenericVector<int, 3, olb::Vector<int, 3> >> = {<No data fields>}, <No data fields>}, _data = {_M_elems = {69, 24, 47}}}, _size = {<olb::ScalarVector<int, 3, olb::Vector<int, 3> >> = {<olb::GenericVector<int, 3, olb::Vector<int, 3> >> = {<No data fields>}, <No data fields>}, _data = {_M_elems = {75, 30, 53}}}, _projection = {<olb::ScalarVector<int, 3, olb::Vector<int, 3> >> = {<olb::GenericVector<int, 3, olb::Vector<int, 3> >> = {<No data fields>}, <No data fields>}, _data = {_M_elems = {1590, 53, 1}}}, _padding = 3}, <olb::Serializable> = {_vptr.Serializable = 0x555555f17db0 <vtable for olb::ConcreteBlockLattice<float, olb::descriptors::D3Q19<>, (olb::Platform)2>+16>}, _platform = olb::Platform::GPU_CUDA, _statisticsEnabled = true, _statistics = 0x5555561a39b0, _introspectable = true} insideI = @0x7fffffff90e4: {<olb::BlockF3D<bool>> = {<olb::GenericF<bool, int>> = {_vptr.GenericF = 0x5555, _name = Python Exception <class 'OverflowError'>: int too big to convert , _n = 116, _m = -28368, _ptrCalcC = <error reading variable: Cannot access memory at address 0xc>}, _blockStructure = @0x0}, _block = @0xffffb24000000000, _cachedData = 0x67194ef000007fff} outsideI = warning: RTTI symbol not found for class 'olb::SuperGeometry<float, 3u>' @0x7fffffffb240: {<olb::BlockF3D<bool>> = {<olb::GenericF<bool, int>> = {_vptr.GenericF = 0x555555f185f8 <vtable for olb::SuperGeometry<float, 3u>+16>, _name = "", _n = 1441958440, _m = 21845, _ptrCalcC = std::shared_ptr<olb::GenericF<bool, int>> (empty) = {get() = 0x0}}, _blockStructure = @0x0}, _block = @0x0, _cachedData = 0x0} iC = 0 warning: RTTI symbol not found for class 'olb::RefinedLoadBalancer<float, 3u>' loadBalancerFine = @0x7fffffff95c0: {<olb::LoadBalancer<float>> = {<olb::BufferSerializable> = {<olb::Serializable> = {_vptr.Serializable = 0x555555f18268 <vtable for olb::RefinedLoadBalancer<float, 3u>+16>}, _dataBuffer = std::vector of length 0, capacity 0, _sizeBuffer = std::vector of length 0, capacity 0}, _size = 2, _loc = std::map with 3 elements = {[0] = 0, [1] = 0, [2] = 1}, _glob = std::vector of length 2, capacity 2 = {1, 2}, _rank = std::map with 3 elements = {[0] = 0, [1] = 1, [2] = 1}, _platform = std::map with 2 elements = {[1] = olb::Platform::GPU_CUDA, [2] = olb::Platform::GPU_CUDA}, _doOutput = std::map with 0 elements}, _coarseLoc = std::map with 2 elements = {[0] = 1, [1] = 2}} loadBalancerCoarse = @0x7fffffff9470: {<olb::BufferSerializable> = {<olb::Serializable> = {_vptr.Serializable = 0x555555f182c8 <vtable for olb::HeuristicLoadBalancer<float>+16>}, _dataBuffer = std::vector of length 0, capacity 0, _sizeBuffer = std::vector of length 0, capacity 0}, _size = 3, _loc = std::map with 3 elements = {[1] = 0, [7] = 1, [8] = 2}, _glob = std::vector of length 3, capacity 4 = {1, 7, 8}, _rank = std::map with 9 elements = {[0] = 0, [1] = 1, [2] = 0, [3] = 0, [4] = 0, [5] = 0, [6] = 0, [7] = 1, [8] = 1}, _platform = std::map with 0 elements, _doOutput = std::map with 0 elements} cDecompositionFine = @0x7fffffff9760: {_motherCuboid = {_origin = {<olb::ScalarVector<float, 3, olb::Vector<float, 3> >> = {<olb::GenericVector<float, 3, olb::Vector<float, 3> >> = {<No data fields>}, <No data fields>}, _data = {_M_elems = {-0.0181818176, -0.0181818176, -0.0181818176}}}, _extent = {<olb::ScalarVector<int, 3, olb::Vector<int, 3> >> = {<olb::GenericVector<int, 3, olb::Vector<int, 3> >> = {<No data fields>}, <No data fields>}, _data = {_M_elems = {279, 59, 59}}}, _delta = 0.0090909088, _weight = 1008000}, _cuboids = std::vector of length 3, capacity 9 = {{_origin = {<olb::ScalarVector<float, 3, olb::Vector<float, 3> >> = {<olb::GenericVector<float, 3, olb::Vector<float, 3> >> = {<No data fields>}, <No data fields>}, _data = {_M_elems = {0.363636345, 0.0363636389, 0.0363636389}}}, _extent = {<olb::ScalarVector<int, 3, olb::Vector<int, 3> >> = {<olb::GenericVector<int, 3, olb::Vector<int, 3> >> = {<No data fields>}, <No data fields>}, _data = {_M_elems = {70, 47, 47}}}, _delta = 0.0090909088, _weight = 161280}, {_origin = {<olb::ScalarVector<float, 3, olb::Vector<float, 3> >> = {<olb::GenericVector<float, 3, olb::Vector<float, 3> >> = {<No data fields>}, <No data fields>}, _data = {_M_elems = {1, 0.0363636389, 0.0363636389}}}, _extent = {<olb::ScalarVector<int, 3, olb::Vector<int, 3> >> = {<olb::GenericVector<int, 3, olb::Vector<int, 3> >> = {<No data fields>}, <No data fields>}, _data = {_M_elems = {69, 24, 47}}}, _delta = 0.0090909088, _weight = 80640}, {_origin = {<olb::ScalarVector<float, 3, olb::Vector<float, 3> >> = {<olb::GenericVector<float, 3, olb::Vector<float, 3> >> = {<No data fields>}, <No data fields>}, _data = {_M_elems = {1, 0.25454545, 0.0363636389}}}, _extent = {<olb::ScalarVector<int, 3, olb::Vector<int, 3> >> = {<olb::GenericVector<int, 3, olb::Vector<int, 3> >> = {<No data fields>}, <No data fields>}, _data = {_M_elems = {69, 23, 47}}}, _delta = 0.0090909088, _weight = 80640}}, _periodicityOn = {<olb::ScalarVector<bool, 3, olb::Vector<bool, 3> >> = {<olb::GenericVector<bool, 3, olb::Vector<bool, 3> >> = {<No data fields>}, <No data fields>}, _data = {_M_elems = {false, false, false}}}} converterCoarse = @0x7fffffff9b10: {<olb::UnitConverterBase> = {_vptr.UnitConverterBase = 0x555555f18388 <vtable for olb::UnitConverterFromResolutionAndRelaxationTime<float, olb::descriptors::D3Q19<> >+16>}, _conversionLength = 0.0181818176, _conversionTime = 0.00137740897, _conversionVelocity = 13.2000141, _conversionDensity = 1, _conversionMass = 6.01051761e-06, _conversionViscosity = 0.240000233, _conversionForce = 0.0576001145, _conversionTorque = 0.00104727468, _conversionPressure = 174.240356, _charPhysLength = 0.200000003, _charPhysVelocity = 0.200000003, _physViscosity = 0.000400000019, _physDensity = 1, _charPhysPressure = 0, _resolution = 11, _latticeRelaxationTime = 0.504999995, _charLatticeVelocity = 0.0151514988, clout = {<std::basic_ostream<char, std::char_traits<char> >> = {<No data fields>}, buffer = {<std::__cxx11::basic_stringbuf<char, std::char_traits<char>, std::allocator<char> >> = {<No data fields>}, output = 0x555555f2c540 <std::cout@GLIBCXX_3.4>, text = "UnitConverter", static multiOutput = false}}} warning: RTTI symbol not found for class 'olb::SuperGeometry<float, 3u>' c2fFrontierI = {<olb::SuperIndicatorF3D<float>> = {<olb::SuperF3D<float, bool>> = {<olb::GenericF<bool, int>> = {_vptr.GenericF = 0x555555f17b68 <vtable for olb::SuperIndicatorDomainFrontierDistanceF<float, olb::descriptors::D3Q19<> >+16>, _name = "", _n = 1, _m = 4, _ptrCalcC = std::shared_ptr<olb::GenericF<bool, int>> (empty) = {get() = 0x0}}, _superStructure = Thread 1 "sphere3d" received signal SIGSEGV, Segmentation fault. 0x00005555556cb808 in olb::BlockCommunicationNeighborhood<float, 3u>::requestCell (this=0x106e0000106d, latticeR=...) at /home/users/astar/ihpc/zhaox/olb-release_multiGPU_openmpi_mixed/src/communication/blockCommunicationNeighborhood.hh:69 69 const auto& cuboid = _cuboidDecomposition.get(_iC); #0 0x00005555556cb808 in olb::BlockCommunicationNeighborhood<float, 3u>::requestCell (this=0x106e0000106d, latticeR=...) at /home/users/astar/ihpc/zhaox/olb-release_multiGPU_openmpi_mixed/src/communication/blockCommunicationNeighborhood.hh:69 cuboid = @0x55555576cc02: {_origin = {<olb::ScalarVector<float, 3, olb::Vector<float, 3> >> = {<olb::GenericVector<float, 3, olb::Vector<float, 3> >> = {<No data fields>}, <No data fields>}, _data = {_M_elems = {2.40370637e-29, 219112.469, -2.41820512e-33}}}, _extent = {<olb::ScalarVector<int, 3, olb::Vector<int, 3> >> = {<olb::GenericVector<int, 3, olb::Vector<int, 3> >> = {<No data fields>}, <No data fields>}, _data = {_M_elems = {-1991710595, -1991712651, -1958160299}}}, _delta = 7.13655101e-13, _weight = 16736936732031449157} paddedBlock = {_core = {<olb::ScalarVector<int, 3, olb::Vector<int, 3> >> = {<olb::GenericVector<int, 3, olb::Vector<int, 3> >> = {<No data fields>}, <No data fields>}, _data = {_M_elems = {2, 0, -28980}}}, _size = {<olb::ScalarVector<int, 3, olb::Vector<int, 3> >> = {<olb::GenericVector<int, 3, olb::Vector<int, 3> >> = {<No data fields>}, <No data fields>}, _data = {_M_elems = {32767, -29088, 32767}}}, _projection = {<olb::ScalarVector<int, 3, olb::Vector<int, 3> >> = {<olb::GenericVector<int, 3, olb::Vector<int, 3> >> = {<No data fields>}, <No data fields>}, _data = {_M_elems = {1432841789, 21845, 2}}}, _padding = 0} #1 0x00005555556a0381 in olb::SuperCommunicator<float, olb::SuperLattice<float, olb::descriptors::D3Q19<> > >::requestCell(olb::Vector<int, 4u>) (this=0x55556c537120, latticeR=...) at /home/users/astar/ihpc/zhaox/olb-release_multiGPU_openmpi_mixed/src/communication/superCommunicator.hh:151 No locals. #2 0x0000555555683e0f in olb::refinement::lagrava::makeCoarseToFineCoupler<float, olb::descriptors::D3Q19<> >(olb::SuperLattice<float, olb::descriptors::D3Q19<> >&, olb::SuperGeometry<float, olb::descriptors::D3Q19<>::d>&, olb::SuperLattice<float, olb::descriptors::D3Q19<> >&, olb::SuperGeometry<float, olb::descriptors::D3Q19<>::d>&)::{lambda(olb::Vector<int, 3u>)#1}::operator()(olb::Vector<int, 3u>) const (__closure=0x7fffffff8fd0, fineLatticeR=...) at /home/users/astar/ihpc/zhaox/olb-release_multiGPU_openmpi_mixed/src/refinement/algorithm/lagrava.h:304 coarseLatticeR = {<olb::ScalarVector<int, 4, olb::Vector<int, 4> >> = {<olb::GenericVector<int, 4, olb::Vector<int, 4> >> = {<No data fields>}, <No data fields>}, _data = {_M_elems = {6, 35, 0, 0}}} warning: RTTI symbol not found for class 'olb::SuperGeometry<float, 3u>' @0x7fffffffb240, _blockF = std::vector of length 2, capacity 2 = {std::unique_ptr<olb::BlockF3D<bool>> = {get() = 0x5555671b53d0}, std::unique_ptr<olb::BlockF3D<bool>> = {get() = 0x5555671d62d0}}}, _superGeometry = @0x7fffffffb240, _cachedData = std::unique_ptr<olb::SuperData<3, float, bool>> = {get() = 0x0}}, <No data fields>} coarseToFineCommunicatorCoarse = @0x55556c537120: {_super = @0x7fffffffa170, _tagCoordinator = {_loadBalancer = @0x7fffffff9470, _tags = std::map with 0 elements}, _neighborhoodComm = 0x55556c537200, _communicatorComm = 0x55556c539400, _blockNeighborhoods = std::vector of length 6, capacity 8 = {std::unique_ptr<olb::BlockCommunicationNeighborhood<float, 3>> = {get() = 0x55556c53f670}, std::unique_ptr<olb::BlockCommunicationNeighborhood<float, 3>> = {get() = 0x55556c53e510}, std::unique_ptr<olb::BlockCommunicationNeighborhood<float, 3>> = {get() = 0x55556c543eb0}, std::unique_ptr<olb::BlockCommunicationNeighborhood<float, 3>> = {get() = 0x55556c546d10}, std::unique_ptr<olb::BlockCommunicationNeighborhood<float, 3>> = {get() = 0x55556c549c80}, std::unique_ptr<olb::BlockCommunicationNeighborhood<float, 3>> = {get() = 0x55556c554460}}, _blockCommunicators = std::vector of length 0, capacity 0, _fieldsRequested = std::vector of length 1, capacity 1 = {{_M_target = 0x555555f22380 <typeinfo for olb::descriptors::POPULATION>}}, _remoteCuboidNeighborhood = std::set with 0 elements, _enabled = false, _ready = false} warning: RTTI symbol not found for class 'olb::RefinedLoadBalancer<float, 3u>' loadBalancerFine = @0x7fffffff95c0: {<olb::LoadBalancer<float>> = {<olb::BufferSerializable> = {<olb::Serializable> = {_vptr.Serializable = 0x555555f18268 <vtable for olb::RefinedLoadBalancer<float, 3u>+16>}, _dataBuffer = std::vector of length 0, capacity 0, _sizeBuffer = std::vector of length 0, capacity 0}, _size = 1, _loc = std::map with 3 elements = {[0] = 0, [1] = 0, [2] = 1}, _glob = std::vector of length 1, capacity 1 = {0}, _rank = std::map with 3 elements = {[0] = 0, [1] = 1, [2] = 1}, _platform = std::map with 1 element = {[0] = olb::Platform::GPU_CUDA}, _doOutput = std::map with 0 elements}, _coarseLoc = std::map with 1 element = {[0] = 5}} loadBalancerCoarse = @0x7fffffff9470: {<olb::BufferSerializable> = {<olb::Serializable> = {_vptr.Serializable = 0x555555f182c8 <vtable for olb::HeuristicLoadBalancer<float>+16>}, _dataBuffer = std::vector of length 0, capacity 0, _sizeBuffer = std::vector of length 0, capacity 0}, _size = 6, _loc = std::map with 6 elements = {[0] = 0, [2] = 1, [3] = 2, [4] = 3, [5] = 4, [6] = 5}, _glob = std::vector of length 6, capacity 8 = {0, 2, 3, 4, 5, 6}, _rank = std::map with 9 elements = {[0] = 0, [1] = 1, [2] = 0, [3] = 0, [4] = 0, [5] = 0, [6] = 0, [7] = 1, [8] = 1}, _platform = std::map with 0 elements, _doOutput = std::map with 0 elements} coarseToFine = std::unique_ptr<olb::SuperLatticeRefinement<float, olb::descriptors::D3Q19<> >> = {get() = 0x55556904fab0} fBlock = @0x5555693150c0: {<olb::BlockStructureD<3>> = {_core = {<olb::ScalarVector<int, 3, olb::Vector<int, 3> >> = {<olb::GenericVector<int, 3, olb::Vector<int, 3> >> = {<No data fields>}, <No data fields>}, _data = {_M_elems = {70, 47, 47}}}, _size = {<olb::ScalarVector<int, 3, olb::Vector<int, 3> >> = {<olb::GenericVector<int, 3, olb::Vector<int, 3> >> = {<No data fields>}, <No data fields>}, _data = {_M_elems = {76, 53, 53}}}, _projection = {<olb::ScalarVector<int, 3, olb::Vector<int, 3> >> = {<olb::GenericVector<int, 3, olb::Vector<int, 3> >> = {<No data fields>}, <No data fields>}, _data = {_M_elems = {2809, 53, 1}}}, _padding = 3}, <olb::Serializable> = {_vptr.Serializable = 0x555555f17db0 <vtable for olb::ConcreteBlockLattice<float, olb::descriptors::D3Q19<>, (olb::Platform)2>+16>}, _platform = olb::Platform::GPU_CUDA, _statisticsEnabled = true, _statistics = 0x55556c53ed30, _introspectable = true} iC = @0x7fffffff904c: 0 c2fInsideI = {<olb::SuperIndicatorF3D<float>> = {<olb::SuperF3D<float, bool>> = {<olb::GenericF<bool, int>> = {_vptr.GenericF = 0x555555f17b68 <vtable for olb::SuperIndicatorDomainFrontierDistanceF<float, olb::descriptors::D3Q19<> >+16>, _name = "", _n = 1, _m = 4, _ptrCalcC = std::shared_ptr<olb::GenericF<bool, int>> (empty) = {get() = 0x0}}, _superStructure = warning: RTTI symbol not found for class 'olb::SuperGeometry<float, 3u>' c2fFrontierI = @0x7fffffff90f0: {<olb::SuperIndicatorF3D<float>> = {<olb::SuperF3D<float, bool>> = {<olb::GenericF<bool, int>> = {_vptr.GenericF = 0x555555f17b68 <vtable for olb::SuperIndicatorDomainFrontierDistanceF<float, olb::descriptors::D3Q19<> >+16>, _name = "", _n = 1, _m = 4, _ptrCalcC = std::shared_ptr<olb::GenericF<bool, int>> (empty) = {get() = 0x0}}, _superStructure = warning: RTTI symbol not found for class 'olb::SuperGeometry<float, 3u>' @0x7fffffffb240, _blockF = std::vector of length 1, capacity 1 = {std::unique_ptr<olb::BlockF3D<bool>> = {get() = 0x5555690cb7b0}}}, _superGeometry = warning: RTTI symbol not found for class 'olb::SuperGeometry<float, 3u>' @0x7fffffffb240, _cachedData = std::unique_ptr<olb::SuperData<3, float, bool>> = {get() = 0x0}}, <No data fields>} #3 0x00005555556a0450 in olb::BlockStructureD<3u>::forSpatialLocations<olb::refinement::lagrava::makeCoarseToFineCoupler<float, olb::descriptors::D3Q19<> >(olb::SuperLattice<float, olb::descriptors::D3Q19<> >&, olb::SuperGeometry<float, olb::descriptors::D3Q19<>::d>&, olb::SuperLattice<float, olb::descriptors::D3Q19<> >&, olb::SuperGeometry<float, olb::descriptors::D3Q19<>::d>&)::{lambda(olb::Vector<int, 3u>)#1}>(olb::refinement::lagrava::makeCoarseToFineCoupler<float, olb::descriptors::D3Q19<> >(olb::SuperLattice<float, olb::descriptors::D3Q19<> >&, olb::SuperGeometry<float, olb::descriptors::D3Q19<>::d>&, olb::SuperLattice<float, olb::descriptors::D3Q19<> >&, olb::SuperGeometry<float, olb::descriptors::D3Q19<>::d>&)::{lambda(olb::Vector<int, 3u>)#1}) const (this=0x5555693150c8, f=...) at /home/users/astar/ihpc/zhaox/olb-release_multiGPU_openmpi_mixed/src/core/blockStructure.h:245 iZ = 0 iY = 0 iX = 70 warning: RTTI symbol not found for class 'olb::SuperGeometry<float, 3u>' @0x7fffffffb240, _blockF = std::vector of length 2, capacity 2 = {std::unique_ptr<olb::BlockF3D<bool>> = {get() = 0x5555671d63a0}, std::unique_ptr<olb::BlockF3D<bool>> = {get() = 0x5555671d6130}}}, _superGeometry = @0x7fffffffb240, _cachedData = std::unique_ptr<olb::SuperData<3, float, bool>> = {get() = 0x0}}, <No data fields>} warning: RTTI symbol not found for class 'olb::SuperGeometry<float, 3u>' c2fOutsideI = {<olb::SuperIndicatorF3D<float>> = {<olb::SuperF3D<float, bool>> = {<olb::GenericF<bool, int>> = {_vptr.GenericF = 0x555555f17b68 <vtable for olb::SuperIndicatorDomainFrontierDistanceF<float, olb::descriptors::D3Q19<> >+16>, _name = "", _n = 1, _m = 4, _ptrCalcC = std::shared_ptr<olb::GenericF<bool, int>> (empty) = {get() = 0x0}}, _superStructure = @0x7fffffffb240, _blockF = std::vector of length 2, capacity 2 = {std::unique_ptr<olb::BlockF3D<bool>> = {get() = 0x5555671d69f0}, std::unique_ptr<olb::BlockF3D<bool>> = {get() = 0x5555671d6e20}}}, _superGeometry = warning: RTTI symbol not found for class 'olb::SuperGeometry<float, 3u>' @0x7fffffffb240, _cachedData = std::unique_ptr<olb::SuperData<3, float, bool>> = {get() = 0x0}}, <No data fields>} coarseToFine = std::unique_ptr<olb::SuperLatticeRefinement<float, olb::descriptors::D3Q19<> >> = {get() = 0x5555671da080} coarseToFineCommunicatorCoarse = @0x5555674b07c0: {_super = @0x7fffffffa170, _tagCoordinator = {_loadBalancer = @0x7fffffff9470, _tags = std::map with 0 elements}, _neighborhoodComm = 0x5555674b08a0, _communicatorComm = 0x555567357ac0, _blockNeighborhoods = std::vector of length 3, capacity 4 = {std::unique_ptr<olb::BlockCommunicationNeighborhood<float, 3>> = {get() = 0x555567359d20}, std::unique_ptr<olb::BlockCommunicationNeighborhood<float, 3>> = {get() = 0x55556735c4f0}, std::unique_ptr<olb::BlockCommunicationNeighborhood<float, 3>> = {get() = 0x5555674b7a60}}, _blockCommunicators = std::vector of length 0, capacity 0, _fieldsRequested = std::vector of length 1, capacity 1 = {{_M_target = 0x555555f22380 <typeinfo for olb::descriptors::POPULATION>}}, _remoteCuboidNeighborhood = std::set with 0 elements, _enabled = false, _ready = false} #7 0x0000555555665f96 in main (argc=1, argv=0x7fffffffb9c8) at sphere3d.cpp:283 clout = {<std::basic_ostream<char, std::char_traits<char> >> = {<No data fields>}, buffer = {<std::__cxx11::basic_stringbuf<char, std::char_traits<char>, std::allocator<char> >> = {<No data fields>}, output = 0x555555f2c540 <std::cout@GLIBCXX_3.4>, text = "main", static multiOutput = false}} args = {_tokens = std::vector of length 0, capacity 0} N = 11 Re = 100 maxPhysT = 16 #4 0x0000555555685957 in olb::refinement::lagrava::makeCoarseToFineCoupler<float, olb::descriptors::D3Q19<> >(olb::SuperLattice<float, olb::descriptors::D3Q19<> >&, olb::SuperGeometry<float, olb::descriptors::D3Q19<>::d>&, olb::SuperLattice<float, olb::descriptors::D3Q19<> >&, olb::SuperGeometry<float, olb::descriptors::D3Q19<>::d>&) (sLatticeCoarse=..., sGeometryCoarse=warning: RTTI symbol not found for class 'olb::SuperGeometry<float, 3u>' converterLevel0 = {<olb::UnitConverter<float, olb::descriptors::D3Q19<> >> = {<olb::UnitConverterBase> = {_vptr.UnitConverterBase = 0x555555f18388 <vtable for olb::UnitConverterFromResolutionAndRelaxationTime<float, olb::descriptors::D3Q19<> >+16>}, _conversionLength = 0.0181818176, _conversionTime = 0.00137740897, _conversionVelocity = 13.2000141, _conversionDensity = 1, _conversionMass = 6.01051761e-06, _conversionViscosity = 0.240000233, _conversionForce = 0.0576001145, _conversionTorque = 0.00104727468, _conversionPressure = 174.240356, _charPhysLength = 0.200000003, _charPhysVelocity = 0.200000003, _physViscosity = 0.000400000019, _physDensity = 1, _charPhysPressure = 0, _resolution = 11, _latticeRelaxationTime = 0.504999995, _charLatticeVelocity = 0.0151514988, clout = {<std::basic_ostream<char, std::char_traits<char> >> = {<No data fields>}, buffer = {<std::__cxx11::basic_stringbuf<char, std::char_traits<char>, std::allocator<char> >> = {<No data fields>}, output = 0x555555f2c540 <std::cout@GLIBCXX_3.4>, text = "UnitConverter", static multiOutput = false}}}, <No data fields>} domainI = {<olb::IndicatorF3D<float>> = {<olb::GenericF<bool, float>> = {_vptr.GenericF = 0x555555f18770 <vtable for olb::IndicatorCuboid3D<float>+16>, _name = "", _n = 1, _m = 3, _ptrCalcC = std::shared_ptr<olb::GenericF<bool, float>> (empty) = {get() = 0x0}}, _myMin = {<olb::ScalarVector<float, 3, olb::Vector<float, 3> >> = {<olb::GenericVector<float, 3, olb::Vector<float, 3> >> = {<No data fields>}, <No data fields>}, _data = {_M_elems = {0, 0, 0}}}, _myMax = {<olb::ScalarVector<float, 3, olb::Vector<float, 3> >> = {<olb::GenericVector<float, 3, olb::Vector<float, 3> >> = {<No data fields>}, <No data fields>}, _data = {_M_elems = {2.5, 0.5, 0.5}}}}, _center = {<olb::ScalarVector<float, 3, olb::Vector<float, 3> >> = {<olb::GenericVector<float, 3, olb::Vector<float, 3> >> = {<No data fields>}, <No data fields>}, _data = {_M_elems = {1.25, 0.25, 0.25}}}, _xLength = 2.5, _yLength = 0.5, _zLength = 0.5} domainLayerI = {<olb::IndicatorF3D<float>> = {<olb::GenericF<bool, float>> = {_vptr.GenericF = 0x555555f186d8 <vtable for olb::IndicatorLayer3D<float>+16>, _name = "", _n = 1, _m = 3, _ptrCalcC = std::shared_ptr<olb::GenericF<bool, float>> (empty) = {get() = 0x0}}, _myMin = {<olb::ScalarVector<float, 3, olb::Vector<float, 3> >> = {<olb::GenericVector<float, 3, olb::Vector<float, 3> >> = {<No data fields>}, <No data fields>}, _data = {_M_elems = {-0.0181818176, -0.0181818176, -0.0181818176}}}, _myMax = {<olb::ScalarVector<float, 3, olb::Vector<float, 3> >> = {<olb::GenericVector<float, 3, olb::Vector<float, 3> >> = {<No data fields>}, <No data fields>}, _data = {_M_elems = {2.5181818, 0.518181801, 0.518181801}}}}, _indicatorF = {_ownF = std::unique_ptr<olb::IndicatorF3D<float>> = {get() = 0x0}, _sharedF = std::shared_ptr<olb::IndicatorF3D<float>> (empty) = {get() = 0x0}, _f = 0x7fffffff9800, _owning = false}, _layerSize = 0.0181818176} cuboidDecompositionLevel0 = {_motherCuboid = {_origin = {<olb::ScalarVector<float, 3, olb::Vector<float, 3> >> = {<olb::GenericVector<float, 3, olb::Vector<float, 3> >> = {<No data fields>}, <No data fields>}, _data = {_M_elems = {-0.0181818176, -0.0181818176, -0.0181818176}}}, _extent = {<olb::ScalarVector<int, 3, olb::Vector<int, 3> >> = {<olb::GenericVector<int, 3, olb::Vector<int, 3> >> = {<No data fields>}, <No data fields>}, _data = {_M_elems = {140, 30, 30}}}, _delta = 0.0181818176, _weight = 126000}, _cuboids = std::vector of length 9, capacity 12 = {{_origin = {<olb::ScalarVector<float, 3, olb::Vector<float, 3> >> = {<olb::GenericVector<float, 3, olb::Vector<float, 3> >> = {<No data fields>}, <No data fields>}, _data = {_M_elems = {-0.0181818176, -0.0181818176, -0.0181818176}}}, _extent = {<olb::ScalarVector<int, 3, olb::Vector<int, 3> >> = {<olb::GenericVector<int, 3, olb::Vector<int, 3> >> = {<No data fields>}, <No data fields>}, _data = {_M_elems = {21, 30, 30}}}, _delta = 0.0181818176, _weight = 18900}, {_origin = {<olb::ScalarVector<float, 3, olb::Vector<float, 3> >> = {<olb::GenericVector<float, 3, olb::Vector<float, 3> >> = {<No data fields>}, <No data fields>}, _data = {_M_elems = {1.63636363, -0.0181818176, -0.0181818176}}}, _extent = {<olb::ScalarVector<int, 3, olb::Vector<int, 3> >> = {<olb::GenericVector<int, 3, olb::Vector<int, 3> >> = {<No data fields>}, <No data fields>}, _data = {_M_elems = {49, 30, 30}}}, _delta = 0.0181818176, _weight = 44100}, {_origin = {<olb::ScalarVector<float, 3, olb::Vector<float, 3> >> = {<olb::GenericVector<float, 3, olb::Vector<float, 3> >> = {<No data fields>}, <No data fields>}, _data = {_M_elems = {0.363636345, -0.0181818176, -0.0181818176}}}, _extent = {<olb::ScalarVector<int, 3, olb::Vector<int, 3> >> = {<olb::GenericVector<int, 3, olb::Vector<int, 3> >> = {<No data fields>}, <No data fields>}, _data = {_M_elems = {70, 3, 30}}}, _delta = 0.0181818176, _weight = 6300}, {_origin = {<olb::ScalarVector<float, 3, olb::Vector<float, 3> >> = {<olb::GenericVector<float, 3, olb::Vector<float, 3> >> = {<No data fields>}, <No data fields>}, _data = {_M_elems = {0.363636345, 0.472727269, -0.0181818176}}}, _extent = {<olb::ScalarVector<int, 3, olb::Vector<int, 3> >> = {<olb::GenericVector<int, 3, olb::Vector<int, 3> >> = {<No data fields>}, <No data fields>}, _data = {_M_elems = {70, 3, 30}}}, _delta = 0.0181818176, _weight = 6300}, {_origin = {<olb::ScalarVector<float, 3, olb::Vector<float, 3> >> = {<olb::GenericVector<float, 3, olb::Vector<float, 3> >> = {<No data fields>}, <No data fields>}, _data = {_M_elems = {0.363636345, 0.0363636389, -0.0181818176}}}, _extent = {<olb::ScalarVector<int, 3, olb::Vector<int, 3> >> = {<olb::GenericVector<int, 3, olb::Vector<int, 3> >> = {<No data fields>}, <No data fields>}, _data = {_M_elems = {70, 24, 3}}}, _delta = 0.0181818176, _weight = 5040}, {_origin = {<olb::ScalarVector<float, 3, olb::Vector<float, 3> >> = {<olb::GenericVector<float, 3, olb::Vector<float, 3> >> = {<No data fields>}, <No data fields>}, _data = {_M_elems = {0.363636345, 0.0363636389, 0.472727269}}}, _extent = {<olb::ScalarVector<int, 3, olb::Vector<int, 3> >> = {<olb::GenericVector<int, 3, olb::Vector<int, 3> >> = {<No data fields>}, <No data fields>}, _data = {_M_elems = {70, 24, 3}}}, _delta = 0.0181818176, _weight = 5040}, {_origin = {<olb::ScalarVector<float, 3, olb::Vector<float, 3> >> = {<olb::GenericVector<float, 3, olb::Vector<float, 3> >> = {<No data fields>}, <No data fields>}, _data = {_M_elems = {0.363636345, 0.0363636389, 0.0363636389}}}, _extent = {<olb::ScalarVector<int, 3, olb::Vector<int, 3> >> = {<olb::GenericVector<int, 3, olb::Vector<int, 3> >> = {<No data fields>}, <No data fields>}, _data = {_M_elems = {35, 24, 24}}}, _delta = 0.0181818176, _weight = 329420}, {_origin = {<olb::ScalarVector<float, 3, olb::Vector<float, 3> >> = {<olb::GenericVector<float, 3, olb::Vector<float, 3> >> = {<No data fields>}, <No data fields>}, _data = {_M_elems = {1, 0.0363636389, 0.0363636389}}}, _extent = {<olb::ScalarVector<int, 3, olb::Vector<int, 3> >> = {<olb::GenericVector<int, 3, olb::Vector<int, 3> >> = {<No data fields>}, <No data fields>}, _data = {_M_elems = {35, 12, 24}}}, _delta = 0.0181818176, _weight = 165744}, {_origin = {<olb::ScalarVector<float, 3, olb::Vector<float, 3> >> = {<olb::GenericVector<float, 3, olb::Vector<float, 3> >> = {<No data fields>}, <No data fields>}, _data = {_M_elems = {1, 0.25454545, 0.0363636389}}}, _extent = {<olb::ScalarVector<int, 3, olb::Vector<int, 3> >> = {<olb::GenericVector<int, 3, olb::Vector<int, 3> >> = {<No data fields>}, <No data fields>}, _data = {_M_elems = {35, 12, 24}}}, _delta = 0.0181818176, _weight = 159258}}, _periodicityOn = {<olb::ScalarVector<bool, 3, olb::Vector<bool, 3> >> = {<olb::GenericVector<bool, 3, olb::Vector<bool, 3> >> = {<No data fields>}, <No data fields>}, _data = {_M_elems = {false, false, false}}}} cuboidDecompositionLevel1 = {_motherCuboid = {_origin = {<olb::ScalarVector<float, 3, olb::Vector<float, 3> >> = {<olb::GenericVector<float, 3, olb::Vector<float, 3> >> = {<No data fields>}, <No data fields>}, _data = {_M_elems = {-0.0181818176, -0.0181818176, -0.0181818176}}}, _extent = {<olb::ScalarVector<int, 3, olb::Vector<int, 3> >> = {<olb::GenericVector<int, 3, olb::Vector<int, 3> >> = {<No data fields>}, <No data fields>}, _data = {_M_elems = {279, 59, 59}}}, _delta = 0.0090909088, _weight = 1008000}, _cuboids = std::vector of length 3, capacity 9 = {{_origin = {<olb::ScalarVector<float, 3, olb::Vector<float, 3> >> = {<olb::GenericVector<float, 3, olb::Vector<float, 3> >> = {<No data fields>}, <No data fields>}, _data = {_M_elems = {0.363636345, 0.0363636389, 0.0363636389}}}, _extent = {<olb::ScalarVector<int, 3, olb::Vector<int, 3> >> = {<olb::GenericVector<int, 3, olb::Vector<int, 3> >> = {<No data fields>}, <No data fields>}, _data = {_M_elems = {70, 47, 47}}}, _delta = 0.0090909088, _weight = 161280}, {_origin = {<olb::ScalarVector<float, 3, olb::Vector<float, 3> >> = {<olb::GenericVector<float, 3, olb::Vector<float, 3> >> = {<No data fields>}, <No data fields>}, _data = {_M_elems = {1, 0.0363636389, 0.0363636389}}}, _extent = {<olb::ScalarVector<int, 3, olb::Vector<int, 3> >> = {<olb::GenericVector<int, 3, olb::Vector<int, 3> >> = {<No data fields>}, <No data fields>}, _data = {_M_elems = {69, 24, 47}}}, _delta = 0.0090909088, _weight = 80640}, {_origin = {<olb::ScalarVector<float, 3, olb::Vector<float, 3> >> = {<olb::GenericVector<float, 3, olb::Vector<float, 3> >> = {<No data fields>}, <No data fields>}, _data = {_M_elems = {1, 0.25454545, 0.0363636389}}}, _extent = {<olb::ScalarVector<int, 3, olb::Vector<int, 3> >> = {<olb::GenericVector<int, 3, olb::Vector<int, 3> >> = {<No data fields>}, <No data fields>}, _data = {_M_elems = {69, 23, 47}}}, _delta = 0.0090909088, _weight = 80640}}, _periodicityOn = {<olb::ScalarVector<bool, 3, olb::Vector<bool, 3> >> = {<olb::GenericVector<bool, 3, olb::Vector<bool, 3> >> = {<No data fields>}, <No data fields>}, _data = {_M_elems = {false, false, false}}}} toBeRefinedI = {<olb::IndicatorF3D<float>> = {<olb::GenericF<bool, float>> = {_vptr.GenericF = 0x555555f18770 <vtable for olb::IndicatorCuboid3D<float>+16>, _name = "", _n = 1, _m = 3, _ptrCalcC = std::shared_ptr<olb::GenericF<bool, float>> (empty) = {get() = 0x0}}, _myMin = {<olb::ScalarVector<float, 3, olb::Vector<float, 3> >> = {<olb::GenericVector<float, 3, olb::Vector<float, 3> >> = {<No data fields>}, <No data fields>}, _data = {_M_elems = {0.400000006, 0.200000003, 0.200000003}}}, _myMax = {<olb::ScalarVector<float, 3, olb::Vector<float, 3> >> = {<olb::GenericVector<float, 3, olb::Vector<float, 3> >> = {<No data fields>}, <No data fields>}, _data = {_M_elems = {1.39999998, 0.300000012, 0.300000012}}}}, _center = {<olb::ScalarVector<float, 3, olb::Vector<float, 3> >> = {<olb::GenericVector<float, 3, olb::Vector<float, 3> >> = {<No data fields>}, <No data fields>}, _data = {_M_elems = {0.899999976, 0.25, 0.25}}}, _xLength = 1, _yLength = 0.100000001, _zLength = 0.100000001} loadBalancerLevel0 = {<olb::LoadBalancer<float>> = {<olb::BufferSerializable> = {<olb::Serializable> = {_vptr.Serializable = 0x555555f182c8 <vtable for olb::HeuristicLoadBalancer<float>+16>}, _dataBuffer = std::vector of length 0, capacity 0, _sizeBuffer = std::vector of length 0, capacity 0}, _size = 3, _loc = std::map with 3 elements = {[1] = 0, [7] = 1, [8] = 2}, _glob = std::vector of length 3, capacity 4 = {1, 7, 8}, _rank = std::map with 9 elements = {[0] = 0, [1] = 1, [2] = 0, [3] = 0, [4] = 0, [5] = 0, [6] = 0, [7] = 1, [8] = 1}, _platform = std::map with 0 elements, _doOutput = std::map with 0 elements}, _mpiNbHelper = {_size = 0, _mpiRequest = std::unique_ptr<ompi_request_t *[]> = {get() = 0x0}, _mpiStatus = std::unique_ptr<ompi_status_public_t []> = {get() = 0x0}}, _cGeometry3d = 0x7fffffff9710, _cGeometry2d = 0x0, _ratioFullEmpty = 1} sGeometryLevel0 = {<olb::SuperStructure<float, 3>> = {_vptr.SuperStructure = 0x555555f185f8 <vtable for olb::SuperGeometry<float, 3u>+16>, _cuboidDecomposition = @0x7fffffff9710, _loadBalancer = @0x7fffffff9470, _overlap = 3, clout = {<std::basic_ostream<char, std::char_traits<char> >> = {<No data fields>}, buffer = {<std::__cxx11::basic_stringbuf<char, std::char_traits<char>, std::allocator<char> >> = {<No data fields>}, output = 0x555555f2c540 <std::cout@GLIBCXX_3.4>, text = "SuperStructure3D", static multiOutput = false}}}, <olb::BufferSerializable> = {<olb::Serializable> = {_vptr.Serializable = 0x555555f18638 <vtable for olb::SuperGeometry<float, 3u>+80>}, _dataBuffer = std::vector of length 0, capacity 0, _sizeBuffer = std::vector of length 0, capacity 0}, _block = std::vector of length 3, capacity 4 = {std::unique_ptr<olb::BlockGeometry<float, 3>> = {get() = 0x555556bd4820}, std::unique_ptr<olb::BlockGeometry<float, 3>> = {get() = 0x555556be6820}, std::unique_ptr<olb::BlockGeometry<float, 3>> = {get() = 0x555556bf8820}}, _communicator = std::unique_ptr<olb::SuperCommunicator<float, olb::SuperGeometry<float, 3> >> = {get() = 0x555556befaf0}, _communicationNeeded = false, _statistics = {_superGeometry = 0x7fffffffac10, _statisticsUpdateNeeded = false, _nMaterials = 6, _material2n = std::map with 4 elements = {[1] = 108192, [2] = 16240, [3] = 784, [4] = 784}, _material2min = std::map with 4 elements = {[1] = {<olb::ScalarVector<float, 3, olb::Vector<float, 3> >> = {<olb::GenericVector<float, 3, olb::Vector<float, 3> >> = {<No data fields>}, <No data fields>}, _data = {_M_elems = {0, 0, 0}}}, [2] = {<olb::ScalarVector<float, 3, olb::Vector<float, 3> >> = {<olb::GenericVector<float, 3, olb::Vector<float, 3> >> = {<No data fields>}, <No data fields>}, _data = {_M_elems = {-0.0181818176, -0.0181818176, -0.0181818176}}}, [3] = {<olb::ScalarVector<float, 3, olb::Vector<float, 3> >> = {<olb::GenericVector<float, 3, olb::Vector<float, 3> >> = {<No data fields>}, <No data fields>}, _data = {_M_elems = {-0.0181818176, 0, 0}}}, [4] = {<olb::ScalarVector<float, 3, olb::Vector<float, 3> >> = {<olb::GenericVector<float, 3, olb::Vector<float, 3> >> = {<No data fields>}, <No data fields>}, _data = {_M_elems = {2.5090909, 0, 0}}}}, _material2max = std::map with 4 elements = {[1] = {<olb::ScalarVector<float, 3, olb::Vector<float, 3> >> = {<olb::GenericVector<float, 3, olb::Vector<float, 3> >> = {<No data fields>}, <No data fields>}, _data = {_M_elems = {2.4909091, 0.4909091, 0.4909091}}}, [2] = {<olb::ScalarVector<float, 3, olb::Vector<float, 3> >> = {<olb::GenericVector<float, 3, olb::Vector<float, 3> >> = {<No data fields>}, <No data fields>}, _data = {_M_elems = {2.5090909, 0.5090909, 0.5090909}}}, [3] = {<olb::ScalarVector<float, 3, olb::Vector<float, 3> >> = {<olb::GenericVector<float, 3, olb::Vector<float, 3> >> = {<No data fields>}, <No data fields>}, _data = {_M_elems = {-0.0181818176, 0.49090907, 0.49090907}}}, [4] = {<olb::ScalarVector<float, 3, olb::Vector<float, 3> >> = {<olb::GenericVector<float, 3, olb::Vector<float, 3> >> = {<No data fields>}, <No data fields>}, _data = {_M_elems = {2.5090909, 0.49090907, 0.49090907}}}}, _minOverMaterial = {<olb::ScalarVector<float, 3, olb::Vector<float, 3> >> = {<olb::GenericVector<float, 3, olb::Vector<float, 3> >> = {<No data fields>}, <No data fields>}, _data = {_M_elems = {-0.0181818176, -0.0181818176, -0.0181818176}}}, _maxOverMaterial = {<olb::ScalarVector<float, 3, olb::Vector<float, 3> >> = {<olb::GenericVector<float, 3, olb::Vector<float, 3> >> = {<No data fields>}, <No data fields>}, _data = {_M_elems = {2.5090909, 0.5090909, 0.5090909}}}, clout = {<std::basic_ostream<char, std::char_traits<char> >> = {<No data fields>}, buffer = {<std::__cxx11::basic_stringbuf<char, std::char_traits<char>, std::allocator<char> >> = {<No data fields>}, output = 0x555555f2c540 <std::cout@GLIBCXX_3.4>, text = "SuperGeometryStatistics3D", static multiOutput = false}}, const_this = 0x7fffffffae30}, _clout = {<std::basic_ostream<char, std::char_traits<char> >> = {<No data fields>}, buffer =warning: RTTI symbol not found for class 'olb::SuperGeometry<float, 3u>' ..., sLatticeFine=..., sGeometryFine=...) at /home/users/astar/ihpc/zhaox/olb-release_multiGPU_openmpi_mixed/src/refinement/algorithm/lagrava.h:292 fBlock = @0x5555693150c0: {<olb::BlockStructureD<3>> = {_core = {<olb::ScalarVector<int, 3, olb::Vector<int, 3> >> = {<olb::GenericVector<int, 3, olb::Vector<int, 3> >> = {<No data fields>}, <No data fields>}, _data = {_M_elems = {70, 47, 47}}}, _size = {<olb::ScalarVector<int, 3, olb::Vector<int, 3> >> = {<olb::GenericVector<int, 3, olb::Vector<int, 3> >> = {<No data fields>}, <No data fields>}, _data = {_M_elems = {76, 53, 53}}}, _projection = {<olb::ScalarVector<int, 3, olb::Vector<int, 3> >> = {<olb::GenericVector<int, 3, olb::Vector<int, 3> >> = {<No data fields>}, <No data fields>}, _data = {_M_elems = {2809, 53, 1}}}, _padding = 3}, <olb::Serializable> = {_vptr.Serializable = 0x555555f17db0 <vtable for olb::ConcreteBlockLattice<float, olb::descriptors::D3Q19<>, (olb::Platform)2>+16>}, _platform = olb::Platform::GPU_CUDA, _statisticsEnabled = true, _statistics = 0x55556c53ed30, _introspectable = true} Python Exception <class 'OverflowError'>: int too big to convert insideI = @0x7fffffff90e4: {<olb::BlockF3D<bool>> = {<olb::GenericF<bool, int>> = {_vptr.GenericF = 0x5555, _name = {<std::__cxx11::basic_stringbuf<char, std::char_traits<char>, std::allocator<char> >> = {<No data fields>}, output = 0x555555f2c540 <std::cout@GLIBCXX_3.4>, text = "SuperGeometry3D", static multiOutput = false}}, _iConstructionT = 6, _writeIncrementalVtkEnabled = true, static d = 3} , _n = 116, _m = -28368, _ptrCalcC = <error reading variable: Cannot access memory at address 0xc>}, _blockStructure = @0x0}, _block = @0xffffb24000000000, _cachedData = 0x6c168e3000007fff} warning: RTTI symbol not found for class 'olb::SuperGeometry<float, 3u> outsideI = ' @0x7fffffffb240: {<olb::BlockF3D<bool>> = {<olb::GenericF<bool, int>> = {_vptr.GenericF = 0x555555f185f8 <vtable for olb::SuperGeometry<float, 3u>+16>, _name = "", _n = 1441958440, _m = 21845, _ptrCalcC = std::shared_ptr<olb::GenericF<bool, int>> (empty) = {get() = 0x0}}, _blockStructure = @0x0}, _block = @0x0, _cachedData = 0x0} iC = 0 warning: RTTI symbol not found for class 'olb::RefinedLoadBalancer<float, 3u>' loadBalancerFine = @0x7fffffff95c0: {<olb::LoadBalancer<float>> = {<olb::BufferSerializable> = {<olb::Serializable> = {_vptr.Serializable = 0x555555f18268 <vtable for olb::RefinedLoadBalancer<float, 3u>+16>}, _dataBuffer = std::vector of length 0, capacity 0, _sizeBuffer = std::vector of length 0, capacity 0}, _size = 1, _loc = std::map with 3 elements = {[0] = 0, [1] = 0, [2] = 1}, _glob = std::vector of length 1, capacity 1 = {0}, _rank = std::map with 3 elements = {[0] = 0, [1] = 1, [2] = 1}, _platform = std::map with 1 element = {[0] = olb::Platform::GPU_CUDA}, _doOutput = std::map with 0 elements}, _coarseLoc = std::map with 1 element = {[0] = 5}} loadBalancerCoarse = @0x7fffffff9470: {<olb::BufferSerializable> = {<olb::Serializable> = {_vptr.Serializable = 0x555555f182c8 <vtable for olb::HeuristicLoadBalancer<float>+16>}, _dataBuffer = std::vector of length 0, capacity 0, _sizeBuffer = std::vector of length 0, capacity 0}, _size = 6, _loc = std::map with 6 elements = {[0] = 0, [2] = 1, [3] = 2, [4] = 3, [5] = 4, [6] = 5}, _glob = std::vector of length 6, capacity 8 = {0, 2, 3, 4, 5, 6}, _rank = std::map with 9 elements = {[0] = 0, [1] = 1, [2] = 0, [3] = 0, [4] = 0, [5] = 0, [6] = 0, [7] = 1, [8] = 1}, _platform = std::map with 0 elements, _doOutput = std::map with 0 elements} cDecompositionFine = @0x7fffffff9760: {_motherCuboid = {_origin = {<olb::ScalarVector<float, 3, olb::Vector<float, 3> >> = {<olb::GenericVector<float, 3, olb::Vector<float, 3> >> = {<No data fields>}, <No data fields>}, _data = {_M_elems = {-0.0181818176, -0.0181818176, -0.0181818176}}}, _extent = {<olb::ScalarVector<int, 3, olb::Vector<int, 3> >> = {<olb::GenericVector<int, 3, olb::Vector<int, 3> >> = {<No data fields>}, <No data fields>}, _data = {_M_elems = {279, 59, 59}}}, _delta = 0.0090909088, _weight = 1008000}, _cuboids = std::vector of length 3, capacity 9 = {{_origin = {<olb::ScalarVector<float, 3, olb::Vector<float, 3> >> = {<olb::GenericVector<float, 3, olb::Vector<float, 3> >> = {<No data fields>}, <No data fields>}, _data = {_M_elems = {0.363636345, 0.0363636389, 0.0363636389}}}, _extent = {<olb::ScalarVector<int, 3, olb::Vector<int, 3> >> = {<olb::GenericVector<int, 3, olb::Vector<int, 3> >> = {<No data fields>}, <No data fields>}, _data = {_M_elems = {70, 47, 47}}}, _delta = 0.0090909088, _weight = 161280}, {_origin = {<olb::ScalarVector<float, 3, olb::Vector<float, 3> >> = {<olb::GenericVector<float, 3, olb::Vector<float, 3> >> = {<No data fields>}, <No data fields>}, _data = {_M_elems = {1, 0.0363636389, 0.0363636389}}}, _extent = {<olb::ScalarVector<int, 3, olb::Vector<int, 3> >> = {<olb::GenericVector<int, 3, olb::Vector<int, 3> >> = {<No data fields>}, <No data fields>}, _data = {_M_elems = {69, 24, 47}}}, _delta = 0.0090909088, _weight = 80640}, {_origin = {<olb::ScalarVector<float, 3, olb::Vector<float, 3> >> = {<olb::GenericVector<float, 3, olb::Vector<float, 3> >> = {<No data fields>}, <No data fields>}, _data = {_M_elems = {1, 0.25454545, 0.0363636389}}}, _extent = {<olb::ScalarVector<int, 3, olb::Vector<int, 3> >> = {<olb::GenericVector<int, 3, olb::Vector<int, 3> >> = {<No data fields>}, <No data fields>}, _data = {_M_elems = {69, 23, 47}}}, _delta = 0.0090909088, _weight = 80640}}, _periodicityOn = {<olb::ScalarVector<bool, 3, olb::Vector<bool, 3> >> = {<olb::GenericVector<bool, 3, olb::Vector<bool, 3> >> = {<No data fields>}, <No data fields>}, _data = {_M_elems = {false, false, false}}}} sLatticeLevel0 = {<olb::SuperStructure<float, 3>> = {_vptr.SuperStructure = 0x555555f184d8 <vtable for olb::SuperLattice<float, olb::descriptors::D3Q19<> >+16>, _cuboidDecomposition = @0x7fffffff9710, _loadBalancer = @0x7fffffff9470, _overlap = 3, clout = {<std::basic_ostream<char, std::char_traits<char> >> = {<No data fields>}, buffer = {<std::__cxx11::basic_stringbuf<char, std::char_traits<char>, std::allocator<char> >> = {<No data fields>}, output = 0x555555f2c540 <std::cout@GLIBCXX_3.4>, text = "SuperStructure3D", static multiOutput = false}}}, <olb::BufferSerializable> = {<olb::Serializable> = {_vptr.Serializable = 0x555555f18520 <vtable for olb::SuperLattice<float, olb::descriptors::D3Q19<> >+88>}, _dataBuffer = std::vector of length 0, capacity 0, _sizeBuffer = std::vector of length 0, capacity 0}, _converter = std::optional<const olb::UnitConverter<float, olb::descriptors::D3Q19<> > *> = {[contained value] = 0x7fffffff9b10}, _block = std::vector of length 3, capacity 4 = {std::unique_ptr<olb::BlockLattice<float, olb::descriptors::D3Q19<> >> = {get() = 0x5555564c0e10}, std::unique_ptr<olb::BlockLattice<float, olb::descriptors::D3Q19<> >> = {get() = 0x55555cab4940}, std::unique_ptr<olb::BlockLattice<float, olb::descriptors::D3Q19<> >> = {get() = 0x55555f175440}}, _communicator = std::map with 3 elements = {[{_M_target = 0x555555f22450 <typeinfo for olb::stage::PostStream>}] = std::unique_ptr<olb::SuperCommunicator<float, olb::SuperLattice<float, olb::descriptors::D3Q19<> > >> = {get() = 0x555561a0b680}, [{_M_target = 0x555555f221a0 <typeinfo for olb::stage::PostCollide>}] = std::unique_ptr<olb::SuperCommunicator<float, olb::SuperLattice<float, olb::descriptors::D3Q19<> > >> = {get() = 0x555557f7e970}, [{_M_target = 0x555555f221c8 <typeinfo for olb::stage::Full>}] = std::unique_ptr<olb::SuperCommunicator<float, olb::SuperLattice<float, olb::descriptors::D3Q19<> > >> = {get() = 0x5555618c72e0}}, _communicationNeeded = true, _customTasks = std::map with 0 elements, _backgroundTasks = std::map with 0 elements, _statistics = {AverageT = olb::LatticeStatistics<float>::avRho, MaxT = olb::LatticeStatistics<float>::maxU, clout = {<std::basic_ostream<char, std::char_traits<char> >> = {<No data fields>}, buffer = {<std::__cxx11::basic_stringbuf<char, std::char_traits<char>, std::allocator<char> >> = {<No data fields>}, output = 0x555555f2c540 <std::cout@GLIBCXX_3.4>, text = "LatticeStatistics", static multiOutput = false}}, tmpAv = std::vector of length 2, capacity 2 = {0, 0}, tmpSum = std::vector of length 0, capacity 0, tmpMin = std::vector of length 0, capacity 0, tmpMax = std::vector of length 1, capacity 1 = {0}, tmpNumCells = 0, averageVect = std::vector of length 2, capacity 2 = {1, 0}, sumVect = std::vector of length 0, capacity 0, minVect = std::vector of length 0, capacity 0, maxVect = std::vector of length 1, capacity 1 = {0}, numCells = 762080353, latticeTime = 0, firstCall = true}, _statisticsEnabled = true, _initialized = true, static d = 3} loadBalancerLevel1 = {<olb::LoadBalancer<float>> = {<olb::BufferSerializable> = {<olb::Serializable> = {_vptr.Serializable = 0x555555f18268 <vtable for olb::RefinedLoadBalancer<float, 3u>+16>}, _dataBuffer = std::vector of length 0, capacity 0, _sizeBuffer = std::vector of length 0, capacity 0}, _size = 2, _loc = std::map with 3 elements = {[0] = 0, [1] = 0, [2] = 1}, _glob = std::vector of length 2, capacity 2 = {1, 2}, _rank = std::map with 3 elements = {[0] = 0, [1] = 1, [2] = 1}, _platform = std::map with 2 elements = {[1] = olb::Platform::GPU_CUDA, [2] = olb::Platform::GPU_CUDA}, _doOutput = std::map with 0 elements}, _coarseLoc = std::map with 2 elements = {[0] = 1, [1] = 2}} converterLevel1 = {<olb::UnitConverterBase> = {_vptr.UnitConverterBase = 0x555555f186a0 <vtable for olb::UnitConverter<float, olb::descriptors::D3Q19<> >+16>}, _conversionLength = 0.0090909088, _conversionTime = 0.000688704487, _conversionVelocity = 13.2000141, _conversionDensity = 1, _conversionMass = 7.51314701e-07, _conversionViscosity = 0.120000117, _conversionForce = 0.0144000286, _conversionTorque = 0.000130909335, _conversionPressure = 174.240356, _charPhysLength = 0.200000003, _charPhysVelocity = 0.200000003, _physViscosity = 0.000400000019, _physDensity = 1, _charPhysPressure = 0, _resolution = 22, _latticeRelaxationTime = 0.50999999, _charLatticeVelocity = 0.0151514988, clout = {<std::basic_ostream<char, std::char_traits<char> >> = {<No data fields>}, buffer = {<std::__cxx11::basic_stringbuf<char, std::char_traits<char>, std::allocator<char> >> = {<No data fields>}, output = 0x555555f2c540 <std::cout@GLIBCXX_3.4>, text = "UnitConverter", static multiOutput = false}}} warning: RTTI symbol not found for class 'olb::RefinedLoadBalancer<float, 3u>' sGeometryLevel1 = {<olb::SuperStructure<float, 3>> = {_vptr.SuperStructure = 0x555555f185f8 <vtable for olb::SuperGeometry<float, 3u>+16>, _cuboidDecomposition = @0x7fffffff9760, _loadBalancer = @0x7fffffff95c0, _overlap = 3, clout = {<std::basic_ostream<char, std::char_traits<char> >> = {<No data fields>}, buffer = {<std::__cxx11::basic_stringbuf<char, std::char_traits<char>, std::allocator<char> >> = {<No data fields>}, output = 0x555555f2c540 <std::cout@GLIBCXX_3.4>, text = "SuperStructure3D", static multiOutput = false}}}, <olb::BufferSerializable> = {<olb::Serializable> = {_vptr.Serializable = 0x555555f18638 <vtable for olb::SuperGeometry<float, 3u>+80>}, _dataBuffer = std::vector of length 0, capacity 0, _sizeBuffer = std::vector of length 0, capacity 0}, _block = std::vector of length 2, capacity 2 = {std::unique_ptr<olb::BlockGeometry<float, 3>> = {get() = 0x555561c9ea10}, std::unique_ptr<olb::BlockGeometry<float, 3>> = {get() = 0x555561c9eec0}}, _communicator = std::unique_ptr<olb::SuperCommunicator<float, olb::SuperGeometry<float, 3> >> = {get() = 0x5555618cfe60}, _communicationNeeded = false, _statistics = {_superGeometry = 0x7fffffffb240, _statisticsUpdateNeeded = true, _nMaterials = 4, _material2n = std::map with 3 elements = {[0] = 3568, [1] = 301435, [2] = 2048}, _material2min = std::map with 3 elements = {[0] = {<olb::ScalarVector<float, 3, olb::Vector<float, 3> >> = {<olb::GenericVector<float, 3, olb::Vector<float, 3> >> = {<No data fields>}, <No data fields>}, _data = {_M_elems = {0.463636339, 0.163636357, 0.163636357}}}, [1] = {<olb::ScalarVector<float, 3, olb::Vector<float, 3> >> = {<olb::GenericVector<float, 3, olb::Vector<float, 3> >> = {<No data fields>}, <No data fields>}, _data = {_M_elems = {0.363636345, 0.0363636389, 0.0363636389}}}, [2] = {<olb::ScalarVector<float, 3, olb::Vector<float, 3> >> = {<olb::GenericVector<float, 3, olb::Vector<float, 3> >> = {<No data fields>}, <No data fields>}, _data = {_M_elems = {0.454545438, 0.154545456, 0.154545456}}}}, _material2max = std::map with 3 elements = {[0] = {<olb::ScalarVector<float, 3, olb::Vector<float, 3> >> = {<olb::GenericVector<float, 3, olb::Vector<float, 3> >> = {<No data fields>}, <No data fields>}, _data = {_M_elems = {0.636363626, 0.336363614, 0.336363614}}}, [1] = {<olb::ScalarVector<float, 3, olb::Vector<float, 3> >> = {<olb::GenericVector<float, 3, olb::Vector<float, 3> >> = {<No data fields>}, <No data fields>}, _data = {_M_elems = {1.61818182, 0.454545438, 0.454545438}}}, [2] = {<olb::ScalarVector<float, 3, olb::Vector<float, 3> >> = {<olb::GenericVector<float, 3, olb::Vector<float, 3> >> = {<No data fields>}, <No data fields>}, _data = {_M_elems = {0.645454526, 0.345454544, 0.345454544}}}}, _minOverMaterial = {<olb::ScalarVector<float, 3, olb::Vector<float, 3> >> = {<olb::GenericVector<float, 3, olb::Vector<float, 3> >> = {<No data fields>}, <No data fields>}, _data = {_M_elems = {0, 0, 0}}}, _maxOverMaterial = {<olb::ScalarVector<float, 3, olb::Vector<float, 3> >> = {<olb::GenericVector<float, 3, olb::Vector<float, 3> >> = {<No data fields>}, <No data fields>}, _data = {_M_elems = {1.61818182, 0.454545438, 0.454545438}}}, clout = {<std::basic_ostream<char, std::char_traits<char> >> = {<No data fields>}, buffer = {<std: converterCoarse = @0x7fffffff9b10: {<olb::UnitConverterBase> = {_vptr.UnitConverterBase = 0x555555f18388 <vtable for olb::UnitConverterFromResolutionAndRelaxationTime<float, olb::descriptors::D3Q19<> >+16>}, _conversionLength = 0.0181818176, _conversionTime = 0.00137740897, _conversionVelocity = 13.2000141, _conversionDensity = 1, _conversionMass = 6.01051761e-06, _conversionViscosity = 0.240000233, _conversionForce = 0.0576001145, _conversionTorque = 0.00104727468, _conversionPressure = 174.240356, _charPhysLength = 0.200000003, _charPhysVelocity = 0.200000003, _physViscosity = 0.000400000019, _physDensity = 1, _charPhysPressure = 0, _resolution = 11, _latticeRelaxationTime = 0.504999995, _charLatticeVelocity = 0.0151514988, clout = {<std::basic_ostream<char, std::char_traits<char> >> = {<No data fields>}, buffer = {<std::__cxx11::basic_stringbuf<char, std::char_traits<char>, std::allocator<char> >> = {<No data fields>}, output = 0x555555f2c540 <std::cout@GLIBCXX_3.4>, text = "UnitConverter", static multiOutput = false}}} warning: RTTI symbol not found for class 'olb::SuperGeometry<float, 3u>' c2fFrontierI = {<olb::SuperIndicatorF3D<float>> = {<olb::SuperF3D<float, bool>> = {<olb::GenericF<bool, int>> = {_vptr.GenericF = 0x555555f17b68 <vtable for olb::SuperIndicatorDomainFrontierDistanceF<float, olb::descriptors::D3Q19<> >+16>, _name = "", _n = 1, _m = 4, _ptrCalcC = std::shared_ptr<olb::GenericF<bool, int>> (empty) = {get() = 0x0}}, _superStructure = :__cxx11::basic_stringbuf<char, std::char_traits<char>, std::allocator<char> >> = {<No data fields>}, output = 0x555555f2c540 <std::cout@GLIBCXX_3.4>, text = "SuperGeometryStatistics3D", static multiOutput = false}}, const_this = 0x7fffffffb460}, _clout = {<std::basic_ostream<char, std::char_traits<char> >> = {<No data fields>}, buffer = {<std::__cxx11::basic_stringbuf<char, std::char_traits<char>, std::allocator<char> >> = {<No data fields>}, output = 0x555555f2c540 <std::cout@GLIBCXX_3.4>, text = "SuperGeometry3D", static multiOutput = false}}, _iConstructionT = 3, _writeIncrementalVtkEnabled = true, static d = 3} warning: RTTI symbol not found for class 'olb::RefinedLoadBalancer<float, 3u>' sLatticeLevel1 = {<olb::SuperStructure<float, 3>> = {_vptr.SuperStructure = 0x555555f184d8 <vtable for olb::SuperLattice<float, olb::descriptors::D3Q19<> >+16>, _cuboidDecomposition = @0x7fffffff9760, _loadBalancer = @0x7fffffffb240, _blockF = std::vector of length 1, capacity 1 = {std::unique_ptr<olb::BlockF3D<bool>> = {get() = 0x5555690cb7b0}}}, _superGeometry = warning: RTTI symbol not found for class 'olb::SuperGeometry<float, 3u>' @0x7fffffffb240, _cachedData = std::unique_ptr<olb::SuperData<3, float, bool>> = {get() = 0x0}}, <No data fields>} c2fInsideI = {<olb::SuperIndicatorF3D<float>> = {<olb::SuperF3D<float, bool>> = {<olb::GenericF<bool, int>> = {_vptr.GenericF = 0x555555f17b68 <vtable for olb::SuperIndicatorDomainFrontierDistanceF<float, olb::descriptors::D3Q19<> >+16>, _name = "", _n = 1, _m = 4, _ptrCalcC = std::shared_ptr<olb::GenericF<bool, int>> (empty) = {get() = 0x0}}, _superStructure = warning: RTTI symbol not found for class 'olb::SuperGeometry<float, 3u>' @0x7fffffff95c0, _overlap = 3, clout = {<std::basic_ostream<char, std::char_traits<char> >> = {<No data fields>}, buffer = {<std::__cxx11::basic_stringbuf<char, std::char_traits<char>, std::allocator<char> >> = {<No data fields>}, output = 0x555555f2c540 <std::cout@GLIBCXX_3.4>, text = "SuperStructure3D", static multiOutput = false}}}, <olb::BufferSerializable> = {<olb::Serializable> = {_vptr.Serializable = 0x555555f18520 <vtable for olb::SuperLattice<float, olb::descriptors::D3Q19<> >+88>}, _dataBuffer = std::vector of length 0, capacity 0, _sizeBuffer = std::vector of length 0, capacity 0}, _converter = std::optional<const olb::UnitConverter<float, olb::descriptors::D3Q19<> > *> = {[contained value] = 0x7fffffff9d00}, _block = std::vector of length 2, capacity 2 = {std::unique_ptr<olb::BlockLattice<float, olb::descriptors::D3Q19<> >> = {get() = 0x555561e6c0d0}, std::unique_ptr<olb::BlockLattice<float, olb::descriptors::D3Q19<> >> = {get() = 0x555564794070}}, _communicator = std::map with 3 elements = {[{_M_@0x7fffffffb240, _blockF = std::vector of length 1, capacity 1 = {std::unique_ptr<olb::BlockF3D<bool>> = {get() = 0x55556c1ad4f0}}}, _superGeometry = warning: RTTI symbol not found for class 'olb::SuperGeometry<float, 3u>' @0x7fffffffb240, _cachedData = std::unique_ptr<olb::SuperData<3, float, bool>> = {get() = 0x0}}, <No data fields>} warning: RTTI symbol not found for class 'olb::SuperGeometry<float, 3u>' c2fOutsideI = {<olb::SuperIndicatorF3D<float>> = {<olb::SuperF3D<float, bool>> = {<olb::GenericF<bool, int>> = {_vptr.GenericF = 0x555555f17b68 <vtable for olb::SuperIndicatorDomainFrontierDistanceF<float, olb::descriptors::D3Q19<> >+16>, _name = "", _n = 1, _m = 4, _ptrCalcC = std::shared_ptr<olb::GenericF<bool, int>> (empty) = {get() = 0x0}}, _superStructure = target = 0x555555f22450 <typeinfo for olb::stage::PostStream>}] = std::unique_ptr<olb::SuperCommunicator<float, olb::SuperLattice<float, olb::descriptors::D3Q19<> > >> = {get() = 0x5555671d6ec0}, [{_M_target = 0x555555f221a0 <typeinfo for olb::stage::PostCollide>}] = std::unique_ptr<olb::SuperCommunicator<float, olb::SuperLattice<float, olb::descriptors::D3Q19<> > >> = {get() = 0x5555670a3dd0}, [{_M_target = 0x555555f221c8 <typeinfo for olb::stage::Full>}] = std::unique_ptr<olb::SuperCommunicator<float, olb::SuperLattice<float, olb::descriptors::D3Q19<> > >> = {get() = 0x5555670e1a30}}, _communicationNeeded = true, _customTasks = std::map with 0 elements, _backgroundTasks = std::map with 0 elements, _statistics = {AverageT = (unknown: 0x508f1aa0), MaxT = (unknown: 0x1555), clout = {<std::basic_ostream<char, std::char_traits<char> >> = {<No data fields>}, buffer = {<std::__cxx11::basic_stringbuf<char, std::char_traits<char>, std::allocator<char> >> = {<No data fields>}, output = 0x555555f2c540 <std::cout@GLIBCXX_3.4>, text = "LatticeStatistics", static multiOutput = false}}, tmpAv = std::vector of length 2, capacity 2 = {0, 0}, tmpSum = std::vector of length 0, capacity 0, tmpMin = std::vector of length 0, capacity 0, tmpMax = std::vector of length 1, capacity 1 = {0}, tmpNumCells = 0, averageVect = std::vector of length 2, capacity 2 = {1, 0}, sumVect = std::vector of length 0, capacity 0, minVect = std::vector of length 0, capacity 0, maxVect = std::vector of length 1, capacity 1 = {0}, numCells = 23456177709976, latticeTime = 0, firstCall = true}, _statisticsEnabled = true, _initialized = true, static d = 3} @0x7fffffffb240, _blockF = std::vector of length 1, capacity 1 = {std::unique_ptr<olb::BlockF3D<bool>> = {get() = 0x55556c1ac6d0}}}, _superGeometry = warning: RTTI symbol not found for class 'olb::SuperGeometry<float, 3u>' @0x7fffffffb240, _cachedData = std::unique_ptr<olb::SuperData<3, float, bool>> = {get() = 0x0}}, <No data fields>} coarseToFine = std::unique_ptr<olb::SuperLatticeRefinement<float, olb::descriptors::D3Q19<> >> = {get() = 0x55556904fab0} coarseToFineCommunicatorCoarse = @0x55556c537120: {_super = @0x7fffffffa170, _tagCoordinator = {_loadBalancer = @0x7fffffff9470, _tags = std::map with 0 elements}, _neighborhoodComm = 0x55556c537200, _communicatorComm = 0x55556c539400, _blockNeighborhoods = std::vector of length 6, capacity 8 = {std::unique_ptr<olb::BlockCommunicationNeighborhood<float, 3>> = {get() = 0x55556c53f670}, std::unique_ptr<olb::BlockCommunicationNeighborhood<float, 3>> = {get() = 0x55556c53e510}, std::unique_ptr<olb::BlockCommunicationNeighborhood<float, 3>> = {get() = 0x55556c543eb0}, std::unique_ptr<olb::BlockCommunicationNeighborhood<float, 3>> = {get() = 0x55556c546d10}, std::unique_ptr<olb::BlockCommunicationNeighborhood<float, 3>> = {get() = 0x55556c549c80}, std::unique_ptr<olb::BlockCommunicationNeighborhood<float, 3>> = {get() = 0x55556c554460}}, _blockCommunicators = std::vector of length 0, capacity 0, _fieldsRequested = std::vector of length 1, capacity 1 = {{_M_target = 0x555555f22380 <typeinfo for olb::descriptors::POPULATION>}}, _remoteCuboidNeighborhood = std::set with 0 elements, _enabled = false, _ready = false} #5 0x0000555555665f96 in main (argc=1, argv=0x7fffffffb9c8) at sphere3d.cpp:283 mpi = @0x555555f2f720: {numTasks = 2, taskId = 1, ok = true, clout = {<std::basic_ostream<char, std::char_traits<char> >> = {<No data fields>}, buffer = {<std::__cxx11::basic_stringbuf<char, std::char_traits<char>, std::allocator<char> >> = {<No data fields>}, output = 0x555555f2c540 <std::cout@GLIBCXX_3.4>, text = "MpiManager", static multiOutput = false}}} nc0 = 9 nc1 = 3 coarseToFine = std::unique_ptr<olb::SuperLatticeRefinement<float, olb::descriptors::D3Q19<> >> = {get() = 0x5555671da080} fineToCoarse = std::unique_ptr<olb::SuperLatticeRefinement<float, olb::descriptors::D3Q19<> >> = {get() = 0x0} clout = {<std::basic_ostream<char, std::char_traits<char> >> = {<No data fields>}, buffer = {<std::__cxx11::basic_stringbuf<char, std::char_traits<char>, std::allocator<char> >> = {<No data fields>}, output = 0x555555f2c540 <std::cout@GLIBCXX_3.4>, text = "main", static multiOutput = false}} args = {_tokens = std::vector of length 0, capacity 0} N = 11 Re = 100 maxPhysT = 16 timer = {clout = {<std::basic_ostream<char, std::char_traits<char> >> = {<No data fields>}, buffer = {<std::__cxx11::basic_stringbuf<char, std::char_traits<char>, std::allocator<char> >> = {<No data fields>}, output = 0x0, text = "", static multiOutput = false}}, cpuTimeStart = 0, cpuTimeCur = 0, cpuTimeEnd = 0, sTimeStart = 0, sTimeCur = 0, sTimeEnd = 0, tp = 0x0, msTimeStart = {tv_sec = 0, tv_usec = 0}, msTimeCur = {tv_sec = 0, tv_usec = 0}, msTimeEnd = {tv_sec = 0, tv_usec = 0}, msTimeLast = {tv_sec = 0, tv_usec = 0}, deltaTS = 0, curTS = 0, maxTS = 0, numFC = 0, ltPas = 0, ltRem = 0, ltTot = 0, ctPas = 0, ctRem = 0, ctTot = 0, rtPas = 0, rtRem = 0, rtTot = 0, rtPasMs = 0, rtRemMs = 0, rtTotMs = 0, _printModeSummary = 0} -------------------------------------------------------------------------- mpirun has exited due to process rank 1 with PID 387882 on node H64Y574 exiting improperly. There are three reasons this could occur: 1. this process did not call "init" before exiting, but others in the job did. This can cause a job to hang indefinitely while it waits for all processes to call "init". By rule, if one process calls "init", then ALL processes must call "init" prior to termination. 2. this process called "init", but exited without calling "finalize". By rule, all processes that call "init" MUST call "finalize" prior to exiting or it will be considered an "abnormal termination" 3. this process called "MPI_Abort" or "orte_abort" and the mca parameter orte_create_session_dirs is set to false. In this case, the run-time cannot detect that the abort call was an abnormal termination. Hence, the only error message you will receive is this one. This may have caused other processes in the application to be terminated by signals sent by mpirun (as reported here). You can avoid this message by specifying -quiet on the mpirun command line. --------------------------------------------------------------------------I’m running multi-GPU tests in OpenLB 1.8-1 and wanted to confirm whether grid refinement is officially supported on multiple GPUs, and to ask for help with a crash I’m seeing.
Yes, you can use multi-GPU for the grid refinement (there was a bug in the setter in the release that has since been fixed, I just re-confirmed that the sphere case works on multi-GPU in the current head of the public repository.
Both fail on two H200 GPUs at the coupler construction:
Just a side note: Using two H200 is overkill for these cases unless you scale to many hundreds of millions of cells (of course for testing using two is useful).
If yes, are there known limitations or configuration requirements for refinement::lagrava::makeCoarseToFineCoupler on multi-GPU?
Any guidance on fixes or patches would be much appreciated.
I am using the Rohde scheme for large applications on multiple GPUs internally without problems with the coupling itself. The main challenge right now is that the block decomposition and mesh in general needs to be set up manually to fit for refinement. e.g. you can not scale this up easily / adapt the mesh easily without manual intervention. The API for this will be improved in future releases.
[GPU_CUDA:0] Found 2 CUDA devices but only one can be used per MPI process.
It seems like the mpirun command is not quite right (CUDA_VISIBLE_DEVICES is not adjusted, so each process sees both GPUs and selects the first one). In the example configs there is a example SLURM script with a example command for multi-GPU use.
Good to hear that it works now!
Viewing 4 posts - 1 through 4 (of 4 total)
- You must be logged in to reply to this topic.