MPI run — pvd file fault

[steed188]

Hello,
When I use OpenMPI to parallelly run a case, I found that the pvd file, which should be the main interface for vtk file groups, has some faults ,leading to an error when read in ParaView. Part of the pvd file is like belows:

<DataSet timestep=”0″ group=”” part=” 5″ file=”data/Cube<DataSet timestep=”0″ group=”” part=” 5″ file=”data/Cube<DataSet timestep=”0″ group=”” part=” 0″ file=”data/Cube<DataSet timestep=”0″ group=”” part=” 6″ file=”data/Cube_iT0000000iC00006.vti”/>
<DataSet timestep=”0″ group=”” part=” 2″ file=”data/Cube<DataSet timestep=”0″ group=”” part=” 4″ file=”data/Cube_iT0000000iC00004.vti”/>
<DataSet timestep=”0″ group=”” part=” 3″ file=”data/Cube_iT0000000iC00003.vti”/>
</Collection>
</VTKFile>
</Collection>
</VTKFile>
<DataSet timestep=”0″ group=”” part=” 5″ file=”data/Cube_iT0000000iC00005.vti”/>
<DataSet timestep=”0″ group=”” part=” 1″ file=”data/Cube_iT0000000iC00001.vti”/>
</Collection>
</VTKFile>
</Collection>
</VTKFile>
<DataSet timestep=”0″ group=”” part=” 6″ file=”data/Cube<DataSet timestep=”0″ gro<DataSet timestep=”0″ group=”” part=” 1″ file=”data/Cube<DataSet timestep=”0″ group=”” part=” 3″ file=”data/Cube_iT0000000iC00003.vti”/>
up=”” part=” 6″ file=”data/Cube_iT0000000iC00006.vti”/>
<DataSet timestep=”0″ group=”” part=” 5″ file=”data/Cube<DataSet timestep=”0″ group=”” part=” 2″ file=”data/Cube_iT0000000iC00002.vti”/>
</Collection>
</VTKFile>
</Collection>

You can see that the order of the vti files is chaotic, even when I ran the example cases. It will only be normal if I ran the case sequential.And I am using OpenLB 1.0, in the platform of CentOS linux 7.3.1611 with OpenMPI 1.6.3.

your sincerely
steed188

[steed188]

And I am using OpenLB 1.0, in the platform of CentOS linux 7.3.1611 with OpenMPI 1.6.3

[albert.mink]

Dear Steed188,

I attached the sample pvd from cylinder3d. No matter if you you execute sequential or parallel, yout pvd file should look like this.

<?xml version=”1.0″?>
<VTKFile type=”Collection” version=”0.1″ byte_order=”LittleEndian”>
<Collection>
<DataSet timestep=”0″ group=”” part=”” file=”data/cylinder3d_iT0000000.vtm”/>
<DataSet timestep=”900″ group=”” part=”” file=”data/cylinder3d_iT0000900.vtm”/>
…
</Collection>
</VTKFile>

By the way, can you open the pvd file with a web browser?

Regards,
Albert

[steed188]

Dear albert.min-k,
Thank you so much for your focus on my question.
When I execute sequential, the pvd file is correct. But when I excute Parellel, the file is wrong. I seems like that the processors scrambled to write the vti files in the pvd file with out an order sequencing, leading the file fault.
And I can’t open the wrong pvd file with web browser, showing error.
Regards,
steed188

[albert.mink]

Yes, I agree. In your situation, several processes want to access the same file and then it crashes somehow.

So this means you might have a careful look on you MPI.

Can you post YOUR Makefile.inc in the OpenLB project?

[steed188]

Dear albert.min-k,
I’d love to. I agree with you that it may be my MPI’s fault. I have tried OpenLB1.1. The fault still existed.
Would you mind telling me how can I post the Makefile.inc in the OpenLB project?

[albert.mink]

I see, the forum does not support any kind of attachement. That is not good.

By now, just copy the first 66 lines of the Makefile.inc in the thread.

Best regards,
Albert

[steed188]

Dear albert.min-k,
you mean the Makefile.inc file in the root folder of OpenLB? Listed below:

# This file is part of the OpenLB library
#
# Copyright (C) 2007 Mathias Krause
# E-mail contact: info@openlb.net
# The most recent release of OpenLB can be downloaded at
# <http://www.openlb.net/&gt;
#
# This program is free software; you can redistribute it and/or
# modify it under the terms of the GNU General Public License
# as published by the Free Software Foundation; either version 2
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public
# License along with this program; if not, write to the Free
# Software Foundation, Inc., 51 Franklin Street, Fifth Floor,
# Boston, MA 02110-1301, USA.

###########################################################################
###########################################################################
## DEFINITIONS TO BE CHANGED

CXX := g++
#CXX := icpc -D__aligned__=ignored
#CXX := mpiCC
#CXX := mpic++

OPTIM := -O3 -Wall
DEBUG := -g -DOLB_DEBUG

CXXFLAGS := $(OPTIM)
#CXXFLAGS := $(DEBUG)

# to enable std::shared_ptr in functor arithmetik
# works in gcc 4.3 and later, source https://gcc.gnu.org/projects/cxx0x.html
CXXFLAGS += -std=c++0x
# works in gcc 4.7 and later (recommended)
#CXXFLAGS += -std=c++11

#CXXFLAGS += -fdiagnostics-color=auto
#CXXFLAGS += -std=gnu++14

ARPRG := ar
#ARPRG := xiar # mandatory for intel compiler

LDFLAGS :=

PARALLEL_MODE := OFF
#PARALLEL_MODE := MPI
#PARALLEL_MODE := OMP
#PARALLEL_MODE := HYBRID

MPIFLAGS :=
OMPFLAGS := -fopenmp

BUILDTYPE := precompiled
#BUILDTYPE := generic

###########################################################################
## conditional settings

ifeq ($(BUILDTYPE), precompiled)
CXXFLAGS := -DOLB_PRECOMPILED $(CXXFLAGS)
endif

ifeq ($(PARALLEL_MODE), MPI)
CXXFLAGS := -DPARALLEL_MODE_MPI $(MPIFLAGS) $(CXXFLAGS)
endif

ifeq ($(PARALLEL_MODE), OMP)
CXXFLAGS := -DPARALLEL_MODE_OMP $(OMPFLAGS) $(CXXFLAGS)
LDFLAGS := $(OMPFLAGS) $(LDFLAGS)
endif

ifeq ($(PARALLEL_MODE), HYBRID)
CXXFLAGS := -DPARALLEL_MODE_OMP -DPARALLEL_MODE_MPI $(OMPFLAGS) $(MPIFLAGS) $(CXXFLAGS)
LDFLAGS := $(OMPFLAGS) $(LDFLAGS)
endif

###########################################################################
## defines shell

SHELL := /bin/sh

###########################################################################
## dependencies, object, library directory and library name

DEPENDDIR := build/$(BUILDTYPE)/dep
OBJDIR := build/$(BUILDTYPE)/obj
LIBDIR := build/$(BUILDTYPE)/lib
LIB := olb

###########################################################################
## search directories

SUBDIRS := src/boundary \
src/communication \
src/dynamics \
src/core \
src/geometry \
src/external/tinyxml \
src/functors \
src/functors/indicator \
src/io \
src/particles \
src/particles/forces \
src/particles/boundaries \
src/utilities

EXAMPLEDIRS := examples/aorta3d \
examples/bifurcation3d/eulerEuler \
examples/bifurcation3d/eulerLagrange \
examples/bstep2d \
examples/bstep3d \
examples/cavity2d/sequential \
examples/cavity2d/parallel \
examples/cavity3d/sequential \
examples/cavity3d/parallel \
examples/cylinder2d \
examples/cylinder3d \
examples/multiComponent2d \
examples/multiComponent3d \
examples/nozzle3d \
examples/phaseSeparation2d \
examples/phaseSeparation3d \
examples/poiseuille2d/bgk \
examples/poiseuille2d/forced \
examples/poiseuille2d/mrt \
examples/thermal2d \
examples/thermal3d \
examples/venturi3d

INCLUDEDIRS := src \
src/ \
src/external

BUILDTYPEDIRS := build/precompiled \
build/generic

SUBDIRSH := $(foreach d,$(SUBDIRS),$(ROOT)/$(d))
IDIR := $(foreach d,$(INCLUDEDIRS),-I$(ROOT)/$(d))

###########################################################################
###########################################################################

[albert.mink]

Well, this is the entire Makefile.inc 😉

Anyway, modify it according
CXX := mpic++
PARALLEL_MODE := MPI

Then, run
make cleanbuild
make clean
make

// Till now, you executed a sequential programm on several threads. With the above modifications the compiler will generate a parallel program.

Albert

[steed188]

I tried to modified the Makefile.inc like belowed

#CXX := g++
#CXX := icpc -D__aligned__=ignored
#CXX := mpiCC
CXX := mpic++
…….
#PARALLEL_MODE := OFF
PARALLEL_MODE := MPI
#PARALLEL_MODE := OMP
#PARALLEL_MODE := HYBRID

and made it again. But it didn’t work. The pvd file was still bad.

yours steed188

[steed188]

Hi Albert,
I got the point. My OpenMPI’s version was 1.6.3. I changed it to version 1.10.0. And it works.
I don’t know why.
Thank you very much.
Yours steed188

[Kai]

With mpich-3.1.3, openLB 1.1r0 has the same problem when running in parallel.

“Error parsing XML in stream at line 25, column 7, byte index 4045797: not well-formed(invalid token)” …

The example is multiComponent2d.

[Markus Mohrhard]

Quote:

Quote from Kai on July 29, 2017, 11:01
With mpich-3.1.3, openLB 1.1r0 has the same problem when running in parallel.

“Error parsing XML in stream at line 25, column 7, byte index 4045797: not well-formed(invalid token)” …

The example is multiComponent2d.

Hi Kai,

can you paste your Makefile.inc and make sure that you used a MPI frontend compiler, e.g. mpic++ or mpiCC and used the MPI mode.

In Makefile.inc the following two lines are important:

CXX := mpic++
PARALLEL_MODE := MPI

After making sure that these are the only CXX and PARALLEL_MODE lines that are not prefixed with a # go to examples/multComponent2d and call:

make clean cleanbuild && make && mpirun -np 4 ./rayleighTaylor2d

This should make sure that the code is built with MPI support and that the code is run in parallel through MPI. If you still see the problem please paste the Makefile.inc file so that we can continue searching the problem.

Regards,
Markus

[Kai]

Hi, Markus, I found the reason now, as posted in another board.

“There are more than one mpi installation in my machine. I used mpirun and mpic++ from different installations (maybe more things were messed up). Now it works fine.”

It was a mess/mistake from my side.

[Author]

Posts