[Adrian]

Yes, we recently started directly pushing more developments to the public. I amended the description on the download page accordingly. If you want to stay on a specific release you can checkout the tag.

We actually will push release 1.9 today / tomorrow so you can consider these case changes as “non-WIP” 🙂

• This reply was modified 1 month, 3 weeks ago by Adrian.

[Adrian]

This is now fixed. Can you pull the latest master and confirm that it also fixes your case?

[Adrian]

Sorry, I can not reproduce this. Did you try retracing the working steps in a clean checkout? This would ensure that the trial and error did not leave any broken objects lying around for the linker to pick up.

[Adrian]

On not copying the “edges”: You may need to trigger an additional overlap communication as each process only updates its own local data with the (interpolated) values.

On the memory corruption: Did you maybe accidentally destruct the old lattice before / during interpolation? Seeing just the message I can only guess.

The decomposition will be the same if you use the default settings (which you do) and the same number of processes load balanced by the same load balancer.

[Adrian]

I see, this is interesting!

An efficient general implementation will require custom MPI communication I think, executing parts on CPU won’t change the fundamental problem of the data not being available where it needs to be.

For an intermediary step: How large of a simulation domain / how many cells do you currently plan to use this on? If you can e.g. be sure that this will fit on a single node or even single GPU (which can fit a lot of cells nowadays) the implementation will be much simpler.

[Adrian]

You should not need the communicateToAllflags here, overlap communication (to ensure the interpolation stencil has the correct neighbor information) is enough. This all relies on the exact same block decomposition being used between the resolutions (s.t. both the coarse and fine information for any given cell reside on the same process). Can you confirm whether your code works (for a smaller resolution) on a single process / detail the way you execute this?

[Adrian]

Overlap communication is not designed to encompass the entire domain. You will need custom code to realize such a complex scheme. E.g. for our FSI integrals we implemented custom PerBlock scoped post processors with CPU / GPU specific implementations.

Can you explain what you want to implement in more detail? I do not quite get why you need to access cells at the opposite side of the domain (unless it is for periodicity which is a different question and doesn’t require the overlap to span the entire domain).

[Adrian]

Did you confirm that the data is set up how you expect e.g. by viewing the fields in Paraview?

For the GPU case: Did you switch the processing context s.t. the data is synced to GPU before you execute the post processor?

[Adrian]

You can use the AnalyticalFfromSuperF interpolation functor to interpolate data (exposed by functors, e.g. SuperLatticeFieldF) between resolutions. The geometry setup should be separate and will need special handling depending on the case (as there is no generic way to transfer the boundary setup between resolutions for arbitrary BCs)

[Adrian]

Thanks for these additional details and good to know that the same problem occurs on CPU. Looking at your debug output I suspect that the issue is a out-of-bounds / undefined access to a neighbor cell (leading to the most likely invalid cell index of ~2e9, especially considering the index of the original cell). Looking at your interpolation function I can not exclude the possibility that very far / out of bounds neighbors are accessed. For performance reasons the index validity is not checked by default leading to the segfault we observe.

What is the max range of neighbors accessed from a cell? Did you consider that the overlap needs to be large enough to cover these accesses (if the post processor is executed at the outermost non-overlap cells). The issue might also be faulty initialization of your extent fields.

Side note: It seems to me that e.g. DOMAIN_EXTENT could be a parameter instead of a per-cell field. Using parameters where possible will help reduce your memory load.

[Adrian]

So I think I was able to reproduce your exact issue by:

1. fixing the escape issue for GNU Make 4.2.1
2. compiling without GPU support (e.g. by using the default config.mk)
3. Switching on mixed compilation mode GPU support
4. Recompiling just the case, not the core library
5. => linker errors for the same symbols that you got

So your setup should be fixable by executing make clean-core additionally and then rebuilding (forgot to mention this before)

I am still looking into a general fix for the escape issue as escaping the include fixes 4.2.1 but breaks 4.4.1. Will keep you updated.

[Adrian]

Sorry for the delay, I can reproduce the issue in GNU Make 4.2.1, it works for 4.4. I’ll look into it and provide a fix for you.

[Adrian]

Good to hear that it works now!

[Adrian]

How do you compile exactly? If the cpu::Cell is used this is not running on GPU. Square bracket access to populations is definitely implemented for the GPU cells. The way the transparent execution works is that the operator apply is instantiated for the platform-specific cells. They all share the same outer functionality just implemented differently.

The code you posted should work on GPU (assuming it was called correctly in the apply (can you also show the code of this or ideally your entire post processor?)

Right now I suspect that it is just a compilation issue.

[Adrian]

This is unusual, can you share which version of GNU Make you are using? The change to the escape in the mixed config should not be necessary (I also just re-tested the build of channel3d in mixed mode to be sure).

In any case: Did you do a full rebuild of the example after you tweaked the config? (make clean; make) Also, just to be safe: Did you change anything else beside the config? Is there a specific reason why you added the OLB_MIXED_MODE_INCLUDE_CPP := ON to the config.mk? This flag is already set by default in the example’s Makefile.

[Author]

Posts