Processes returns wrong results on OpenMPI 4.0.4 unless $OMPI_MCA_pml is set to "^ucx"

[ksiazekm]

OpenMPI 4.0.4
Installed from official Fedora repository with use of dnf.
OS: Fedora 33, the latest
gcc 10.2.1 20201125 (Red Hat 10.2.1-9)
AMD FX8350,

some processes randomly return wrong results on the given configuration. When
tested the same source code on Debian GNU/Linux 7 (wheezy) with OpenMPI 1.4.5
it returns correct results each time. Test on each configuration was executed
30 times in a loop. OpenMPI 4.0.4 started to work properly after set
$OMPI_MCA_pml to "^ucx".

The source code: gist link

[hppritcha]

It would be useful to know the version of UCX is being used.

[ksiazekm]

It would be useful to know the version of UCX is being used.

ucx.x86_64 1.8.1-3.fc33

[jsquyres]

Can you provide more detail about exactly what is going wrong, and/or a small example showing the problem?

[jsquyres]

I'm the worst -- you provided a gist link with a sample and I missed it. We'll investigate...

[hppritcha]

@ksiazekm i'm having reproducing this. First how many MPI processes are you using when you observe the problem and what is the tabsize you're supplying?

[ksiazekm]

Here you have 30 executions of the code from the previous gist: Test Results

[ggouaillardet]

@hppritcha FWIW, I was able to reproduce the issue on a Fedora 33 virtual machine, using Open MPI and UCX provided by the distro. I was only able to find discarding the pml/ucx component was a workaround.
I did not have time to dig a bit more and figure out whether the root cause is in Open MPI, or UCX or both.

[awlauria]

So far I can't reproduce on power9 + rhel8.2. I tried v4.0.x latest and also the v4.0.4 release tarball.

[hoopoepg]

@hppritcha FWIW, I was able to reproduce the issue on a Fedora 33 virtual machine, using Open MPI and UCX provided by the distro

hi
have you any specific configuration in virtual machine? what processor of host system?
we are trying to reproduce issue but have no success

[ggouaillardet]

That is a pretty standard Virtualbox VM running on a x86_64 (intel) processor.

[hoopoepg]

what is number of CPU? memory volume?
and could you run there ucx_info -d there & post output here?

thank you