'nonblocking3' BVT test fails

[tjcw]

This is the second of the MPICH BVT tests which fails with today's 'git clone' of OMPI.

• https://gist.github.com/tjcw/530ba5bb593a0f480b08de89d0497dc6

It fails with the following message

--------------------------------------------------------------------------
[1,0]<stderr>:[oc0436844531:11983] *** Process received signal ***
[1,0]<stderr>:[oc0436844531:11983] Signal: Segmentation fault (11)
[1,0]<stderr>:[oc0436844531:11983] Signal code:  (128)
[1,0]<stderr>:[oc0436844531:11983] Failing at address: (nil)
[1,1]<stderr>:[oc0436844531:11984] *** Process received signal ***
[1,1]<stderr>:[oc0436844531:11984] Signal: Segmentation fault (11)
[1,1]<stderr>:[oc0436844531:11984] Signal code:  (128)
[1,1]<stderr>:[oc0436844531:11984] Failing at address: (nil)
[1,0]<stderr>:[oc0436844531:11983] [ 0] /lib64/libpthread.so.0(+0xf100)[0x7f55dc43d100]
[1,0]<stderr>:[oc0436844531:11983] [ 1] /usr/local/lib/libmpi.so.0(PMPI_Ialltoallw+0x13e)[0x7f55dc6aadce]
[1,1]<stderr>:[oc0436844531:11984] [ 0] [1,0]<stderr>:[oc0436844531:11983] [ 2] nonblocking3[0x40383d]
[1,0]<stderr>:[oc0436844531:11983] [ 3] nonblocking3[0x403ab5]
[1,0]<stderr>:[oc0436844531:11983] [ 4] [1,1]<stderr>:/lib64/libpthread.so.0(+0xf100)[0x7f57c14f1100]
[1,1]<stderr>:[oc0436844531:11984] [ 1] [1,0]<stderr>:/lib64/libc.so.6(__libc_start_main+0xf5)[0x7f55dc08db15]
[1,0]<stderr>:[oc0436844531:11983] [ 5] nonblocking3[0x402bf9]
[1,0]<stderr>:[oc0436844531:11983] *** End of error message ***
[1,1]<stderr>:/usr/local/lib/libmpi.so.0(PMPI_Ialltoallw+0x13e)[0x7f57c175edce]
[1,1]<stderr>:[oc0436844531:11984] [ 2] nonblocking3[0x40383d]
[1,1]<stderr>:[oc0436844531:11984] [ 3] nonblocking3[0x403ab5]
[1,1]<stderr>:[oc0436844531:11984] [ 4] [1,1]<stderr>:/lib64/libc.so.6(__libc_start_main+0xf5)[0x7f57c1141b15]
[1,1]<stderr>:[oc0436844531:11984] [ 5] nonblocking3[0x402bf9]
[1,1]<stderr>:[oc0436844531:11984] *** End of error message ***
-------------------------------------------------------
Primary job  terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
-------------------------------------------------------
--------------------------------------------------------------------------
mpirun noticed that process rank 0 with PID 0 on node oc0436844531 exited on signal 11 (Segmentation fault).
--------------------------------------------------------------------------
[tjcw@oc0436844531 coll]$ 

[tjcw]

The 'gist' for this problem is here <script src="https://gist.github.com/tjcw/530ba5bb593a0f480b08de89d0497dc6.js"></script>

[tjcw]

nonblocking3.zip

[tjcw]

Trying again with the 'gist' for this problem

https://gist.github.com/tjcw/530ba5bb593a0f480b08de89d0497dc6

[ggouaillardet]

can you please double check your source file ? or post a link to mpich git
i do not see any MPI_Ialltoallw in the gist (broken link) nor mpich nonblocking.c nor nonblicking3.c

[tjcw]

I think I have fixed the 'gist' link now.

[ggouaillardet]

thanks, i will have a look at it
btw, how many MPI tasks are you running ?
can you also please post yoyr configure command line ?

[tjcw]

This was with 2 MPI tasks. I configured with './configure' .

[jsquyres]

Is this a duplicate issue of #2150?

[ggouaillardet]

@jsquyres these are two distinct issues, i can reproduce both and am now on it

[ggouaillardet]

@tjcw the issue here is we do not support (yet) MPI_Op_free until the non blocking request has completed

[jjhursey]

@ggouaillardet @bosilca I wanted to check in on this ticket to see if there has been progress. And where I can be of help.

I see two PRs that are related:

• PR non-blocking collectives: retain MPI_op and MPI_Datatype(s) #2154 which has one approach
• PR Don't refcount the predefined datatypes. #2393 which seems to have another somewhat related approach

[bosilca]

@jjhursey the 2 PR you mentions are entirely different. The first one (PR #2154) addresses specifically the refcount for op and datatypes for non-blocking collectives by adding a completion callback. The latter (PR #2393) prevents any refcount update for all send/recv operation for predefined data. I see them as complementary, solving different parts of a larger problem.

[ggouaillardet]

@jjhursey as pointed by @bosilca #2154 addresses only the non blocking collectives but at the MPI level.
(and btw, it also prevents updating refcounts of predefined datatypes/ops)

@bosilca had a negative comment about the implementation itself and i acknowledge that, though i did not had much time to revamp it.

currently, refcounts are updated at the pml level, and per our SC discussion, we should do that at the MPI level, regardless communications are point to point vs collective or blocking vs non blocking.
that requires we extend the current ompi_request_t structure (e.g. add two pointers to non predefined datatype, one for op and the last one for communicator), that will greatly simplify #2154 (e.g. no more need to insert a callback)...
except when handling the infamous MPI_{A,Ia}lltoallw and MPI_{N,In}eighbor_alltoallw since we potentially have 2*comm_size)

makes sense ?