MPI comm_spawn is hanging on disconnect

[rhc54]

Seeing this on the current master (using orte/test/mpi/simple_spawn.c):

$ mpirun -H rhc001:24 -n 3 ./simple_spawn
[17760257:0 pid 176346] starting up on node rhc001!
[17760257:1 pid 176347] starting up on node rhc001!
[17760257:2 pid 176348] starting up on node rhc001!
0 completed MPI_Init
2 completed MPI_Init
Parent [pid 176348] about to spawn!
Parent [pid 176346] about to spawn!
1 completed MPI_Init
Parent [pid 176347] about to spawn!
[17760258:0 pid 176355] starting up on node rhc001!
[17760258:1 pid 176356] starting up on node rhc001!
[17760258:2 pid 176357] starting up on node rhc001!
Parent done with spawn
Parent sending message to child
Parent done with spawn
Parent done with spawn
2 completed MPI_Init
Hello from the child 2 of 3 on host rhc001 pid 176357
0 completed MPI_Init
Hello from the child 0 of 3 on host rhc001 pid 176355
1 completed MPI_Init
Hello from the child 1 of 3 on host rhc001 pid 176356
Parent disconnected
Child 2 disconnected
Parent disconnected
Child 1 disconnected
Child 0 received msg: 38
<hang forever>

I'm not sure when this started. @ggouaillardet Would you have a chance to take a peek? If it is PMIx related, please let me know and I'll dive into it.

[ggouaillardet]

git bisect points to c696e04 (move from PMIx v2 to v3).
i will try both v2 and v3 external components with the latest master from now.

[ggouaillardet]

so i tried the latest Open MPI master, with external PMIx from both latest master and v2.1 branches.

bottom line, pmix/ext2x works fine, but pmix/ext3x hangs (most of the time)

(note i pushed 3b4b3bb in order to fix pmix/ext3x)

[ggouaillardet]

fwiw

i set a breakpoint in MPI_Disconnect() in the 3 parents and let the children run.
when the parent end up executing MPI_Disconnect(), mpirun executes pmix_server_disconnect but at that time, pmix_server_globals.nspaces contains only 2 namespaces (job 0 aka mpirun and job 1 aka the parents), which means job 2 (aka the children) has already been removed.
so when the parent try to PMIx_disconnect() from jobs 2, the PMIX_ERR_NOT_FOUND error is raised because there is no such namespace in pmix_server_globals.nspaces.

to me, that suggests something is not correctly refcount'ed.

i will resume investigations tomorrow if needed.

[ggouaillardet]

my bad about the refcount thing, orte_state_base_check_all_complete() invokes PMIx_server_deregister_nspace()

[rhc54]

sounds like it is the result of the new connect/disconnect code in PMIx. I'll take a look.

Thanks for the detective work!

[ggouaillardet]

yep, and i do not understand the new logic here.

with ext2x, a disconnect was a collective operation involving the 6 MPI tasks (e.g. 3 parents and 3 children).
but with ext3x, a disconnect is now two collective operations, the first one involves the 3 parents, and the second one involve the 3 children.
and from what i can observe, the 3 children do note even wait for the parents before exiting.

[rhc54]

odd - i'm surprised that the pmix component would split the two like that. let me look and see what is going on.

[ggouaillardet]

well, with PMIx v3, the pmix component splits the disconnect into 6 invocations of PMIX_Disconnect(), and then on the server side, these requests are "re-assembled" with all the procs from the same namespace.
In this case, we endup with 2 namespaces of 6 tasks each
from pmix_server_disconnect(), in PMIx v3

        if (NULL == (trk = new_tracker(&proc, 1, PMIX_DISCONNECTNB_CMD))) ...
        [...]
        trk->nlocal = nspace->nlocalprocs;

nspace->nlocalprocs is 6

but in PMIx v2.1, PMIx_Disconnect() is invoked only once with the 6 procs (coming from both namespaces)
from pmix_server_connect(), in PMIx v2.1

        if (NULL == (trk = new_tracker(procs, nprocs, type))) {

nprocs is received from the MPI app and is 6

[rhc54]

yeah, i know - i'm working on the problem now. The issue is that PMIx v3 assigns a new nspace to the collection, and we aren't tracking it since OMPI doesn't assign a new jobid to the combined procs. We just need to track things correctly.

[bosilca]

First let me state that I haven't looked at the code yet, but I wanted to make sure we have a correct implementation of MPI_Disconnect because it's a very special function. In most cases it behaves almost as MPI_COMM_FREE (plus it waits for the completion of all pending operations), except when it really affect the last "connection" between sets of processes from different MPI_COMM_WORLD. In this particular case it allows these processes to clean all information about the other set of processes, or in OMPI terms it allows to clean the modex of the peers.

Thus, MPI_COMM_DISCONNECT should not directly call pmix_server_disconnect because there might be extra connections to the processes in the other MPI_COMM_WORLD. As an example it is absolutely correct to have the following sequence:

MPI_COMM_SPAWN( ..., &scomm);
MPI_COMM_DUP(scomm, &ncomm);
MPI_COMM_DISCONNECT(&scomm);
/* what about ncomm ? */

Instead if we are unable to compute the transitive relationship between connected processes we should simply replace MPI_COMM_DISCONNECT by a function that waits for all pending communications and then MPI_COMM_FREE.

[rhc54]

I believe we have always relied on the code in the MPI layer (specifically, ompi_mpi_disconnect in ompi/dpm/dpm.c) to provide that level of protection. What the pmix code does is simply communicate the disconnect command to the procs passed down to it.