Singleton MPI initialization and spawn

[dalcinl]

Looks like singleton MPI init and spawn is broken in the main branch.

Look at this reproducer:

#include <mpi.h>
#include <stdio.h>

int main(int argc, char *argv[])
{
  MPI_Comm parent, intercomm;

  MPI_Init(NULL, NULL);

  MPI_Comm_get_parent(&parent);
  if (MPI_COMM_NULL != parent)
    MPI_Comm_disconnect(&parent);
  
  if (argc > 1) {
    printf("Spawning '%s' ... ", argv[1]);
    MPI_Comm_spawn(argv[1], MPI_ARGV_NULL,
                   1, MPI_INFO_NULL, 0, MPI_COMM_SELF,
                   &intercomm, MPI_ERRCODES_IGNORE);
    MPI_Comm_disconnect(&intercomm);
    printf("OK\n");
  }

  MPI_Finalize();
}

Now I run that code using Open MPI v4.1.2 (system package from Fedora 36) the following two ways:

$ mpiexec -n 1 ./a.out ./a.out 
Spawning './a.out' ... OK

$ ./a.out ./a.out 
Spawning './a.out' ... OK

Note that the second way does not use mpiexec (that is, what the MPI standard calls singleton MPI initialization).

Next I run the code with ompi/main. I've configured with:

./configure \
    --without-ofi \
    --without-ucx \
    --with-pmix=internal \
    --with-prrte=internal \
    --with-libevent=internal \
    --with-hwloc=internal \
    --enable-debug \
    --enable-mem-debug \
    --disable-man-pages \
    --disable-sphinx

The first way (using mpiexec) seems to works just fine. The second way (singleton MPI init) fails:

$ mpiexec -n 1 ./a.out ./a.out 
Spawning './a.out' ... OK

$ ./a.out ./a.out 
[kw61149:1105609] OPAL ERROR: Error in file ../../ompi/dpm/dpm.c at line 2122
[kw61149:00000] *** An error occurred in MPI_Comm_spawn
[kw61149:00000] *** reported by process [440139776,0]
[kw61149:00000] *** on communicator MPI_COMM_SELF
[kw61149:00000] *** MPI_ERR_UNKNOWN: unknown error
[kw61149:00000] *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
[kw61149:00000] ***    and MPI will try to terminate your MPI job as well)

PS: Lack of singleton MPI initialization complicate some Python users wanting to dynamically spawn MPI processes as needed via mpi4py without requiring the parent process to be launched through mpiexec.

[dalcinl]

@awlauria After your recent update of submodule pointers, things got even worse. Now I cannot even call MPI_Init_thread() if mpiexec is not used:

$ mpiexec -n 1 python -c "from mpi4py import MPI"

$ python -c "from mpi4py import MPI"
--------------------------------------------------------------------------
It looks like MPI runtime init failed for some reason; your parallel process is
likely to abort.  There are many reasons that a parallel process can
fail during RTE init; some of which are due to configuration or environment
problems.  This failure appears to be an internal failure; here's some
additional information (which may only be relevant to an Open MPI
developer):

  local peers
  --> Returned "Not found" (-13) instead of "Success" (0)
--------------------------------------------------------------------------
--------------------------------------------------------------------------
It looks like MPI_INIT failed for some reason; your parallel process is
likely to abort.  There are many reasons that a parallel process can
fail during MPI_INIT; some of which are due to configuration or environment
problems.  This failure appears to be an internal failure; here's some
additional information (which may only be relevant to an Open MPI
developer):

  ompi_mpi_init: ompi_rte_init failed
  --> Returned "Not found" (-13) instead of "Success" (0)
--------------------------------------------------------------------------
--------------------------------------------------------------------------
It looks like MPI_INIT failed for some reason; your parallel process is
likely to abort.  There are many reasons that a parallel process can
fail during MPI_INIT; some of which are due to configuration or environment
problems.  This failure appears to be an internal failure; here's some
additional information (which may only be relevant to an Open MPI
developer):

  ompi_mpi_init: ompi_mpi_instance_init failed
  --> Returned "Not found" (-13) instead of "Success" (0)
--------------------------------------------------------------------------
*** An error occurred in MPI_Init_thread
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and MPI will try to terminate your MPI job as well)
[kw61149:1130097] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!

[awlauria]

@dalcinl sigh

I didn't expect it to solve this issue, but good to know. Thanks

[rhc54]

FWIW: the error appears to be in the MPI layer, and not in PMIx or PRRTE. A quick look identifies the following culprit code (taken from main branch):

    val = NULL;
    OPAL_MODEX_RECV_VALUE(rc, PMIX_LOCAL_PEERS,
                          &pname, &val, PMIX_STRING);
    if (PMIX_SUCCESS == rc && NULL != val) {
        peers = opal_argv_split(val, ',');
        free(val);
    } else {
        ret = opal_pmix_convert_status(rc);
        error = "local peers";
        goto error;
    }
    /* if we were unable to retrieve the #local peers, set it here */
    if (0 == opal_process_info.num_local_peers) {
        opal_process_info.num_local_peers = opal_argv_count(peers) - 1;
    }

You can see that not finding "local_peers", which you won't find in the case of a singleton, incorrectly results in return of an error, even though the following code correctly knows how to deal with that situation.

[jjhursey]

I took a crack at this and got singleton without spawn working, but with spawn I'm getting:

[jjhursey@f5n17 mpi] ./hello_c 
  0/  1) [f5n17] 2998639 Hello, world!
[jjhursey@f5n17 mpi]  ./simple_spawn ./simple_spawn
[f5n17:2999204] OPAL ERROR: Error in file dpm/dpm.c at line 2122
[f5n17:00000] *** An error occurred in MPI_Comm_spawn
[f5n17:00000] *** reported by process [1890451456,0]
[f5n17:00000] *** on communicator MPI_COMM_SELF
[f5n17:00000] *** MPI_ERR_UNKNOWN: unknown error
[f5n17:00000] *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
[f5n17:00000] ***    and MPI will try to terminate your MPI job as well)

I have a lead on what needs fixing (something with starting the prte environment). I'm looking into it.

[jjhursey]

Two changes required to fix this issue:

• OMPI : Fix Singletons and Singleton Spawn #10688 (PMIX_LOCAL_PEERS handing, and a prte CLI fix)
• PRRTE : Fix Keepalive CLI option openpmix/prrte#1443 (--keepalive CLI handing)

With those two fixes I was able to run a singleton without MPI_Comm_spawn and a singleton with MPI_Comm_spawn (same program as was provided in this issue)

[jjhursey@f5n17 mpi] ./hello_c 
  0/  1) [f5n17] 3347380 Hello, world!
[jjhursey@f5n17 mpi] ./simple_spawn ./simple_spawn
Spawning './simple_spawn' ... OK

[rhc54]

PRRTE change has been committed - thanks Josh! Will now port it over to PRRTE v3.0.

Note: the fix was done in the schizo/prrte component and therefore only applies to prterun with the PRRTE personality. You might want to check to see if it also needs to go into the OMPI personality.

[jjhursey]

Open MPI master now seems to have a fully working singleton spawn after the merging of the following PRs:

• Fix the dpm to fwd IO to the spawning parent #10695
• Fix Singletons and Singleton Spawn #10688
• DPM: Force 'hash' GDS component when singleton spawn #10700
• Update PRRte and PMIx pointers. #10699

shell$ date
Thu Aug 25 09:55:37 EDT 2022
shell$  git rev-parse HEAD
96fadd9d6860bd1dc89f15e88a472f310ff13c89
shell$ git submodule status
 ac7abc6e432cd3fe2d5a72809a987f180133d668 3rd-party/openpmix (v1.1.3-3604-gac7abc6e)
 0a7547330050854f8164c4805fafd8e32a2786cd 3rd-party/prrte (psrvr-v2.0.0rc1-4420-g0a754733)
shell$ git branch
* main

shell$ cd /tmp/ompi-tests-public/singleton
shell$ make clean all
rm -f hello_c simple_spawn simple_spawn_multiple *.o
mpicc hello_c.c -Wall -g -O0 -o hello_c
mpicc simple_spawn.c -Wall -g -O0 -o simple_spawn
mpicc simple_spawn_multiple.c -Wall -g -O0 -o simple_spawn_multiple
shell$ ./run.sh 
=====================
Testing: Hello with mpirun
=====================
  0/  1) [f5n18] 367194 Hello, world!
=====================
Testing: Hello as a singleton
=====================
  0/  1) [f5n18] 367300 Hello, world!
=====================
Testing: MPI_Comm_spawn with mpirun
=====================
Spawning './simple_spawn' ... OK
=====================
Testing: MPI_Comm_spawn as a singleton
=====================
Spawning './simple_spawn' ... OK
=====================
Testing: MPI_Comm_spawn_multiple with mpirun
=====================
Hello from a Child (B)
Hello from a Child (A)
Hello from a Child (B)
Spawning Multiple './simple_spawn_multiple' ... OK
=====================
Testing: MPI_Comm_spawn_multiple as a singleton
=====================
Hello from a Child (B)
Hello from a Child (A)
Hello from a Child (B)
Spawning Multiple './simple_spawn_multiple' ... OK
=====================
Success
=====================