[Fortran] MPI_Ialltoallw fails when using commited MPI datatypes

[mathrack]

Thank you for taking the time to submit an issue!

Background information

What version of Open MPI are you using? (e.g., v3.0.5, v4.0.2, git branch name and hash, etc.)

v4.1.0

Describe how Open MPI was installed (e.g., from a source/distribution tarball, from a git clone, from an operating system distribution package, etc.)

Compiled from source (https://www.open-mpi.org/software/ompi/v4.1/). debug build.

If you are building/installing from a git clone, please copy-n-paste the output from git submodule status.

N/A

Please describe the system on which you are running

• Operating system/version: Ubuntu 20.04
• Computer hardware: i7-10810U
• Network type: N/A

---

Details of the problem

I am trying to implement non-blocking communications in a large code. However, the code tends to fail for such cases. I have reproduced the error below. When running on one CPU, the code below works when switch is set to false but fails when switch is set to true. https://stackoverflow.com/questions/66932156/mpi-alltoallw-working-and-mpi-ialltoallw-failing

program main
      
  use mpi

  implicit none

  logical :: switch
  integer, parameter :: maxSize=128
  integer scounts(maxSize), sdispls(maxSize)
  integer rcounts(maxSize), rdispls(maxSize)
  integer :: types(maxSize)
  double precision sbuf(maxSize), rbuf(maxSize)
  integer comm, size, rank, req
  integer ierr
  integer ii

  call MPI_INIT(ierr)
  comm = MPI_COMM_WORLD
  call MPI_Comm_size(comm, size, ierr)
  call MPI_Comm_rank(comm, rank, ierr)

  switch = .true.

  ! Init
  sbuf(:) = rank
  scounts(:) = 0
  rcounts(:) = 0
  sdispls(:) = 0
  rdispls(:) = 0
  types(:) = MPI_INTEGER
  if (switch) then
    ! Send one time N double precision
    scounts(1)  = 1
    rcounts(1)  = 1
    sdispls(1)  = 0
    rdispls(1)  = 0
    call MPI_Type_create_subarray(1, (/maxSize/), &
                                     (/maxSize/), &
                                     (/0/), &
                                     MPI_ORDER_FORTRAN,MPI_DOUBLE_PRECISION, &
                                     types(1),ierr)
    call MPI_Type_commit(types(1),ierr)
  else
    ! Send N times one double precision
    do ii = 1, maxSize
      scounts(ii)  = 1
      rcounts(ii)  = 1
      sdispls(ii)  = ii-1
      rdispls(ii)  = ii-1
      types(ii)    = MPI_DOUBLE_PRECISION
    enddo
  endif

  call MPI_Ibarrier(comm, req, ierr)
  call MPI_Wait(req, MPI_STATUS_IGNORE, ierr)

  if (switch) then
    call MPI_Ialltoallw(sbuf, scounts, sdispls, types, &
                        rbuf, rcounts, rdispls, types, &
                        comm, req, ierr)
    call MPI_Wait(req, MPI_STATUS_IGNORE, ierr)
    call MPI_TYPE_FREE(types(1), ierr)
  else
    call MPI_alltoallw(sbuf, scounts, sdispls, types, &
                       rbuf, rcounts, rdispls, types, &
                       comm, ierr)
  endif

  call MPI_Finalize( ierr )

end program main

Running the program on one CPU as follows

$ mpirun -np 1 valgrind --vgdb=yes --vgdb-error=0 ./a.out

Valgrind produces the following error

==249074== Invalid read of size 8
==249074==    at 0x4EB0A6D: release_vecs_callback (coll_base_util.c:222)
==249074==    by 0x4EB100A: complete_vecs_callback (coll_base_util.c:245)
==249074==    by 0x74AD1CC: ompi_request_complete (request.h:441)
==249074==    by 0x74AE86D: ompi_coll_libnbc_progress (coll_libnbc_component.c:466)
==249074==    by 0x4FC0C39: opal_progress (opal_progress.c:231)
==249074==    by 0x4E04795: ompi_request_wait_completion (request.h:415)
==249074==    by 0x4E047EB: ompi_request_default_wait (req_wait.c:42)
==249074==    by 0x4E80AF7: PMPI_Wait (pwait.c:74)
==249074==    by 0x48A30D2: mpi_wait (pwait_f.c:76)
==249074==    by 0x10961A: MAIN__ (tmp.f90:61)
==249074==    by 0x1096C6: main (tmp.f90:7)
==249074==  Address 0x7758830 is 0 bytes inside a block of size 8 free'd
==249074==    at 0x483CA3F: free (vg_replace_malloc.c:540)
==249074==    by 0x4899CCC: PMPI_IALLTOALLW (pialltoallw_f.c:125)
==249074==    by 0x1095FC: MAIN__ (tmp.f90:61)
==249074==    by 0x1096C6: main (tmp.f90:7)
==249074==  Block was alloc'd at
==249074==    at 0x483B7F3: malloc (vg_replace_malloc.c:309)
==249074==    by 0x4899B4A: PMPI_IALLTOALLW (pialltoallw_f.c:90)
==249074==    by 0x1095FC: MAIN__ (tmp.f90:61)
==249074==    by 0x1096C6: main (tmp.f90:7)

gdb produces the following error

Thread 1 received signal SIGTRAP, Trace/breakpoint trap.
0x0000000004eb0a6d in release_vecs_callback (request=0x7758af8) at ../../../../openmpi-4.1.0/ompi/mca/coll/base/coll_base_util.c:222
222             if (NULL != request->data.vecs.stypes[i]) {
(gdb) bt
#0  0x0000000004eb0a6d in release_vecs_callback (request=0x7758af8) at ../../../../openmpi-4.1.0/ompi/mca/coll/base/coll_base_util.c:222
#1  0x0000000004eb100b in complete_vecs_callback (req=0x7758af8) at ../../../../openmpi-4.1.0/ompi/mca/coll/base/coll_base_util.c:245
#2  0x00000000074ad1cd in ompi_request_complete (request=0x7758af8, with_signal=true) at ../../../../../openmpi-4.1.0/ompi/request/request.h:441
#3  0x00000000074ae86e in ompi_coll_libnbc_progress () at ../../../../../openmpi-4.1.0/ompi/mca/coll/libnbc/coll_libnbc_component.c:466
#4  0x0000000004fc0c3a in opal_progress () at ../../openmpi-4.1.0/opal/runtime/opal_progress.c:231
#5  0x0000000004e04796 in ompi_request_wait_completion (req=0x7758af8) at ../../openmpi-4.1.0/ompi/request/request.h:415
#6  0x0000000004e047ec in ompi_request_default_wait (req_ptr=0x1ffeffdbb8, status=0x1ffeffdbc0) at ../../openmpi-4.1.0/ompi/request/req_wait.c:42
#7  0x0000000004e80af8 in PMPI_Wait (request=0x1ffeffdbb8, status=0x1ffeffdbc0) at pwait.c:74
#8  0x00000000048a30d3 in ompi_wait_f (request=0x1ffeffe6cc, status=0x10c0a0 <mpi_fortran_status_ignore_>, ierr=0x1ffeffeee0) at pwait_f.c:76
#9  0x000000000010961b in MAIN__ () at tmp.f90:61

[ggouaillardet]

In a nutshell, in ompi/mpi/fortran/mpif-h/ialltoallw_f.c:

    c_recvtypes = (MPI_Datatype *) malloc(size * sizeof(MPI_Datatype));
    [...]
    c_ierr = PMPI_Ialltoallw(sendbuf,
                            OMPI_ARRAY_NAME_CONVERT(sendcounts),
                            OMPI_ARRAY_NAME_CONVERT(sdispls),
                            c_sendtypes,
                            recvbuf,
                            OMPI_ARRAY_NAME_CONVERT(recvcounts),
                            OMPI_ARRAY_NAME_CONVERT(rdispls),
                            c_recvtypes, c_comm, &c_request);
    [...]
    free(c_recvtypes);

c_recvtype (and other variables such as c_sendtypes and even more if INTEGER is larger than int) is a temporary array allocated by the mpif-h layer, that is passed to PMPI_Ialltoallw()
This is currently deallocated when PMPI_Ialltoallw() returns but this is wrong: it can only be deallocated when the request completes.

A temporary workaround is to comment out the free() invocations, but keep in mind this does create a memory leak.

I am having a hard time finding an elegant fix for this issue :-(

[mathrack]

Some kind of structure keeping the c_request, c_sendtypes and c_recvtypes would be needed ? The call to mpi_wait or mpi_waitall would then be able to use such a structure to deallocate the arrays at the right time ? It looks like a relatively large modification indeed.

[ggouaillardet]

That's right. The trick is these structure are in the Fortran layer, but the real work is performed by MPI_Ialltoallw() (written in C). So we would have to get the request returned by the C layer, add the pointers to be freed from the Fortran layer ... and because evil is in the detail, do that avoiding any race conditions that would result in a memory leak ...

[mathrack]

Any chance that we could fix this in the mpi_f08 fortran / C interface to exchange mpi_comm, mpi_datatype and mpi_request directly ?

Currently, when using the mpi_f08 fortran / C interface, the mpi_comm, mpi_datatype and mpi_request are converted to integer arrays using xxx%MPI_VAL.

I have read somewhere that they are bind(C) so we could exchange them directly through pointers ?

[ggouaillardet]

That is a good point!

That would not address the general INTEGER to/from int when a Fortran INTEGER is 8 bytes, but this is not an issue in this case.

In its current implementation, most mpi_f08 bindings invoke (oversimplified) a mpif-h binding, but it does not have to be this way.

@jsquyres do you remember why we chose to implement type(MPI_Datatype) and friends with an integer :: MPI_VAL (and use other Fortran bindings) and not a type(C_PTR) :: MPI_VAL (and use C bindings) ?

[mathrack]

Hello, is there any chance that the present issue will be tagged so that :

• It is recognized as a bug in the fortran layer
• Users have more visibility if solving this bug is planned for a given release

Thanks

[jsquyres]

@mathrack Yes, this was definitely missed because it was not tagged / milestoned. 😦

@ggouaillardet The MPI spec mandates that for an mpi_f08 module handle, the MPI_VAL inside must also be able to be used as the handle to the same underlying object with the mpif.h and mpi module interfaces. That's why it's implemented that way. We didn't use pointers for the mpif.h/mpi module handle values because INTEGER defaults to 32 bits.

Looks like we have the same problem in MPI_Ineighbor_alltoallw. This bug has been around for quite a while (it was there in 4.0.0 -- I didn't check farther back, but I'll bet it's been there since the non-blocking collectives were introduced). ☹️

[ggouaillardet]

@jsquyres thanks for the insight!

I also think the bug has always been there :-(
I will try to implement a fix, but no ETA at this stage.

[jsquyres]

FYI @bosilca: Got any brilliant ideas on how to solve this one?