Regression: UCX osc: dynamic windows broken on master

[devreal]

I noticed that support for dynamic windows is broken in the UCX osc on master. The module's worker pool memory (module->mem) is NULL if it's a dynamic windows but upon flush and window-free this pointer is accessed unconditionally, inadvertently leading to a Segfault. Upon communication, the type of window is checked and the dynamic window info is populated (in get_dynamic_win_info) but never used anywhere, so that will eventually Segfault, too. I'm aware that the UCX osc in master is under development so I just open this issue for tracking purposes.

[artpol84]

@devreal, thanks for the heads up

[devreal]

I just came back to this issue and found that it is still broken. This is actually a regression that exists in 5.0 and master.

Consider the following example:

#include <mpi.h>
#include <stdio.h>
#include <stdlib.h>
#include <assert.h>

#define NUM_REPS 1000

static void
allocate_dynamic(size_t elsize, size_t count, MPI_Win *win, MPI_Aint *disp_set, char **b)
{
    char *base;
    MPI_Aint disp;
    int rank;
    MPI_Comm_rank(MPI_COMM_WORLD, &rank);
    
    if (MPI_Win_create_dynamic(MPI_INFO_NULL, MPI_COMM_WORLD, win) != MPI_SUCCESS) {
        printf("Failed to create dynamic window!\n");
        exit(1);
    }

    if (MPI_Alloc_mem(elsize*count, MPI_INFO_NULL, &base) != MPI_SUCCESS) {
        printf("Failed to allocate memory!\n");
        exit(1);
    }


    if (MPI_Win_attach(*win, base, elsize*count) != MPI_SUCCESS) {
        printf("Failed to attach memory to dynamic window!\n");
        exit(1);
    }


    MPI_Get_address(base, &disp);
    printf("Offset at process %i: %p (%lu)\n", rank, base, disp);
    MPI_Allgather(&disp, 1, MPI_AINT, disp_set, 1, MPI_AINT, MPI_COMM_WORLD);

    *b = base;
}

int main(int argc, char **argv) 
{
  int *baseptr;
  MPI_Win win;
  int rank, size;

  MPI_Init(&argc, &argv);

  MPI_Comm_size(MPI_COMM_WORLD, &size);
  MPI_Comm_rank(MPI_COMM_WORLD, &rank);

  MPI_Aint *disp_set = malloc(sizeof(MPI_Aint)*size);
  allocate_dynamic(sizeof(int), 1, &win, disp_set, &baseptr);
  MPI_Win_lock_all(0, win);

  *baseptr = rank;
  MPI_Barrier(MPI_COMM_WORLD); 

  if (rank == 1) {
    MPI_Request req;
    int val;
    for (int i = 0; i < NUM_REPS; ++i) {
      MPI_Rget(&val, 1, MPI_INT, 0, disp_set[0], 1, MPI_INT, win, &req);
      MPI_Wait(&req, MPI_STATUS_IGNORE);
      assert(val == 0);
    }
  }

  MPI_Barrier(MPI_COMM_WORLD);
  MPI_Win_unlock_all(win);
  MPI_Win_free(&win);

  MPI_Finalize();

  return 0;
}

I get a Segfault with the following backtrace:

#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1  0x00007ffff7c98859 in __GI_abort () at abort.c:79
#2  0x00007ffff7d033ee in __libc_message (action=action@entry=do_abort, fmt=fmt@entry=0x7ffff7e2d285 "%s\n") at ../sysdeps/posix/libc_fatal.c:155
#3  0x00007ffff7d0b47c in malloc_printerr (str=str@entry=0x7ffff7e2b4ae "free(): invalid pointer") at malloc.c:5347
#4  0x00007ffff7d0ccac in _int_free (av=<optimized out>, p=<optimized out>, have_lock=0) at malloc.c:4173
#5  0x00007ffff5fc2cd7 in ucp_wireup_msg_progress (self=0x5555556e52c8) at ../../../src/ucp/wireup/wireup.c:136
#6  0x00007ffff5fc3002 in ucp_request_try_send (pending_flags=0, req_status=0x7fffffffcf8f, req=0x5555556e5210)
    at ucx-1.9.0/build/../src/ucp/core/ucp_request.inl:258
#7  ucp_request_send (pending_flags=0, req=0x5555556e5210) at ucx-1.9.0/build/../src/ucp/core/ucp_request.inl:258
#8  ucp_wireup_msg_send (ep=ep@entry=0x7ffff41f2000, type=type@entry=1 '\001', tl_bitmap=tl_bitmap@entry=1, lanes2remote=lanes2remote@entry=0x0)
    at ../../../src/ucp/wireup/wireup.c:195
#9  0x00007ffff5fc6a9d in ucp_wireup_send_request (ep=ep@entry=0x7ffff41f2000) at ../../../src/ucp/wireup/wireup.c:1113
#10 0x00007ffff5fc7108 in ucp_wireup_connect_remote (ep=ep@entry=0x7ffff41f2000, lane=<optimized out>) at ../../../src/ucp/wireup/wireup.c:1200
#11 0x00007ffff5f7c9dc in ucp_ep_resolve_dest_ep_ptr (lane=<optimized out>, ep=0x7ffff41f2000) at ../../../src/ucp/core/ucp_ep.inl:179
#12 ucp_rkey_resolve_inner (rkey=rkey@entry=0x555555716ec0, ep=ep@entry=0x7ffff41f2000) at ../../../src/ucp/core/ucp_rkey.c:473
#13 0x00007ffff5f7bae8 in ucp_ep_rkey_unpack (ep=ep@entry=0x7ffff41f2000, rkey_buffer=<optimized out>, rkey_p=0x55555570d740) at ../../../src/ucp/core/ucp_rkey.c:283
#14 0x00007ffff68588a9 in _tlocal_mem_create_rkey (target=1433458416, ep=<optimized out>, mem_rec=0x55555570d6f0)
    at ../../../../../opal/mca/common/ucx/common_ucx_wpool.c:750
#15 opal_common_ucx_tlocal_fetch_spath (mem=mem@entry=0x5555556e3db0, target=target@entry=0) at ../../../../../opal/mca/common/ucx/common_ucx_wpool.c:795
#16 0x00007ffff4b9ac72 in opal_common_ucx_tlocal_fetch (_winfo=<synthetic pointer>, _rkey=<synthetic pointer>, _ep=<synthetic pointer>, target=0, mem=0x5555556e3db0)
    at ../../../../../opal/mca/common/ucx/common_ucx_wpool.h:226
#17 opal_common_ucx_wpmem_putget (rem_addr=93824993820312, len=33288, buffer=0x5555556f8cc0, target=0, op=OPAL_COMMON_UCX_GET, mem=0x5555556e3db0)
    at ../../../../../opal/mca/common/ucx/common_ucx_wpool.h:373
#18 get_dynamic_win_info (remote_addr=93824993873312, module=0x5555556d7e80, target=<optimized out>) at ../../../../../ompi/mca/osc/ucx/osc_ucx_comm.c:337
#19 0x00007ffff4b9cb27 in ompi_osc_ucx_get (origin_addr=origin_addr@entry=0x7fffffffd560, origin_count=origin_count@entry=1, 
    origin_dt=origin_dt@entry=0x555555558120 <ompi_mpi_int>, target=target@entry=0, target_disp=target_disp@entry=93824993873312, target_count=target_count@entry=1, 
    target_dt=0x555555558120 <ompi_mpi_int>, win=0x5555556c5690) at ../../../../../ompi/mca/osc/ucx/osc_ucx_comm.c:553
#20 0x00007ffff4b9fbdf in ompi_osc_ucx_rget (origin_addr=<optimized out>, origin_count=<optimized out>, origin_dt=<optimized out>, target=<optimized out>, 
    target_disp=<optimized out>, target_count=<optimized out>, target_dt=<optimized out>, win=<optimized out>, request=<optimized out>)
    at ../../../../../ompi/mca/osc/ucx/osc_ucx_comm.c:1124
#21 0x00007ffff7f2e137 in PMPI_Rget (origin_addr=0x7fffffffd560, origin_count=1, origin_datatype=0x555555558120 <ompi_mpi_int>, target_rank=0, 
    target_disp=93824993873312, target_count=<optimized out>, target_datatype=0x555555558120 <ompi_mpi_int>, win=0x5555556c5690, request=0x7fffffffd578) at prget.c:82
#22 0x000055555555565b in main ()

Looking at the code, the problem still exists that the rkeys populated by get_dynamic_win_info are never actually used. I have verified that the test above succeeds in v4.0.x.

[jsquyres]

@open-mpi/ucx This came up again on the call today. Can someone please have a look?

[hoopoepg]

@janjust will you have chance to look at this issue?

[janjust]

Yes.
FWIW I thought this this stale PR #8212 is addressing this issue and several other osc issues; however, it's gone stale.
I'll rebase it, and see if it fixes it.

[hjelmn]

@janjust Please do. This bug has been open for far too long. If it can't be addressed the component should be deprioritized below osc/rdma. We shouldn't have something like this known to be broken by default for this long in Open MPI releases.