Memory Leaks in pmix4x & mpool

[ax3l]

Background information

What version of Open MPI are you using?

OpenMPI master as of 3a4a1f9

Describe how Open MPI was installed

via spack with:

spack install openmpi@develop %[email protected] cflags="-g" cxxflags="-g"
spack load --dependencies openmpi@develop %[email protected] cflags="-g" cxxflags="-g"

Performing the test below with Clang, e.g. v6.0.0, shows the same results.

Please describe the system on which you are running

• Operating system/version: Debian GNU/Linux 9.6 (stretch)
• Computer hardware: Lenovo Thinkpad
• Network type: Ethernet

---

Details of the problem

I am trying to run CI on downstream software (libs and apps) on which I would like to constantly enable address and memory sanitizers. Unfortunately, OpenMPI reports a lot of (probably valid) memory leaks, tested with with recent versions of GCC and Clang.

Use the following snippet:

#include <mpi.h>

int main(int, char**) {
    MPI_Init(nullptr, nullptr);
    MPI_Finalize();
    return 0;
}

And compile it with:

mpic++ -std=c++11 main.cpp -fsanitize=address

When running it with:

mpirun -np 2 ./a.out

We will see the following issues in pmix_control.c:72 and mpool_hugepage_component.c:150

=================================================================
==24815==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 912 byte(s) in 2 object(s) allocated from:
    #0 0x7f67bd867d28 in malloc (/usr/lib/x86_64-linux-gnu/libasan.so.3+0xc1d28)
    #1 0x7f67bb850a4f in pmix_obj_new /tmp/axel/spack-stage/spack-stage-wykkr4qu/ompi/opal/mca/pmix/pmix4x/pmix/src/class/pmix_object.h:475
    #2 0x7f67bb850a4f in query_cbfunc common/pmix_control.c:72

Direct leak of 64 byte(s) in 1 object(s) allocated from:
    #0 0x7f67bd867d28 in malloc (/usr/lib/x86_64-linux-gnu/libasan.so.3+0xc1d28)
    #1 0x7f67bb7c073c in opal_obj_new /home/axel/src/spack/var/spack/stage/openmpi-develop-ggblyddjtjurkz72s2wro4fdkfhgoc73/ompi/opal/class/opal_object.h:486
    #2 0x7f67bb7c073c in mca_mpool_hugepage_find_hugepages /tmp/axel/spack-stage/spack-stage-wykkr4qu/ompi/opal/mca/mpool/hugepage/mpool_hugepage_component.c:260
    #3 0x7f67bb7c073c in mca_mpool_hugepage_open /tmp/axel/spack-stage/spack-stage-wykkr4qu/ompi/opal/mca/mpool/hugepage/mpool_hugepage_component.c:150

Indirect leak of 15 byte(s) in 1 object(s) allocated from:
    #0 0x7f67bd867d28 in malloc (/usr/lib/x86_64-linux-gnu/libasan.so.3+0xc1d28)
    #1 0x7f67bc3163b9 in __strdup (/lib/x86_64-linux-gnu/libc.so.6+0x803b9)

SUMMARY: AddressSanitizer: 991 byte(s) leaked in 4 allocation(s).

The memory leak is visible in:

• clang's address sanitizer
• gcc's address sanitizer
• valgrind

[rhc54]

FWIW: the PMIx community has been working on eliminating memory leaks in PMIx, but that work is done in the PMIx repository. It only gets reflected in OMPI once someone updates the OMPI-embedded version of the code. This happens infrequently.

You might have better luck if you configure OMPI against an external copy of PMIx so you can keep up with the upstream changes.

[ax3l]

I tried to find the PMIx repo, is it public and has an issue tracker?

Where is mpool/hugepage developed?

[ax3l]

Ah, here it is: https://github.com/pmix/pmix/issues
Since you author both, can you move this upstream? I don't know how to work with or test PMIx.

[rhc54]

I can try, but the problem is that you are looking at a rather stale version of PMIx and so those lines may not even exist any more. If you build OMPI against the PMIx master branch, then any memory leaks you find will at least reference current code and can be dealt with accordingly.

[rhc54]

mpool/hugepage is developed in OMPI, so those issues belong here