v3.0.0rc1: XLC-13.1 odd failure running ring_c

[PHHargrove]

I have configured on a Linux/ppc64 system with xlc-13.1 as follows:

[path-to]/configure --prefix=[...] --enable-debug CC=xlc CXX=xlC FC=xlf \
        CFLAGS=-q64 --with-wrapper-cflags=-q64 \
        CXXFLAGS=-q64 --with-wrapper-cxxflags=-q64 \
        FCFLAGS=-q64 --with-wrapper-fcflags=-q64 --disable-oshmem-fortran

This build was previously failing due to #3811, but a patch from @ggouaillardet gets me past that and on to the next problem:

$ mpirun -mca btl sm,self -np 2 examples/ring_c'
[login2:71837] Component file data does not match filename: /gpfs-biou/phh1/OMPI/openmpi-3.0.0rc1-linux-ppc64-xlc-13.1/INST/lib/pmix/mca_ptl_tcp (ptl / tcp) != ptl  -- ignored
[login2:71842] Component file data does not match filename: /gpfs-biou/phh1/OMPI/openmpi-3.0.0rc1-linux-ppc64-xlc-13.1/INST/lib/pmix/mca_ptl_tcp (ptl / tcp) != ptl  -- ignored
[login2:71843] Component file data does not match filename: /gpfs-biou/phh1/OMPI/openmpi-3.0.0rc1-linux-ppc64-xlc-13.1/INST/lib/pmix/mca_ptl_tcp (ptl / tcp) != ptl  -- ignored
[login2:71842] *** Process received signal ***
[login2:71842] Signal: Segmentation fault (11)
[login2:71842] Signal code: Address not mapped (1)
[login2:71842] Failing at address: 0xdeafbeeddeafbf35
[login2:71842] [ 0] [0xfffad2e0448]
[login2:71842] [ 1] [0x0]
[login2:71842] [ 2] /gpfs-biou/phh1/OMPI/openmpi-3.0.0rc1-linux-ppc64-xlc-13.1/INST/lib/libopen-pal.so.40(+0x108fd8)[0xfffac858fd8]
[login2:71842] [ 3] [login2:71843] *** Process received signal ***
[login2:71843] Signal: Segmentation fault (11)
[login2:71843] Signal code: Address not mapped (1)
[login2:71843] Failing at address: 0xdeafbeeddeafbf35
[login2:71843] [ 0] [0xfffaad60448]
[login2:71843] [ 1] [0x0]

Note that the failing address of 0xdeafbeeddeafbf35 is suspiciously similar to
./opal/class/opal_object.h:#define OPAL_OBJ_MAGIC_ID ((0xdeafbeedULL << 32) + 0xdeafbeedULL)

From gdb on a core (different run than output above, may not match exactly):

Core was generated by `examples/ring_c '.
Program terminated with signal 11, Segmentation fault.
#0  0x00000fffa46fc0b4 in pmix_ptl_base_recv_handler (sd=14, flags=2, cbdata=0xfffa4798c78)
    at /gpfs-biou/phh1/OMPI/openmpi-3.0.0rc1-linux-ppc64-xlc-13.1/openmpi-3.0.0rc1/opal/mca/pmix/pmix2x/pmix/src/mca/ptl/base/ptl_base_sendrecv.c:401
401                             (NULL == peer) ? "NULL" : peer->info->nptr->nspace,
(gdb) where
#0  0x00000fffa46fc0b4 in pmix_ptl_base_recv_handler (sd=14, flags=2, cbdata=0xfffa4798c78)
    at /gpfs-biou/phh1/OMPI/openmpi-3.0.0rc1-linux-ppc64-xlc-13.1/openmpi-3.0.0rc1/opal/mca/pmix/pmix2x/pmix/src/mca/ptl/base/ptl_base_sendrecv.c:401
#1  0x00000fffa53b8fd8 in .event_persist_closure ()
   from /gpfs-biou/phh1/OMPI/openmpi-3.0.0rc1-linux-ppc64-xlc-13.1/INST/lib/libopen-pal.so.40
#2  0x00000fffa53b9354 in .event_process_active_single_queue ()
   from /gpfs-biou/phh1/OMPI/openmpi-3.0.0rc1-linux-ppc64-xlc-13.1/INST/lib/libopen-pal.so.40
#3  0x00000fffa53b9738 in .event_process_active ()
   from /gpfs-biou/phh1/OMPI/openmpi-3.0.0rc1-linux-ppc64-xlc-13.1/INST/lib/libopen-pal.so.40
#4  0x00000fffa53ba5e0 in .opal_libevent2022_event_base_loop ()
   from /gpfs-biou/phh1/OMPI/openmpi-3.0.0rc1-linux-ppc64-xlc-13.1/INST/lib/libopen-pal.so.40
#5  0x00000fffa46d1d4c in progress_engine (obj=0x100314d1ef8)
    at /gpfs-biou/phh1/OMPI/openmpi-3.0.0rc1-linux-ppc64-xlc-13.1/openmpi-3.0.0rc1/opal/mca/pmix/pmix2x/pmix/src/runtime/pmix_progress_threads.c:109
#6  0x000000800cdac5dc in .start_thread () from /lib64/libpthread.so.0
#7  0x000000800ccda9bc in .__clone () from /lib64/libc.so.6

[ggouaillardet]

the first warning is a red flag.
can you please attach your /gpfs-biou/phh1/OMPI/openmpi-3.0.0rc1-linux-ppc64-xlc-13.1/INST/lib/pmix/mca_ptl_tcp (the binary, i do not know which extension it has on ppc64) ?
the name (tcp should have been statically initialized, just like the type (ptl) but it was not ...
that sounds pretty odd to me ...

[PHHargrove]

@ggouaillardet
Here is mca_ptl_tcp.so, which I assume is the binary file you are asking for.
It is wrapped up inside mca_ptl_tcp.zip to satisfy github.

[ggouaillardet]

thanks, the binary looks bad to me
for example

  char pmix_mca_component_name[PMIX_MCA_BASE_MAX_COMPONENT_NAME_LEN + 1];
  /**< This comopnent's string name. */
  int pmix_mca_component_major_version;
  /**< This component's major version number. */
  int pmix_mca_component_minor_version;
  /**< This component's minor version number. */
  int pmix_mca_component_release_version;
  /**< This component's release version number. */

all these fields are zero'ed.
they should have been

"tcp",
3,
0,
0

this makes very little sense to me ... can you simple preprocess ptl_tcp_component.c so maybe we can figure out what is going wrong ?
(a simple way to achieve this is

cd opal/mca/pmix/pmix2x/pmix/src/mca/ptl/tcp
make clean
make V=1 ptl_tcp_component.lo

so you can get the full compile command.
and then simply copy/paste it and replace -c with -E

[PHHargrove]

@ggouaillardet
preprocess look normal:

#line 76
  pmix_ptl_tcp_component_t mca_ptl_tcp_component = {
    .super = {
        .base = {
            .pmix_mca_major_version = 2, .pmix_mca_minor_version = 1, .pmix_mca_release_version = 0, .pmix_mca_project_name = "pmix", .pmix_mca_project_major_version = 3, .pmix_mca_project_minor_version = 0, .pmix_mca_project_release_version = 0, .pmix_mca_type_name = "ptl", .pmix_mca_type_major_version = 1, .pmix_mca_type_minor_version = 0, .pmix_mca_type_release_version = 0,
#line 82
            .pmix_mca_component_name = "tcp",
#line 86
.pmix_mca_component_major_version = 3, .pmix_mca_component_minor_version = 0, .pmix_mca_component_release_version = 0,
#line 89
            .pmix_mca_open_component = component_open,
            .pmix_mca_close_component = component_close,
            .pmix_mca_register_component_params = component_register,
            .pmix_mca_query_component = component_query
        },
        .priority = 30,
        .uri = ((void *)0),
        .setup_listener = setup_listener
    },
    .tmpdir = ((void *)0),
    .if_include = ((void *)0),
    .if_exclude = ((void *)0),
    .ipv4_port = 0,
    .ipv6_port = 0,
    .disable_ipv4_family = 0,
    .disable_ipv6_family = 1,
    .filename = ((void *)0)
};

[PHHargrove]

I am now rebuilding from scratch to see if this was something random.

[PHHargrove]

I am now rebuilding from scratch to see if this was something random.

Failure is exactly the same after removing the build directory and starting again from configure.

[rhc54]

@jjhursey Can you folks replicate this?

[PHHargrove]

@jjhursey
FYI, this is on a Power7 system where the admins (if they deserve that title) have apparently never applied any updates:

$ xlc -qversion
IBM XL C/C++ for Linux, V13.1
Version: 13.01.0000.0000

[jjhursey]

Yeah I can take a look on Monday when I get back in the office. I have access to a couple Power7 boxes. I don't remember what XL compiler version they are running though.