master opal_init() failing on Absoft 32 bit MTT dlopen_test

[jsquyres]

Absoft's MTT has been failing on the dlopen_test (during make check) in opal_init() on master recently (e.g., see https://mtt.open-mpi.org/index.php?do_redir=2438).

Here's a backtrace provided by @cagoelz (note the assertion failure in opal_pointer_array_add()):

Reading symbols from /home/ompitest/scratches/2017-05-05/mpi-install/8HsK/src/openmpi-master-201705050239-eb03679/ompi/debuggers/.libs/lt-dlopen_test...done.
(gdb) run
Starting program: /home/ompitest/scratches/2017-05-05/mpi-install/8HsK/src/openmpi-master-201705050239-eb03679/ompi/debuggers/.libs/lt-dlopen_test
[Thread debugging using libthread_db enabled]
lt-dlopen_test: class/opal_pointer_array.c:241: opal_pointer_array_add: Assertion `0 == (table->free_bits[__b_idx] & (1UL << __b_pos))' failed.

Program received signal SIGABRT, Aborted.
0x00110402 in __kernel_vsyscall ()
(gdb) bt
#0  0x00110402 in __kernel_vsyscall ()
#1  0x00733b10 in raise () from /lib/libc.so.6
#2  0x00735421 in abort () from /lib/libc.so.6
#3  0x0072cf6b in __assert_fail () from /lib/libc.so.6
#4  0x00399ace in opal_pointer_array_add (table=0x482940, ptr=0x80637f0)
   at class/opal_pointer_array.c:241
#5  0x003d3e13 in register_variable (project_name=0x45b094 "opal",
   framework_name=0x45b090 "dss", component_name=0x0,
   variable_name=0x45b1a9 "buffer_threshold_size", description=0x0,
   type=MCA_BASE_VAR_TYPE_INT, enumerator=0x0, bind=0,
   flags=MCA_BASE_VAR_FLAG_SETTABLE, info_lvl=OPAL_INFO_LVL_8,
   scope=MCA_BASE_VAR_SCOPE_ALL_EQ, synonym_for=-1, storage=0x47f928)
   at mca_base_var.c:1385
#6  0x003d4649 in mca_base_var_register (project_name=0x45b094 "opal",
   framework_name=0x45b090 "dss", component_name=0x0,
   variable_name=0x45b1a9 "buffer_threshold_size", description=0x0,
   type=MCA_BASE_VAR_TYPE_INT, enumerator=0x0, bind=0,
   flags=MCA_BASE_VAR_FLAG_SETTABLE, info_lvl=OPAL_INFO_LVL_8,
   scope=MCA_BASE_VAR_SCOPE_ALL_EQ, storage=0x47f928) at mca_base_var.c:1495
#7  0x003b582f in opal_dss_register_vars () at dss/dss_open_close.c:286
#8  0x0039fe25 in opal_register_params () at runtime/opal_params.c:357
#9  0x0039ec92 in opal_init_util (pargc=0xbfffead0, pargv=0xbfffead4)
   at runtime/opal_init.c:422
#10 0x0039eeb1 in opal_init (pargc=0xbfffead0, pargv=0xbfffead4)
---Type <return> to continue, or q <return> to quit---
   at runtime/opal_init.c:513
#11 0x08048cee in main (argc=Cannot access memory at address 0x77fc) at dlopen_test.c:133
(gdb)

This is a 32 bit build, and we explicitly invoke opal_init() in this test. Are we invoking it incorrectly? Or is something legitimately busted in opal_init() in 32 bit builds?

[jsquyres]

Note that I am unable to reproduce this issue manually on a 64 bit Linux machine/OS/kernel (RHEL 6.x), compiling with CFLAGS=-m32 CXXFLAGS=-m32 FCFLAGS=-m32. I do not have any actual ia32 machines with which to test.

[bosilca]

I have a gcc compiled version of OMPI running on an 32 bits rpi2, and I don't see this assert.

[ggouaillardet]

In SET_BIT assert, we mix uint64_t (free_bits) and 1UL
Should we replace the latter with (uint64_t)1 ?
I will not be able to work on that issue this week...

[bosilca]

Nice catch @ggouaillardet. Based on the warnings printed during compilation it appears that on this architecture "unsigned long" is smaller than size_t, so we clearly need a uint64_t instead of the "1UL" (or things gets truncated). I made a patch (ba46b35), let's see if it fixes the issue.

[jsquyres]

@bosilca Commit ba46b35 made it compile on Absoft 32 bit platforms, but it still failed at run time. Here's some MTT results showing the new assertion failure:

mpirun: class/pmix_pointer_array.c:290: pmix_pointer_array_set_item: Assertion `0 ==
(table->free_bits[__b_idx] & (1UL << __b_pos))' failed.
[oldjoe:29438] *** Process received signal ***
[oldjoe:29438] Signal: Aborted (6)
[oldjoe:29438] [ 0] [0x55a440]
[oldjoe:29438] [ 1] /lib/libc.so.6(abort+0x101)[0x735421]
[oldjoe:29438] [ 2] /lib/libc.so.6(__assert_fail+0xfb)[0x72cf6b]
[oldjoe:29438] [ 3] /home/ompitest/scratches/2017-06-06/installs/ijxR/install/lib/openmpi/mca_pmix_pmix2x.so(pmix_pointer_array_set_item+0x288)[0x5d1886]
[oldjoe:29438] [ 4] /home/ompitest/scratches/2017-06-06/installs/ijxR/install/lib/openmpi/mca_pmix_pmix2x.so(pmix_bfrop_open+0x17a7)[0x65ee97]
[oldjoe:29438] [ 5] /home/ompitest/scratches/2017-06-06/installs/ijxR/install/lib/openmpi/mca_pmix_pmix2x.so(pmix_rte_init+0x5be)[0x6750be]
[oldjoe:29438] [ 6] /home/ompitest/scratches/2017-06-06/installs/ijxR/install/lib/openmpi/mca_pmix_pmix2x.so(OPAL_MCA_PMIX2X_PMIx_server_init+0xa9)[0x61d9cc]
[oldjoe:29438] [ 8] /home/ompitest/scratches/2017-06-06/installs/ijxR/install/lib/libopen-rte.so.0(pmix_server_init+0x76b)[0x2c14fc]
/home/ompitest/scratches/2017-06-06/installs/ijxR/install/lib/openmpi/mca_ess_hnp.so[0x4e01ab]
[oldjoe:29438] [10] /home/ompitest/scratches/2017-06-06/installs/ijxR/install/lib/libopen-rte.so.0(orte_init+0x324)[0x27261f]
[oldjoe:29438] [11] /home/ompitest/scratches/2017-06-06/installs/ijxR/install/lib/libopen-rte.so.0(orte_submit_init+0x1283)[0x2b6a86]
[oldjoe:29438] [12] mpirun(orte_daemon_recv+0x4ad)[0x8048e3d]
[oldjoe:29438] [13] mpirun(orte_daemon_recv+0x196)[0x8048b26]
[oldjoe:29438] [14] /lib/libc.so.6(__libc_start_main+0xdc)[0x720edc]
[15] mpirun(orte_daemon_recv+0xc1)[0x8048a51]

[rhc54]

I need to upstream the corresponding change to PMIx and then bring that back to OMPI. I'm surprised it would compile - it's the same code in PMIx that was in OPAL.

[jsquyres]

@rhc54 Ah, I missed that this was duplicate code in PMIx. Note that @bosilca and @ggouaillardet made a recent code change in this area in OPAL -- that's what started our journey here. So make sure to get both commits (i.e., both 3c6631f and ba46b35).

[rhc54]

ok, working

[rhc54]

#3658

[jsquyres]

This has been fixed.