Compilation problem with atomics on PPC64 and PGI 16.x

[gvallee]

CCLD     opal_wrapper
../../../opal/.libs/libopen-pal.a(opal_progress.o): In function `_opal_progress_unregister':
/autofs/nccs-svm1_sw/gvh/src/openmpi-2.0.1rc2/opal/runtime/opal_progress.c:459: undefined reference to `opal_atomic_swap_64'
../../../opal/.libs/libopen-pal.a(opal_progress.o): In function `_opal_progress_register':
/autofs/nccs-svm1_sw/gvh/src/openmpi-2.0.1rc2/opal/runtime/opal_progress.c:398: undefined reference to `opal_atomic_swap_64'
make[2]: *** [opal_wrapper] Error 2
make[2]: Leaving directory `/autofs/nccs-svm1_sw/gvh/src/openmpi-2.0.1rc2/opal/tools/wrappers'
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory `/autofs/nccs-svm1_sw/gvh/src/openmpi-2.0.1rc2/opal'
make: *** [all-recursive] Error 1

$ nm libopen-pal.a  | grep atomic
                U opal_atomic_cmpset_64
0000000000000ab0 t opal_atomic_cmpset_ptr
                U opal_atomic_wmb
0000000000000950 t opal_lifo_push_atomic
                U opal_atomic_cmpset_acq_32
00000000000003d0 t opal_atomic_lock
0000000000000450 t opal_atomic_unlock
                U opal_atomic_wmb
                U opal_atomic_ll_64
                U opal_atomic_sc_64
                U opal_atomic_wmb
0000000000001010 t opal_lifo_pop_atomic
                U opal_atomic_cmpset_acq_32
00000000000004b0 t opal_atomic_init
00000000000004e0 t opal_atomic_lock
                U opal_atomic_mb
0000000000000560 t opal_atomic_unlock
                U opal_atomic_wmb
                U opal_atomic_add_32
                U opal_atomic_cmpset_acq_32
0000000000000820 t opal_atomic_init
0000000000000850 t opal_atomic_lock
                U opal_atomic_sub_32
                U opal_atomic_swap_64
00000000000008d0 t opal_atomic_unlock
                U opal_atomic_wmb
0000000000000130 t opal_atomic_init
atomic-asm.o:
0000000000000138 T opal_atomic_add_32
0000000000000018 T opal_atomic_cmpset_32
00000000000000c4 T opal_atomic_cmpset_64
000000000000003c T opal_atomic_cmpset_acq_32
00000000000000e8 T opal_atomic_cmpset_acq_64
0000000000000070 T opal_atomic_cmpset_rel_32
0000000000000110 T opal_atomic_cmpset_rel_64
0000000000000000 T opal_atomic_mb
0000000000000008 T opal_atomic_rmb
0000000000000150 T opal_atomic_sub_32
0000000000000010 T opal_atomic_wmb
0000000000002280 t mca_base_pvar_is_atomic
                U opal_atomic_ll_64
                U opal_atomic_sc_64
                U opal_atomic_wmb
0000000000000900 t opal_lifo_pop_atomic

All the output from configure and make, as well as config.log, is in the attached file.

ompi-output.zip

[jsquyres]

@sjeaugey @hjelmn can you have a look at this? Thanks.

[gvallee]

FYI, I checked with both 16.5 and 16.7 and face that problem with both.

[bosilca]

Somehow you are using our hand-generated assembly (opal/asm/base/POWERPC64) and this file has not beed updated to include support for swap. I would propose the following patch (not tested as I have no access to a PPC64):

diff --git a/opal/asm/base/POWERPC64.asm b/opal/asm/base/POWERPC64.asm
index 28da3f4..13d071d 100644
--- a/opal/asm/base/POWERPC64.asm
+++ b/opal/asm/base/POWERPC64.asm
@@ -120,6 +120,21 @@ START_FUNC(opal_atomic_cmpset_rel_64)
 END_FUNC(opal_atomic_cmpset_rel_64)


+START_FUNC(opal_atomic_swap_64)
+       LSYM(12) ldarx   r0, 0, r3
+           stdcx.  r5, 0, r3
+          bne-    REFLSYM(12)
+       blr
+END_FUNC(opal_atomic_swap_64)
+
+START_FUNC(opal_atomic_swap_32)
+       LSYM(13) lwarx   r0, 0, r3
+           stwcx.  r5, 0, r3
+          bne-    REFLSYM(13)
+       blr
+END_FUNC(opal_atomic_swap_64)
+
+
 START_FUNC(opal_atomic_add_32)
        LSYM(5) lwarx r0, 0, r3
             add  r0, r4, r0

[PHHargrove]

I now have PGI-16.7-beta on PPC64el up and running.
I can reproduce a failure similar to Geoffroy's original report of an undefined reference to swap_64, but additionally I see undefined references to ll_64 and sc_64:

PGC/power Linux 16.7-beta: compilation completed with warnings
/bin/bash ../../../libtool  --tag=CC   --mode=link pgcc  -g    -o opal_wrapper opal_wrapper.o ../../../opal/libopen-pal.la -lrt -lutil
libtool: link: pgcc -g -o .libs/opal_wrapper opal_wrapper.o  ../../../opal/.libs/libopen-pal.so -lm -lrt -lutil -Wl,-rpath -Wl,/home/ubuntu/OMPI/openmpi-2.0.1rc3-linux-ppc64el-pgi-16/INST/lib
../../../opal/.libs/libopen-pal.so: undefined reference to `opal_atomic_sc_64'
../../../opal/.libs/libopen-pal.so: undefined reference to `opal_atomic_ll_64'
../../../opal/.libs/libopen-pal.so: undefined reference to `opal_atomic_swap_64'
Makefile:1778: recipe for target 'opal_wrapper' failed

The good news is that George's patch resolves the swap_64 issue.

So if @bosilca (or @hjelmn ?) can provide the POWERPS64.asm bits for LL and SC (32 and 64), then this can probably be resolved quickly.

I can provide access to a Power8E VM with the subject compiler installed if that helps.

[jsquyres]

@gpaulsen @jjhursey If this is a POWER-related issue, IBM -- please chime in here. Thanks.

[bosilca]

Indeed these functions are also missing from our manually generated assembly file.

[jjhursey]

Adding link to this email thread on the issue:
https://mail-archive.com/devel@lists.open-mpi.org/msg19514.html

A couple other related items, for reference:

• PowerPC/Power lifo/fifo improvements #720
• Move powerpc atomics fixes and improvements to v2.x ompi-release#856

[hjelmn]

Not sure why this should be an issue. Doesn't PGI support inline assembly? If not how did PGI 15.x work for Paul?

[PHHargrove]

Not sure why this should be an issue. Doesn't PGI support inline assembly?

The configure output provided by Geoffroy says it does not.

If not how did PGI 15.x work for Paul?

Because when I responded on the devel list I had totally missed that this was PPC64 and was reporting my success with various PGI versions on x86-64 platforms.

-Paul

[hjelmn]

@PHHargrove The config.log from the failed run would be useful. We don't have PGI 16.x and probably never will.