IBM MTT "make check" fail

[jsquyres]

@gpaulsen @jjhursey IBM is getting an MTT make check fail: https://mtt.open-mpi.org/index.php?do_redir=2391

The error is that the opal_fifo test is failing in 32 bit POWER on master.

[jjhursey]

Duplicate of Issue #1893 - we need to investigate. It's intermittent so hard to pin down.

here is the output from test-suite.log in case it helps:

=====================================================================
   Open MPI master-201702120422-81e57bb: test/class/test-suite.log
=====================================================================

# TOTAL: 10
# PASS:  9
# SKIP:  0
# XFAIL: 0
# FAIL:  1
# XPASS: 0
# ERROR: 0

.. contents:: :depth: 2

FAIL: opal_fifo
===============

Exhaustive atomics thread finished. Popped 168490 items. Time: 6 s 451012 us 38287 nsec/poppush
Exhaustive atomics thread finished. Popped 170672 items. Time: 6 s 498624 us 38076 nsec/poppush
Exhaustive atomics thread finished. Popped 170704 items. Time: 6 s 555723 us 38404 nsec/poppush
Exhaustive atomics thread finished. Popped 171759 items. Time: 6 s 559438 us 38189 nsec/poppush
Exhaustive atomics thread finished. Popped 167076 items. Time: 6 s 625385 us 39654 nsec/poppush
Exhaustive atomics thread finished. Popped 167749 items. Time: 6 s 661287 us 39709 nsec/poppush
Exhaustive atomics thread finished. Popped 151657 items. Time: 6 s 659123 us 43909 nsec/poppush
Exhaustive atomics thread finished. Popped 149946 items. Time: 6 s 666284 us 44457 nsec/poppush
 Failure :  fifo push/pop multi-threaded with atomics when there are insufficient items
 Failure :  fifo pop all items
SUPPORT: OMPI Test failed: opal_fifo_t (2 of 8 failed)
Single thread test. Time: 0 s 71404 us 71 nsec/poppush
Atomics thread finished. Time: 0 s 124363 us 124 nsec/poppush
Atomics thread finished. Time: 8 s 346324 us 8346 nsec/poppush
Atomics thread finished. Time: 8 s 362706 us 8362 nsec/poppush
Atomics thread finished. Time: 8 s 398374 us 8398 nsec/poppush
Atomics thread finished. Time: 8 s 403718 us 8403 nsec/poppush
Atomics thread finished. Time: 8 s 600248 us 8600 nsec/poppush
Atomics thread finished. Time: 8 s 658165 us 8658 nsec/poppush
Atomics thread finished. Time: 8 s 870449 us 8870 nsec/poppush
Atomics thread finished. Time: 8 s 859184 us 8859 nsec/poppush
All threads finished. Thread count: 8 Time: 8 s 874685 us 1109 nsec/poppush
All threads finished. Thread count: 8 Time: 6 s 671116 us 833 nsec/poppush
FAIL opal_fifo (exit status: 1)

[jjhursey]

PR #3468 might be related (might fix this issue - need to check)

[jsquyres]

@jjhursey Let us know if #3468 fixes the issue -- we have corresponding PRs for v2.0.x, v2.1.x, and v3.0.x.

[jjhursey]

😞 That PR does not seem to fix the make check issue. We'll have to keep investigating.

[jjhursey]

Instead of failing now it is hanging (I have to manually kill it as it will hang the MTT runs - there is no timeout mechanism in make check). Here is a link to an MTT failure:

• https://mtt.open-mpi.org/index.php?do_redir=2439

I suspect that the change made to fix Issue #3450 made this failure a hang now.

[jsquyres]

@bosilca @hjelmn It looks like the recent changes to opal_fifo just changed the failure mode for IBM. Can you have a look at Josh's comment (#2966 (comment))?

[bosilca]

First and foremost, the failure is on ppc64le, I have no access to such an architecture. What is puzzling is that I was under the impresison that for PPC we were using atomic load/store (via OPAL_HAVE_ATOMIC_LLSC_PTR) but apparently this is not the case.

I run 10k tests on different Intel architectures with an OMPI compiler is optimized mode, and I had no failure. No really sure how to approach this issue.

[jsquyres]

@jjhursey Looks like we're going to need some IBM help on this one...

[jjhursey]

Yeah that's fine. I can investigate, but I'm not back full time yet so it'll be a little while before I can probably get eyes on this. I just wanted to keep this issue updated.
The hang mode is helpful though since I can then attach a debugger and poke around a bit.

[gpaulsen]

I'm going to ask @nysal to drive this from our side, since Josh is apparently skydiving.

[jjhursey]

There was a request to re-assess this ticket after PR #3661 - I did so and the problem still persists. We'll continue to investigate.