Atomics get completion with error on mlx5

[sjeaugey]

A dozen of MTT tests related to atomics still fail using mlx5. They all fail with the same error message :

mlx5: drossetti-ivy4.nvidia.com: got completion with error:
00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000
00000000 00008813 110818b2 000001d0
[[53836,1],0][btl_openib_component.c:3611:handle_wc] from drossetti-ivy4 to: drossetti-ivy4 error
polling LP CQ with status REMOTE ACCESS ERROR status number 10 for wr_id 25e14b8 opcode 3 vendor
error 136 qp_idx 0

[hjelmn]

Do any of the ivb machines have the latest mofed? Looks like this might be a change in behavior from the tested version (2.x).

[sjeaugey]

Yes, they're both running MOFED 3.1.

[sjeaugey]

Narrowing the issue : it only appears with GPU Direct RDMA, and disappears if I remove ATOMIC_FOPS from btl_openib_flags.

I guess you are doing IB atomics directly on the memory location, which won't work if it is GPU memory mapped directly to the HCA. Even if PCIe supports atomics (from the NIC to the GPU) the GPU won't support it through the PCI BAR mapping.

Do you think it would be possible to disable FOPS when the target is GPU memory ? I guess not since the source has no knowledge of the destination buffer ... unless there is a RDV protocol ? Another option would be to disable FOPS when the window is on the GPU.

If we can't find a good way to do that, I'll do a patch to disable ATOMIC_FOPS whenever GDR is activated.

[sjeaugey]

Re-testing, I has in fact nothing to do with GPU Direct. I thought so because the non-GPU Direct test was also running on the tcp BTL ... so, we're back on your first question about MOFED version. Do you have an idea where to look at and what to try ?

[sjeaugey]

On the IBM onesided/c_accumulate test, the crash seems to happens on the first fetch_and_add operation. There are many cswap operations before which work fine.
Edit : fetch_and_add seems out of the problem here, it crashes the same way if we do a cswap.

[sjeaugey]

Adding traces, the crash happens when we do the accumulate on rank 1. Accumulate on rank 0 works.

Btw, it only crashes with multiple ranks per node, it works fine with 1 rank per node or with multiple ranks on a single node. It may therefore be related to #1209.

The Atomics between the nodes seem to work, what doesn't work is :

• rank 0 or rank 1 doing remote atomic to rank 1 memory (local but through IB)
• rank 2 or rank 3 doing remote atomic to rank 3 memory (also local but through IB)

[hjelmn]

Looking at this now. I probably overlooked this problem because it doesn't fail on ugni for some reason. Probably is a difference in how the registration keys work.