verification of the simpleP2P example in HIP

[zjin-lcf]

I convert the CUDA simpleP2P example to HIP. However, the HIP program does not pass the verification. Could you reproduce the issue ? Thanks.

The HIP example is here: https://github.com/zjin-lcf/HeCBench/blob/master/p2p-hip/main.cu

Environment: ROCM 4.5.2 and MI100

[jlgreathouse]

I'm unable to reproduce this on an MI100 system I have access to that is running ROCm 5.2.0. This is using an AMD EPYC 7742 CPU, and I'm able to see this test pass between MI100s in the same XGMI hive and MI100s in separate XGMI hives (i.e., PCIe peers)

$ HIP_VISIBLE_DEVICES=0,1 ./a.out 1
[./a.out] - Starting...
Checking for multiple GPUs...
HIP-capable device count: 2

Checking GPU(s) for support of peer to peer memory access...
> Peer access from  (GPU0) ->  (GPU1) : Yes
> Peer access from  (GPU1) ->  (GPU0) : Yes
Enabling peer access between GPU0 and GPU1...
Allocating buffers (64MB on GPU0, GPU1 and CPU Host)...
hipMemcpyPeer / hipMemcpy between GPU0 and GPU1: 28.13GB/s
Preparing host buffer and memcpy to GPU0...
Run kernel on GPU1, taking source data from GPU0 and writing to GPU1...
Run kernel on GPU0, taking source data from GPU1 and writing to GPU0...
Copy data back to host from GPU0 and verify results...
Disabling peer access...
Shutting down...
Test passed

[jlgreathouse]

Any chance you could try on a newer ROCm version? And could you show the output of lscpu, rocm-smi --showtopo, uname -r (and what OS/distro are you using?), and dkms status?

[zjin-lcf]

I will try to build the example with a newer version. Thanks.

[jlgreathouse]

You might also want to make sure that you have PCIe Large BAR turned on in your system BIOS (this is sometimes marked as "> 4G Decoding", or on some consumer boards, things like "Crypto Mining Mode" because it is used by some mining users to install many GPUs into their system).

You can check this by running lspci -v and searching for the size of the first memory region on your MI100 accelerator devices. For example:

23:00.0 Display controller: Advanced Micro Devices, Inc. [AMD/ATI] Device 738c (rev 01)
        Flags: bus master, fast devsel, latency 0, IRQ 268, NUMA node 1
        **Memory at 2f800000000 (64-bit, prefetchable) [size=32G]**
        Memory at 30000000000 (64-bit, prefetchable) [size=2M]
        I/O ports at 3000 [size=256]
        Memory at a6500000 (32-bit, non-prefetchable) [size=512K]

The fact that the first memory region on the MI100 I'm looking at shows 32GB of range shows that I can see all of the HBM on the device through the PCIe aperture.

If this is smaller (e.g. if it says "size=256M") then you do not have large BAR enabled. In that case, we cannot support PCIe peer-to-peer transfers.

[zjin-lcf]

Close the issue since it is not reproducible. Thanks again.