P2P Access Not Working on Level Zero

[kaanolgu]

Describe the bug

Although documented here :

llvm/sycl/test-e2e/USM/P2P/p2p_access.cpp

Line 1 in ab6c0f5

// REQUIRES: cuda || hip || level_zero

It doesn't work with level_zero is there something else I need to enable or include ?

To reproduce

export SYCL_PI_TRACE=1
export SYCL_PRINT_EXECUTION_GRAPH="always"
export ONEAPI_DEVICE_SELECTOR=level_zero:gpu

./build/run.exe

Caught a SYCL host exception:
Native API failed. Native API returns: -995 (The plugin or device does not support the called function) -995 (The plugin or device does not support the called function)
terminate called after throwing an instance of 'sycl::_V1::runtime_error'
  what():  Native API failed. Native API returns: -995 (The plugin or device does not support the called function) -995 (The plugin or device does not support the called function)

Environment

• OS: [e.g Windows/Linux]
• Target device and vendor: Intel(R) Data Center GPU Max 1550
• DPC++ version: $ icpx --version
  Intel(R) oneAPI DPC++/C++ Compiler 2024.2.1 (2024.2.1.20240711)
  Target: x86_64-unknown-linux-gnu
  Thread model: posix
  InstalledDir: /opt/intel/oneapi/compiler/2024.2/bin/compiler
  Configuration file: /opt/intel/oneapi/compiler/2024.2/bin/compiler/../icpx.cfg
• Dependencies version:

$ sycl-ls --verbose
INFO: Output filtered by ONEAPI_DEVICE_SELECTOR environment variable, which is set to level_zero:gpu.
To see device ids, use the --ignore-device-selectors CLI option.

[level_zero:gpu] Intel(R) Level-Zero, Intel(R) Data Center GPU Max 1550 1.3 [1.3.27191]
[level_zero:gpu] Intel(R) Level-Zero, Intel(R) Data Center GPU Max 1550 1.3 [1.3.27191]
[level_zero:gpu] Intel(R) Level-Zero, Intel(R) Data Center GPU Max 1550 1.3 [1.3.27191]
[level_zero:gpu] Intel(R) Level-Zero, Intel(R) Data Center GPU Max 1550 1.3 [1.3.27191]
[level_zero:gpu] Intel(R) Level-Zero, Intel(R) Data Center GPU Max 1550 1.3 [1.3.27191]
[level_zero:gpu] Intel(R) Level-Zero, Intel(R) Data Center GPU Max 1550 1.3 [1.3.27191]
[level_zero:gpu] Intel(R) Level-Zero, Intel(R) Data Center GPU Max 1550 1.3 [1.3.27191]
[level_zero:gpu] Intel(R) Level-Zero, Intel(R) Data Center GPU Max 1550 1.3 [1.3.27191]
[level_zero:gpu] Intel(R) Level-Zero, Intel(R) Data Center GPU Max 1550 1.3 [1.3.27191]
[level_zero:gpu] Intel(R) Level-Zero, Intel(R) Data Center GPU Max 1550 1.3 [1.3.27191]
[level_zero:gpu] Intel(R) Level-Zero, Intel(R) Data Center GPU Max 1550 1.3 [1.3.27191]
[level_zero:gpu] Intel(R) Level-Zero, Intel(R) Data Center GPU Max 1550 1.3 [1.3.27191]
[level_zero:gpu] Intel(R) Level-Zero, Intel(R) Data Center GPU Max 1550 1.3 [1.3.27191]
[level_zero:gpu] Intel(R) Level-Zero, Intel(R) Data Center GPU Max 1550 1.3 [1.3.27191]
[level_zero:gpu] Intel(R) Level-Zero, Intel(R) Data Center GPU Max 1550 1.3 [1.3.27191]
[level_zero:gpu] Intel(R) Level-Zero, Intel(R) Data Center GPU Max 1550 1.3 [1.3.27191]

Platforms: 1
Platform [#1]:
    Version  : 1.3
    Name     : Intel(R) Level-Zero
    Vendor   : Intel(R) Corporation
    Devices  : 16
        Type       : gpu
        Version    : 1.3
        Name       : Intel(R) Data Center GPU Max 1550
        Vendor     : Intel(R) Corporation
        Driver     : 1.3.27191
        Aspects    : gpu fp16 fp64 online_compiler online_linker queue_profiling usm_device_allocations usm_host_allocations usm_shared_allocations ext_intel_pci_address ext_intel_gpu_eu_count ext_intel_gpu_eu_simd_width ext_intel_gpu_slices ext_intel_gpu_subslices_per_slice ext_intel_gpu_eu_count_per_subslice atomic64 ext_intel_device_info_uuid ext_intel_gpu_hw_threads_per_eu ext_intel_device_id ext_intel_memory_clock_rate ext_intel_memory_bus_width ext_oneapi_bindless_images ext_oneapi_bindless_images_shared_usm ext_oneapi_bindless_images_2d_usm ext_oneapi_mipmap ext_oneapi_mipmap_anisotropy ext_intel_esimd ext_oneapi_ballot_group ext_oneapi_fixed_size_group ext_oneapi_opportunistic_group ext_oneapi_tangle_group ext_intel_matrix ext_oneapi_graph
        info::device::sub_group_sizes: 16 32
        Type       : gpu
        Version    : 1.3
        Name       : Intel(R) Data Center GPU Max 1550
        Vendor     : Intel(R) Corporation
        Driver     : 1.3.27191
        Aspects    : gpu fp16 fp64 online_compiler online_linker queue_profiling usm_device_allocations usm_host_allocations usm_shared_allocations ext_intel_pci_address ext_intel_gpu_eu_count ext_intel_gpu_eu_simd_width ext_intel_gpu_slices ext_intel_gpu_subslices_per_slice ext_intel_gpu_eu_count_per_subslice atomic64 ext_intel_device_info_uuid ext_intel_gpu_hw_threads_per_eu ext_intel_device_id ext_intel_memory_clock_rate ext_intel_memory_bus_width ext_oneapi_bindless_images ext_oneapi_bindless_images_shared_usm ext_oneapi_bindless_images_2d_usm ext_oneapi_mipmap ext_oneapi_mipmap_anisotropy ext_intel_esimd ext_oneapi_ballot_group ext_oneapi_fixed_size_group ext_oneapi_opportunistic_group ext_oneapi_tangle_group ext_intel_matrix ext_oneapi_graph
        info::device::sub_group_sizes: 16 32
        Type       : gpu
        Version    : 1.3
        Name       : Intel(R) Data Center GPU Max 1550
        Vendor     : Intel(R) Corporation
        Driver     : 1.3.27191
        Aspects    : gpu fp16 fp64 online_compiler online_linker queue_profiling usm_device_allocations usm_host_allocations usm_shared_allocations ext_intel_pci_address ext_intel_gpu_eu_count ext_intel_gpu_eu_simd_width ext_intel_gpu_slices ext_intel_gpu_subslices_per_slice ext_intel_gpu_eu_count_per_subslice atomic64 ext_intel_device_info_uuid ext_intel_gpu_hw_threads_per_eu ext_intel_device_id ext_intel_memory_clock_rate ext_intel_memory_bus_width ext_oneapi_bindless_images ext_oneapi_bindless_images_shared_usm ext_oneapi_bindless_images_2d_usm ext_oneapi_mipmap ext_oneapi_mipmap_anisotropy ext_intel_esimd ext_oneapi_ballot_group ext_oneapi_fixed_size_group ext_oneapi_opportunistic_group ext_oneapi_tangle_group ext_intel_matrix ext_oneapi_graph
        info::device::sub_group_sizes: 16 32
        Type       : gpu
        Version    : 1.3
        Name       : Intel(R) Data Center GPU Max 1550
        Vendor     : Intel(R) Corporation
        Driver     : 1.3.27191
        Aspects    : gpu fp16 fp64 online_compiler online_linker queue_profiling usm_device_allocations usm_host_allocations usm_shared_allocations ext_intel_pci_address ext_intel_gpu_eu_count ext_intel_gpu_eu_simd_width ext_intel_gpu_slices ext_intel_gpu_subslices_per_slice ext_intel_gpu_eu_count_per_subslice atomic64 ext_intel_device_info_uuid ext_intel_gpu_hw_threads_per_eu ext_intel_device_id ext_intel_memory_clock_rate ext_intel_memory_bus_width ext_oneapi_bindless_images ext_oneapi_bindless_images_shared_usm ext_oneapi_bindless_images_2d_usm ext_oneapi_mipmap ext_oneapi_mipmap_anisotropy ext_intel_esimd ext_oneapi_ballot_group ext_oneapi_fixed_size_group ext_oneapi_opportunistic_group ext_oneapi_tangle_group ext_intel_matrix ext_oneapi_graph
        info::device::sub_group_sizes: 16 32
        Type       : gpu
        Version    : 1.3
        Name       : Intel(R) Data Center GPU Max 1550
        Vendor     : Intel(R) Corporation
        Driver     : 1.3.27191
        Aspects    : gpu fp16 fp64 online_compiler online_linker queue_profiling usm_device_allocations usm_host_allocations usm_shared_allocations ext_intel_pci_address ext_intel_gpu_eu_count ext_intel_gpu_eu_simd_width ext_intel_gpu_slices ext_intel_gpu_subslices_per_slice ext_intel_gpu_eu_count_per_subslice atomic64 ext_intel_device_info_uuid ext_intel_gpu_hw_threads_per_eu ext_intel_device_id ext_intel_memory_clock_rate ext_intel_memory_bus_width ext_oneapi_bindless_images ext_oneapi_bindless_images_shared_usm ext_oneapi_bindless_images_2d_usm ext_oneapi_mipmap ext_oneapi_mipmap_anisotropy ext_intel_esimd ext_oneapi_ballot_group ext_oneapi_fixed_size_group ext_oneapi_opportunistic_group ext_oneapi_tangle_group ext_intel_matrix ext_oneapi_graph
        info::device::sub_group_sizes: 16 32
        Type       : gpu
        Version    : 1.3
        Name       : Intel(R) Data Center GPU Max 1550
        Vendor     : Intel(R) Corporation
        Driver     : 1.3.27191
        Aspects    : gpu fp16 fp64 online_compiler online_linker queue_profiling usm_device_allocations usm_host_allocations usm_shared_allocations ext_intel_pci_address ext_intel_gpu_eu_count ext_intel_gpu_eu_simd_width ext_intel_gpu_slices ext_intel_gpu_subslices_per_slice ext_intel_gpu_eu_count_per_subslice atomic64 ext_intel_device_info_uuid ext_intel_gpu_hw_threads_per_eu ext_intel_device_id ext_intel_memory_clock_rate ext_intel_memory_bus_width ext_oneapi_bindless_images ext_oneapi_bindless_images_shared_usm ext_oneapi_bindless_images_2d_usm ext_oneapi_mipmap ext_oneapi_mipmap_anisotropy ext_intel_esimd ext_oneapi_ballot_group ext_oneapi_fixed_size_group ext_oneapi_opportunistic_group ext_oneapi_tangle_group ext_intel_matrix ext_oneapi_graph
        info::device::sub_group_sizes: 16 32
        Type       : gpu
        Version    : 1.3
        Name       : Intel(R) Data Center GPU Max 1550
        Vendor     : Intel(R) Corporation
        Driver     : 1.3.27191
        Aspects    : gpu fp16 fp64 online_compiler online_linker queue_profiling usm_device_allocations usm_host_allocations usm_shared_allocations ext_intel_pci_address ext_intel_gpu_eu_count ext_intel_gpu_eu_simd_width ext_intel_gpu_slices ext_intel_gpu_subslices_per_slice ext_intel_gpu_eu_count_per_subslice atomic64 ext_intel_device_info_uuid ext_intel_gpu_hw_threads_per_eu ext_intel_device_id ext_intel_memory_clock_rate ext_intel_memory_bus_width ext_oneapi_bindless_images ext_oneapi_bindless_images_shared_usm ext_oneapi_bindless_images_2d_usm ext_oneapi_mipmap ext_oneapi_mipmap_anisotropy ext_intel_esimd ext_oneapi_ballot_group ext_oneapi_fixed_size_group ext_oneapi_opportunistic_group ext_oneapi_tangle_group ext_intel_matrix ext_oneapi_graph
        info::device::sub_group_sizes: 16 32
        Type       : gpu
        Version    : 1.3
        Name       : Intel(R) Data Center GPU Max 1550
        Vendor     : Intel(R) Corporation
        Driver     : 1.3.27191
        Aspects    : gpu fp16 fp64 online_compiler online_linker queue_profiling usm_device_allocations usm_host_allocations usm_shared_allocations ext_intel_pci_address ext_intel_gpu_eu_count ext_intel_gpu_eu_simd_width ext_intel_gpu_slices ext_intel_gpu_subslices_per_slice ext_intel_gpu_eu_count_per_subslice atomic64 ext_intel_device_info_uuid ext_intel_gpu_hw_threads_per_eu ext_intel_device_id ext_intel_memory_clock_rate ext_intel_memory_bus_width ext_oneapi_bindless_images ext_oneapi_bindless_images_shared_usm ext_oneapi_bindless_images_2d_usm ext_oneapi_mipmap ext_oneapi_mipmap_anisotropy ext_intel_esimd ext_oneapi_ballot_group ext_oneapi_fixed_size_group ext_oneapi_opportunistic_group ext_oneapi_tangle_group ext_intel_matrix ext_oneapi_graph
        info::device::sub_group_sizes: 16 32
        Type       : gpu
        Version    : 1.3
        Name       : Intel(R) Data Center GPU Max 1550
        Vendor     : Intel(R) Corporation
        Driver     : 1.3.27191
        Aspects    : gpu fp16 fp64 online_compiler online_linker queue_profiling usm_device_allocations usm_host_allocations usm_shared_allocations ext_intel_pci_address ext_intel_gpu_eu_count ext_intel_gpu_eu_simd_width ext_intel_gpu_slices ext_intel_gpu_subslices_per_slice ext_intel_gpu_eu_count_per_subslice atomic64 ext_intel_device_info_uuid ext_intel_gpu_hw_threads_per_eu ext_intel_device_id ext_intel_memory_clock_rate ext_intel_memory_bus_width ext_oneapi_bindless_images ext_oneapi_bindless_images_shared_usm ext_oneapi_bindless_images_2d_usm ext_oneapi_mipmap ext_oneapi_mipmap_anisotropy ext_intel_esimd ext_oneapi_ballot_group ext_oneapi_fixed_size_group ext_oneapi_opportunistic_group ext_oneapi_tangle_group ext_intel_matrix ext_oneapi_graph
        info::device::sub_group_sizes: 16 32
        Type       : gpu
        Version    : 1.3
        Name       : Intel(R) Data Center GPU Max 1550
        Vendor     : Intel(R) Corporation
        Driver     : 1.3.27191
        Aspects    : gpu fp16 fp64 online_compiler online_linker queue_profiling usm_device_allocations usm_host_allocations usm_shared_allocations ext_intel_pci_address ext_intel_gpu_eu_count ext_intel_gpu_eu_simd_width ext_intel_gpu_slices ext_intel_gpu_subslices_per_slice ext_intel_gpu_eu_count_per_subslice atomic64 ext_intel_device_info_uuid ext_intel_gpu_hw_threads_per_eu ext_intel_device_id ext_intel_memory_clock_rate ext_intel_memory_bus_width ext_oneapi_bindless_images ext_oneapi_bindless_images_shared_usm ext_oneapi_bindless_images_2d_usm ext_oneapi_mipmap ext_oneapi_mipmap_anisotropy ext_intel_esimd ext_oneapi_ballot_group ext_oneapi_fixed_size_group ext_oneapi_opportunistic_group ext_oneapi_tangle_group ext_intel_matrix ext_oneapi_graph
        info::device::sub_group_sizes: 16 32
        Type       : gpu
        Version    : 1.3
        Name       : Intel(R) Data Center GPU Max 1550
        Vendor     : Intel(R) Corporation
        Driver     : 1.3.27191
        Aspects    : gpu fp16 fp64 online_compiler online_linker queue_profiling usm_device_allocations usm_host_allocations usm_shared_allocations ext_intel_pci_address ext_intel_gpu_eu_count ext_intel_gpu_eu_simd_width ext_intel_gpu_slices ext_intel_gpu_subslices_per_slice ext_intel_gpu_eu_count_per_subslice atomic64 ext_intel_device_info_uuid ext_intel_gpu_hw_threads_per_eu ext_intel_device_id ext_intel_memory_clock_rate ext_intel_memory_bus_width ext_oneapi_bindless_images ext_oneapi_bindless_images_shared_usm ext_oneapi_bindless_images_2d_usm ext_oneapi_mipmap ext_oneapi_mipmap_anisotropy ext_intel_esimd ext_oneapi_ballot_group ext_oneapi_fixed_size_group ext_oneapi_opportunistic_group ext_oneapi_tangle_group ext_intel_matrix ext_oneapi_graph
        info::device::sub_group_sizes: 16 32
        Type       : gpu
        Version    : 1.3
        Name       : Intel(R) Data Center GPU Max 1550
        Vendor     : Intel(R) Corporation
        Driver     : 1.3.27191
        Aspects    : gpu fp16 fp64 online_compiler online_linker queue_profiling usm_device_allocations usm_host_allocations usm_shared_allocations ext_intel_pci_address ext_intel_gpu_eu_count ext_intel_gpu_eu_simd_width ext_intel_gpu_slices ext_intel_gpu_subslices_per_slice ext_intel_gpu_eu_count_per_subslice atomic64 ext_intel_device_info_uuid ext_intel_gpu_hw_threads_per_eu ext_intel_device_id ext_intel_memory_clock_rate ext_intel_memory_bus_width ext_oneapi_bindless_images ext_oneapi_bindless_images_shared_usm ext_oneapi_bindless_images_2d_usm ext_oneapi_mipmap ext_oneapi_mipmap_anisotropy ext_intel_esimd ext_oneapi_ballot_group ext_oneapi_fixed_size_group ext_oneapi_opportunistic_group ext_oneapi_tangle_group ext_intel_matrix ext_oneapi_graph
        info::device::sub_group_sizes: 16 32
        Type       : gpu
        Version    : 1.3
        Name       : Intel(R) Data Center GPU Max 1550
        Vendor     : Intel(R) Corporation
        Driver     : 1.3.27191
        Aspects    : gpu fp16 fp64 online_compiler online_linker queue_profiling usm_device_allocations usm_host_allocations usm_shared_allocations ext_intel_pci_address ext_intel_gpu_eu_count ext_intel_gpu_eu_simd_width ext_intel_gpu_slices ext_intel_gpu_subslices_per_slice ext_intel_gpu_eu_count_per_subslice atomic64 ext_intel_device_info_uuid ext_intel_gpu_hw_threads_per_eu ext_intel_device_id ext_intel_memory_clock_rate ext_intel_memory_bus_width ext_oneapi_bindless_images ext_oneapi_bindless_images_shared_usm ext_oneapi_bindless_images_2d_usm ext_oneapi_mipmap ext_oneapi_mipmap_anisotropy ext_intel_esimd ext_oneapi_ballot_group ext_oneapi_fixed_size_group ext_oneapi_opportunistic_group ext_oneapi_tangle_group ext_intel_matrix ext_oneapi_graph
        info::device::sub_group_sizes: 16 32
        Type       : gpu
        Version    : 1.3
        Name       : Intel(R) Data Center GPU Max 1550
        Vendor     : Intel(R) Corporation
        Driver     : 1.3.27191
        Aspects    : gpu fp16 fp64 online_compiler online_linker queue_profiling usm_device_allocations usm_host_allocations usm_shared_allocations ext_intel_pci_address ext_intel_gpu_eu_count ext_intel_gpu_eu_simd_width ext_intel_gpu_slices ext_intel_gpu_subslices_per_slice ext_intel_gpu_eu_count_per_subslice atomic64 ext_intel_device_info_uuid ext_intel_gpu_hw_threads_per_eu ext_intel_device_id ext_intel_memory_clock_rate ext_intel_memory_bus_width ext_oneapi_bindless_images ext_oneapi_bindless_images_shared_usm ext_oneapi_bindless_images_2d_usm ext_oneapi_mipmap ext_oneapi_mipmap_anisotropy ext_intel_esimd ext_oneapi_ballot_group ext_oneapi_fixed_size_group ext_oneapi_opportunistic_group ext_oneapi_tangle_group ext_intel_matrix ext_oneapi_graph
        info::device::sub_group_sizes: 16 32
        Type       : gpu
        Version    : 1.3
        Name       : Intel(R) Data Center GPU Max 1550
        Vendor     : Intel(R) Corporation
        Driver     : 1.3.27191
        Aspects    : gpu fp16 fp64 online_compiler online_linker queue_profiling usm_device_allocations usm_host_allocations usm_shared_allocations ext_intel_pci_address ext_intel_gpu_eu_count ext_intel_gpu_eu_simd_width ext_intel_gpu_slices ext_intel_gpu_subslices_per_slice ext_intel_gpu_eu_count_per_subslice atomic64 ext_intel_device_info_uuid ext_intel_gpu_hw_threads_per_eu ext_intel_device_id ext_intel_memory_clock_rate ext_intel_memory_bus_width ext_oneapi_bindless_images ext_oneapi_bindless_images_shared_usm ext_oneapi_bindless_images_2d_usm ext_oneapi_mipmap ext_oneapi_mipmap_anisotropy ext_intel_esimd ext_oneapi_ballot_group ext_oneapi_fixed_size_group ext_oneapi_opportunistic_group ext_oneapi_tangle_group ext_intel_matrix ext_oneapi_graph
        info::device::sub_group_sizes: 16 32
        Type       : gpu
        Version    : 1.3
        Name       : Intel(R) Data Center GPU Max 1550
        Vendor     : Intel(R) Corporation
        Driver     : 1.3.27191
        Aspects    : gpu fp16 fp64 online_compiler online_linker queue_profiling usm_device_allocations usm_host_allocations usm_shared_allocations ext_intel_pci_address ext_intel_gpu_eu_count ext_intel_gpu_eu_simd_width ext_intel_gpu_slices ext_intel_gpu_subslices_per_slice ext_intel_gpu_eu_count_per_subslice atomic64 ext_intel_device_info_uuid ext_intel_gpu_hw_threads_per_eu ext_intel_device_id ext_intel_memory_clock_rate ext_intel_memory_bus_width ext_oneapi_bindless_images ext_oneapi_bindless_images_shared_usm ext_oneapi_bindless_images_2d_usm ext_oneapi_mipmap ext_oneapi_mipmap_anisotropy ext_intel_esimd ext_oneapi_ballot_group ext_oneapi_fixed_size_group ext_oneapi_opportunistic_group ext_oneapi_tangle_group ext_intel_matrix ext_oneapi_graph
        info::device::sub_group_sizes: 16 32
default_selector()      : gpu, Intel(R) Level-Zero, Intel(R) Data Center GPU Max 1550 1.3 [1.3.27191]
accelerator_selector()  : No device of requested type available. Please chec...
cpu_selector()          : No device of requested type available. Please chec...
gpu_selector()          : gpu, Intel(R) Level-Zero, Intel(R) Data Center GPU Max 1550 1.3 [1.3.27191]
custom_selector(gpu)    : gpu, Intel(R) Level-Zero, Intel(R) Data Center GPU Max 1550 1.3 [1.3.27191]
custom_selector(cpu)    : No device of requested type available. Please chec...
custom_selector(acc)    : No device of requested type available. Please chec...

Additional context

No response

[kaanolgu]

#14095

I am confused now, is p2p_access available for level_zero or not ?

[JackAKirk]

cc @nrspruit

[JackAKirk]

See #15847 (comment)
I'll try to fix the implementation.

[JackAKirk]

@kaanolgu do you want to check if #15873 fixes your issue using this branch https://github.com/JackAKirk/llvm/tree/p2p-fix-query?
I don't have access to L0 machines but I suspect it might fix your issue.

[kaanolgu]

Hi @JackAKirk ,
Yes it solved the issue in other thread and
I tried this nightly build p2p access and sycl-ls also worked https://github.com/intel/llvm/releases/tag/nightly-2024-10-25 so this clue hopefully help ? Thank you again

[JackAKirk]

Hi @JackAKirk , Yes it solved the issue in other thread and I tried this nightly build p2p access and sycl-ls also worked https://github.com/intel/llvm/releases/tag/nightly-2024-10-25 so this clue hopefully help ? Thank you again

Great, thanks for the info!