[SYCL][CUDA] Select only NVPTX64 device binaries #1223

fwyzard · 2020-03-01T10:41:08Z

Change the behaviour of the program manager to always ask the native runtime to choose a device image, even if only one is available; this should prevent selecting an image that is not compatible with the current device.

Fix the behaviour of the CUDA plugin to search through the available binary images and select the first one compatible with a PTX target, or return PI_INVALID_BINARY if there are no compatible images.

Define the PI_DEVICE_BINARY_TARGET_NVPTX64 target identifier as "nvptx64" for NVIDIA PTX devices.

fwyzard · 2020-03-01T10:41:29Z

Fixes #1194 in my test case.

Add the binary target identifier "nvptx64" for NVIDIA PTX devices. Signed-off-by: Andrea Bocci <[email protected]>

Search through the available binary images and select the first one for the PI_DEVICE_BINARY_TARGET_NVPTX64 ("nvptx64") target. Return PI_INVALID_BINARY if no "nvptx64" image is available. Signed-off-by: Andrea Bocci <[email protected]>

bader · 2020-03-01T11:20:53Z

@fwyzard, would you mind to adding a simple regression test to check that problem #1194 is fixed, please?
It will help us breaking this functionality in the future.

fwyzard · 2020-03-01T11:40:31Z

@fwyzard, would you mind to adding a simple regression test to check that problem #1194 is fixed, please?

I don't mind, but I don't know where to get started.
If you can point me to some existing tests for SYCL or the plugin interface, I can probably take it from there...

bader · 2020-03-01T13:45:58Z

We use LIT infrastructure to check end-to-end behavior and tests are located here: https://github.com/intel/llvm/blob/sycl/sycl/test/. Alternative approach is to use Google Test framework to validate specific parts of the SYCL runtime instead of building a full application.
These tests are located here: https://github.com/intel/llvm/tree/sycl/sycl/unittests

Looking at your changes it seems that CUDA agnostic part might be already covered by @sergey-semenov in 9095749, so need to validate only CUDA plugin changes. It seems to me that Codeplay team is using mostly Google test framework for this: https://github.com/intel/llvm/tree/sycl/sycl/unittests/pi/cuda.
One caveat with CUDA tests is they are all disabled today due to lack of machines in our CI system. We are still working on adding them.
I think it's still useful the add them, but they will be validated post-commit.

@romanovvlad, @Ruyk, does it make sense to you?

fwyzard · 2020-03-01T23:52:11Z

I'm trying to run the existing tests, but I must be missing something.

make check-sycl seems to get started fine, until it gets to the point

                                                                                                     -- Testing: 219 tests, 16 workers --                                                                                                    
 47% [======================================================================================================------------------------------------------------------------------------------------------------------------------] ETA: 00:00:34

However, most of the tests fail due to

/data/user/fwyzard/sycl/build-sycl-4b5d25b2f97/tools/sycl/test/sub_group/Output/scan.cpp.tmp.out: error while loading shared libraries: libsycl.so: cannot open shared object file: No such file or directory

error: command failed with exit status: 127

Do I need to set up the LD_LIBRARY_PATH ? Should't the test machinery take care of it ?

After setting up the environment, I still get

Expected Passes : 186
Unsupported Tests : 6
Unexpected Failures: 27

Most of the failures look like

$ "env" "SYCL_DEVICE_TYPE=HOST" "/data/user/fwyzard/sycl/build-sycl-4b5d25b2f97/tools/sycl/test/usm/Output/prefetch.cpp.tmp1.out"
note: command had no output on stdout or stderr
error: command failed with exit status: -11

and running them by hand I get

Segmentation fault (core dumped)

Am I missing something in my test setup ?

romanovvlad · 2020-03-02T12:24:24Z

I'm trying to run the existing tests, but I must be missing something.

make check-sycl seems to get started fine, until it gets to the point

                                                                                                     -- Testing: 219 tests, 16 workers --                                                                                                    
 47% [======================================================================================================------------------------------------------------------------------------------------------------------------------] ETA: 00:00:34

However, most of the tests fail due to

/data/user/fwyzard/sycl/build-sycl-4b5d25b2f97/tools/sycl/test/sub_group/Output/scan.cpp.tmp.out: error while loading shared libraries: libsycl.so: cannot open shared object file: No such file or directory

error: command failed with exit status: 127

Do I need to set up the LD_LIBRARY_PATH ? Should't the test machinery take care of it ?

After setting up the environment, I still get

Expected Passes : 186
Unsupported Tests : 6
Unexpected Failures: 27

Most of the failures look like

$ "env" "SYCL_DEVICE_TYPE=HOST" "/data/user/fwyzard/sycl/build-sycl-4b5d25b2f97/tools/sycl/test/usm/Output/prefetch.cpp.tmp1.out"
note: command had no output on stdout or stderr
error: command failed with exit status: -11

and running them by hand I get

Segmentation fault (core dumped)

Am I missing something in my test setup ?

I'm trying to run the existing tests, but I must be missing something.

make check-sycl seems to get started fine, until it gets to the point

                                                                                                     -- Testing: 219 tests, 16 workers --                                                                                                    
 47% [======================================================================================================------------------------------------------------------------------------------------------------------------------] ETA: 00:00:34

However, most of the tests fail due to

/data/user/fwyzard/sycl/build-sycl-4b5d25b2f97/tools/sycl/test/sub_group/Output/scan.cpp.tmp.out: error while loading shared libraries: libsycl.so: cannot open shared object file: No such file or directory

error: command failed with exit status: 127

Do I need to set up the LD_LIBRARY_PATH ? Should't the test machinery take care of it ?

After setting up the environment, I still get

Expected Passes : 186
Unsupported Tests : 6
Unexpected Failures: 27

Most of the failures look like

$ "env" "SYCL_DEVICE_TYPE=HOST" "/data/user/fwyzard/sycl/build-sycl-4b5d25b2f97/tools/sycl/test/usm/Output/prefetch.cpp.tmp1.out"
note: command had no output on stdout or stderr
error: command failed with exit status: -11

and running them by hand I get

Segmentation fault (core dumped)

Am I missing something in my test setup ?

No, libsycl.so should be automatically added to LD_LIBRARY_PATH by lit.cfg.py.
Could you please create an issue with information of your system and all steps from having empty dir and clear env to seeing fails when running check-sycl?
Backtrace for segfault would be also helpful.

fwyzard · 2020-03-02T13:22:14Z

OK, will do.

I the meantime I think I've managed to prepare a test for #1194, that shows how it is fixed by these changes:

********************
FAIL: SYCL :: plugins/sycl-targets-order.cpp (135 of 219)
******************** TEST 'SYCL :: plugins/sycl-targets-order.cpp' FAILED ********************
Script:
--
: 'RUN: at line 1';    /data/user/fwyzard/sycl/build/bin/clang++ -fsycl -fsycl-targets=spir64-unknown-unknown-sycldevice,nvptx64-unknown-unknown-sycldevice /data/user/fwyzard/sycl/llvm/sycl/test/plugins/sycl-targets-order.cpp -o /data/us
er/fwyzard/sycl/build/tools/sycl/test/plugins/Output/sycl-targets-order.cpp.tmp-spir64-nvptx64.out
: 'RUN: at line 2';   env SYCL_BE=PI_OPENCL /data/user/fwyzard/sycl/build/tools/sycl/test/plugins/Output/sycl-targets-order.cpp.tmp-spir64-nvptx64.out
: 'RUN: at line 3';   env SYCL_BE=PI_CUDA   /data/user/fwyzard/sycl/build/tools/sycl/test/plugins/Output/sycl-targets-order.cpp.tmp-spir64-nvptx64.out
: 'RUN: at line 4';    /data/user/fwyzard/sycl/build/bin/clang++ -fsycl -fsycl-targets=nvptx64-unknown-unknown-sycldevice,spir64-unknown-unknown-sycldevice /data/user/fwyzard/sycl/llvm/sycl/test/plugins/sycl-targets-order.cpp -o /data/user/fwyzard/sycl/build/tools/sycl/test/plugins/Output/sycl-targets-order.cpp.tmp-nvptx64-spir64.out
: 'RUN: at line 5';   env SYCL_BE=PI_OPENCL /data/user/fwyzard/sycl/build/tools/sycl/test/plugins/Output/sycl-targets-order.cpp.tmp-nvptx64-spir64.out
: 'RUN: at line 6';   env SYCL_BE=PI_CUDA   /data/user/fwyzard/sycl/build/tools/sycl/test/plugins/Output/sycl-targets-order.cpp.tmp-nvptx64-spir64.out
--
Exit Code: -6

Command Output (stdout):
--
$ ":" "RUN: at line 1"
$ "/data/user/fwyzard/sycl/build/bin/clang++" "-fsycl" "-fsycl-targets=spir64-unknown-unknown-sycldevice,nvptx64-unknown-unknown-sycldevice" "/data/user/fwyzard/sycl/llvm/sycl/test/plugins/sycl-targets-order.cpp" "-o" "/data/user/fwyzard/sycl/build/tools/sycl/test/plugins/Output/sycl-targets-order.cpp.tmp-spir64-nvptx64.out"
$ ":" "RUN: at line 2" 
$ "env" "SYCL_BE=PI_OPENCL" "/data/user/fwyzard/sycl/build/tools/sycl/test/plugins/Output/sycl-targets-order.cpp.tmp-spir64-nvptx64.out"
# command output:
Running on SYCL device Intel(R) Gen9 HD Graphics NEO, driver version 20.07.15711
The results are correct!

$ ":" "RUN: at line 3" 
$ "env" "SYCL_BE=PI_CUDA" "/data/user/fwyzard/sycl/build/tools/sycl/test/plugins/Output/sycl-targets-order.cpp.tmp-spir64-nvptx64.out"
# command output:
Running on SYCL device Tesla K40c, driver version CUDA 10.20
  
# command stderr:
terminate called after throwing an instance of 'cl::sycl::feature_not_supported'
  what():  Online compilation is not supported in this context 0 (CL_SUCCESS)

error: command failed with exit status: -6

With these changes there is no mention of this test in the output of make check-sycl, so I assume it was successful.

sycl/test/plugins/sycl-targets-order.cpp

fwyzard · 2020-03-02T14:10:57Z

No, libsycl.so should be automatically added to LD_LIBRARY_PATH by lit.cfg.py.

Ah, I found the issue: sycl/test/CMakeLists.txt uses

set(LLVM_BUILD_LIBRARY_DIRS "${LLVM_BINARY_DIR}/lib/")

instead of

set(LLVM_BUILD_LIBRARY_DIRS "${LLVM_BINARY_DIR}/lib${LLVM_LIBDIR_SUFFIX}/")

so the tests will fail if a LLVM_LIBDIR_SUFFIX is specified.

I've opened #1227 for that.

Ruyk

Few minor comments, thanks for the patch

sycl/test/plugins/sycl-targets-order.cpp

fwyzard · 2020-03-02T16:06:14Z

2a787290e47 breaks sycl/test/kernel_from_file/hw.cpp, at least on my machine.
As it is not necessary t fix this issue, I would drop that commit, and possibly open an issue to keep track of it ?

…rgets Add a LIT test to check that both backends (PI_OPENCL, PI_CUDA) work irrespective of the order of the -fsycl-targets=... arguments. Signed-off-by: Andrea Bocci <[email protected]>

sycl/test/plugins/sycl-targets-order.cpp

bader · 2020-03-03T11:48:45Z

@fwyzard, thanks for working on this!

…ctor_tests * origin/sycl: (32 commits) [SYCL] Fix circular reference between events and queues (intel#1226) [CI][Doc] Use SSH to deploy GitHub Pages (intel#1232) [SYCL][CUDA][Test] Testing for use of CUDA primary context (intel#1174) [SYCL] allow underscore symbol in temporary directory name [SYCL] Reject zero length arrays (intel#1153) [SYCL] Fix static code analyzis concerns (intel#1189) [SYCL] Add more details about the -fintelfpga option (intel#1218) [SYCL][CUDA] Select only NVPTX64 device binaries (intel#1223) [SYCL] Reverse max work-group size order (intel#1177) [SYCL][Doc] Add GroupAlgorithms extension (intel#1079) [SYCL] Fix SYCL internal enumerators conflict with user defined macro (intel#1188) [SYCL][CUDA] Fixes context release and unnamed context scope (intel#1207) [SYCL][CUDA] Fix context creation property parsing [CUDA][PI] clang-format pi.h [SYCL][CUDA] Handle the case of not having any CUDA device (intel#1212) [SYCL] Fix check-sycl-deploy target problems (intel#1165) [SYCL] Disable tests which take more than 5 minutes (intel#1220) [SYCL] Make context constructors explicit to avoid unintended conversions (intel#1219) [SYCL][NFC] Add clang-format configuration file for SYCL LIT tests (intel#1224) [SYCL] Fix command cleanup invoked from multiple threads (intel#1214) ...

…_accessor_refactor * origin/sycl: (38 commits) [SYCL] Fix device::get_devices() with a non-host device type (intel#1235) [SYCL][PI][CUDA] Implement kernel and kernel-group information queries (intel#1180) [SYCL] Remove default error code value in exception (intel#1150) [SYCL] Fix devicelib assert LIT test (intel#1245) [SYCL] Set aux-target-cpu for SYCL offload device compilation (intel#1225) [SYCL] Remove fabs and ceil from the list of unsupported math functions (intel#1217) [SYCL] Fix circular reference between events and queues (intel#1226) [CI][Doc] Use SSH to deploy GitHub Pages (intel#1232) [SYCL][CUDA][Test] Testing for use of CUDA primary context (intel#1174) [SYCL] allow underscore symbol in temporary directory name [SYCL] Reject zero length arrays (intel#1153) [SYCL] Fix static code analyzis concerns (intel#1189) [SYCL] Add more details about the -fintelfpga option (intel#1218) [SYCL][CUDA] Select only NVPTX64 device binaries (intel#1223) [SYCL] Reverse max work-group size order (intel#1177) [SYCL][Doc] Add GroupAlgorithms extension (intel#1079) [SYCL] Fix SYCL internal enumerators conflict with user defined macro (intel#1188) [SYCL][CUDA] Fixes context release and unnamed context scope (intel#1207) [SYCL][CUDA] Fix context creation property parsing [CUDA][PI] clang-format pi.h ...

fwyzard mentioned this pull request Mar 1, 2020

-fsycl-targets is order dependent #1194

Closed

fwyzard added 2 commits March 1, 2020 12:14

[SYCL][CUDA] Add NVPTX64 binary target

2b7d50a

Add the binary target identifier "nvptx64" for NVIDIA PTX devices. Signed-off-by: Andrea Bocci <[email protected]>

[SYCL][CUDA] Select only NVPTX64 device binaries

48ed377

Search through the available binary images and select the first one for the PI_DEVICE_BINARY_TARGET_NVPTX64 ("nvptx64") target. Return PI_INVALID_BINARY if no "nvptx64" image is available. Signed-off-by: Andrea Bocci <[email protected]>

fwyzard force-pushed the fix_issue_1194 branch from 1321eb2 to 2a78729 Compare March 1, 2020 11:15

bader requested a review from sergey-semenov March 1, 2020 11:21

bader assigned romanovvlad Mar 1, 2020

bader added the cuda CUDA back-end label Mar 1, 2020

bader linked an issue Mar 1, 2020 that may be closed by this pull request

-fsycl-targets is order dependent #1194

Closed

bader reviewed Mar 2, 2020

View reviewed changes

sycl/test/plugins/sycl-targets-order.cpp Show resolved Hide resolved

fwyzard force-pushed the fix_issue_1194 branch from 6713a7b to 78de570 Compare March 2, 2020 14:09

Ruyk reviewed Mar 2, 2020

View reviewed changes

sycl/test/plugins/sycl-targets-order.cpp Outdated Show resolved Hide resolved

sycl/test/plugins/sycl-targets-order.cpp Outdated Show resolved Hide resolved

sycl/test/plugins/sycl-targets-order.cpp Show resolved Hide resolved

fwyzard force-pushed the fix_issue_1194 branch from 78de570 to 8c35031 Compare March 2, 2020 15:15

[SYCL] LIT test to check for the impact of the order of the -fsycl-ta…

b955e11

…rgets Add a LIT test to check that both backends (PI_OPENCL, PI_CUDA) work irrespective of the order of the -fsycl-targets=... arguments. Signed-off-by: Andrea Bocci <[email protected]>

fwyzard force-pushed the fix_issue_1194 branch from 8c35031 to b955e11 Compare March 2, 2020 16:09

romanovvlad reviewed Mar 2, 2020

View reviewed changes

sycl/test/plugins/sycl-targets-order.cpp Show resolved Hide resolved

romanovvlad approved these changes Mar 2, 2020

View reviewed changes

fwyzard closed this Mar 2, 2020

fwyzard deleted the fix_issue_1194 branch March 2, 2020 19:38

fwyzard restored the fix_issue_1194 branch March 2, 2020 19:46

fwyzard reopened this Mar 2, 2020

sergey-semenov approved these changes Mar 3, 2020

View reviewed changes

bader merged commit bcefeb3 into intel:sycl Mar 3, 2020

fwyzard deleted the fix_issue_1194 branch March 3, 2020 12:09

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[SYCL][CUDA] Select only NVPTX64 device binaries #1223

[SYCL][CUDA] Select only NVPTX64 device binaries #1223

Uh oh!

fwyzard commented Mar 1, 2020

Uh oh!

fwyzard commented Mar 1, 2020

Uh oh!

bader commented Mar 1, 2020

Uh oh!

fwyzard commented Mar 1, 2020

Uh oh!

bader commented Mar 1, 2020

Uh oh!

fwyzard commented Mar 1, 2020 •

edited

Loading

Uh oh!

romanovvlad commented Mar 2, 2020

Uh oh!

fwyzard commented Mar 2, 2020

Uh oh!

Uh oh!

fwyzard commented Mar 2, 2020 •

edited

Loading

Uh oh!

Ruyk left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

fwyzard commented Mar 2, 2020

Uh oh!

Uh oh!

bader commented Mar 3, 2020

Uh oh!

Uh oh!

[SYCL][CUDA] Select only NVPTX64 device binaries #1223

[SYCL][CUDA] Select only NVPTX64 device binaries #1223

Uh oh!

Conversation

fwyzard commented Mar 1, 2020

Uh oh!

fwyzard commented Mar 1, 2020

Uh oh!

bader commented Mar 1, 2020

Uh oh!

fwyzard commented Mar 1, 2020

Uh oh!

bader commented Mar 1, 2020

Uh oh!

fwyzard commented Mar 1, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

romanovvlad commented Mar 2, 2020

Uh oh!

fwyzard commented Mar 2, 2020

Uh oh!

Uh oh!

fwyzard commented Mar 2, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Ruyk left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

fwyzard commented Mar 2, 2020

Uh oh!

Uh oh!

bader commented Mar 3, 2020

Uh oh!

Uh oh!

fwyzard commented Mar 1, 2020 •

edited

Loading

fwyzard commented Mar 2, 2020 •

edited

Loading