[SYCL][CUDA] reverted #6577

[JackAKirk]

Signed-off-by: JackAKirk [email protected]

[steffenlarsen]

@JackAKirk - Can you confirm that the CUDA test failures are unrelated?

[JackAKirk]

@JackAKirk - Can you confirm that the CUDA test failures are unrelated?

The failure Assert/assert_in_simultaneously_multiple_tus.cpp is I think unrelated to this PR and the changes that this PR is reverting: #6577

At first I thought that the failures of tests like Assert/assert_in_simultaneously_multiple_tus.cpp could be due to #6361 because the failures followed soon afterwards and it initially looked like that PR added calls to increment the context without a mutex but I was too quick to put up #6577 and missed that the refcount was std::atomic, hence the reversion here.
I am working on a fix for what I believe is the true origin of these flaky test failures, as I updated here: #6463 (comment)