CUB building issues on Windows 11

[sktistakis]

I'm trying to build pytorch3d on Windows 11 using CUDA 11.7 and PyTorch 1.9.1 but having some issues coming from CUB.

I am starting the building process on "x64 Native Tools Command Prompt for VS 2019" using a conda environment:
python setup.py install
What I've done so far:

• set CUB_HOME=C:\Program Files\cub-1.17.0\cub
• set DISTUTILS_USE_SDK=1
• installed ninja and added it to the Path.
• Changed setup.py: Commented out those lines:

#if os.name != "nt":
            #nvcc_args.append("-std=c++14")

• Added cl.exe folder to PATH:
  C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.29.30133\bin\Hostx64\x64

• Added this folder to path as well (can't remember why)
  C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\MSBuild\Current\Bin

What I have tried as well:
Using the CUB version which is already integrated in CUDA 11.7.

• set CUB_HOME= C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.7\include
  But this lead to the same issue.

Installing pytorch3d with PyTorch 1.9.1 CPU worked, but not with PyTorch Cuda.

Here is one of the the relevant build errors (it repeats when compiling different 'pytorch3d/csrc/pulsar/cuda/renderer...' files):

C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v11.7/include\cub/device/dispatch/dispatch_segmented_sort.cuh(338): error: invalid combination of type specifiers

C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v11.7/include\cub/device/dispatch/dispatch_segmented_sort.cuh(338): error: expected an identifier

C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v11.7/include\cub/device/dispatch/dispatch_segmented_sort.cuh(379): error: expected a member name

3 errors detected in the compilation of "C:/Users/sktis/source/repos/pytorch3d/pytorch3d/csrc/pulsar/cuda/renderer.construct.gpu.cu".
renderer.construct.gpu.cu
ninja: build stopped: subcommand failed.
Traceback (most recent call last):
  File "C:\Users\sktis\anaconda3\envs\recorder_ds\lib\site-packages\torch\utils\cpp_extension.py", line 1666, in _run_ninja_build
    subprocess.run(
  File "C:\Users\sktis\anaconda3\envs\recorder_ds\lib\subprocess.py", line 528, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['ninja', '-v']' returned non-zero exit status 1.

The full log can be found here:
https://file.io/e3oV2zOkCDqk

[bottler]

(CUDA 11.7 is new to me.) Some of the steps here are odd. Adding to the path for cl for example - if you are in the right prompt you should have the right cl. I wonder if there's a clash of compiler versions.

I would expect with CUDA 11.7 that it's easiest not to set CUB_HOME and not to think about cub. Just let the cuda toolkit do its thing.

Is VS2019 designed to be compatible with CUDA 11.7? Worth checking.

[sktistakis]

Thank you for your reply Jeremy!

I removed cl and msbuild from the path - but it still failed.

Then I removed the CUB_HOME, but it still failed with the same errors.
I noticed that it prints this warning:
" UserWarning: The environment variable CUB_HOME was not found. NVIDIA CUB is required for compilation and can be downloaded..."

[bottler]

That warning shouldn't be fatal. (also VS2019 should be fine).

This problem is not pytorch3d specific. It looks similar to https://forums.developer.nvidia.com/t/cub-1-5-fails-to-build-when-windows-h-was-previously-included/210367 where installing cub 1.16 helped. Can you try that?

[sktistakis]

I tried this, but unfortunately the errors are still the same.
And neither did cub 1.17 help.

[38/65] C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.7\bin\nvcc --generate-dependencies-with-compile --dependency-output C:\Users\sktis\source\repos\pytorch3d\build\temp.win-amd64-cpython-39\Release\Users\sktis\source\repos\pytorch3d\pytorch3d\csrc\pulsar\cuda\renderer.forward.gpu.obj.d --use-local-env -Xcompiler /MD -Xcompiler /wd4819 -Xcompiler /wd4251 -Xcompiler /wd4244 -Xcompiler /wd4267 -Xcompiler /wd4275 -Xcompiler /wd4018 -Xcompiler /wd4190 -Xcompiler /EHsc -Xcudafe --diag_suppress=base_class_has_different_dll_interface -Xcudafe --diag_suppress=field_without_dll_interface -Xcudafe --diag_suppress=dll_interface_conflict_none_assumed -Xcudafe --diag_suppress=dll_interface_conflict_dllexport_assumed -DWITH_CUDA -DTHRUST_IGNORE_CUB_VERSION_CHECK -IC:\Users\sktis\source\repos\pytorch3d\pytorch3d\csrc "-IC:\Program Files\cub-1.16.0\cub" -IC:\Users\sktis\anaconda3\envs\recorder_ds\lib\site-packages\torch\include -IC:\Users\sktis\anaconda3\envs\recorder_ds\lib\site-packages\torch\include\torch\csrc\api\include -IC:\Users\sktis\anaconda3\envs\recorder_ds\lib\site-packages\torch\include\TH -IC:\Users\sktis\anaconda3\envs\recorder_ds\lib\site-packages\torch\include\THC "-IC:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.7\include" -IC:\Users\sktis\anaconda3\envs\recorder_ds\include -IC:\Users\sktis\anaconda3\envs\recorder_ds\Include "-IC:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.29.30133\include" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\ucrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\shared" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\winrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\cppwinrt" -c C:\Users\sktis\source\repos\pytorch3d\pytorch3d\csrc\pulsar\cuda\renderer.forward.gpu.cu -o C:\Users\sktis\source\repos\pytorch3d\build\temp.win-amd64-cpython-39\Release\Users\sktis\source\repos\pytorch3d\pytorch3d\csrc\pulsar\cuda\renderer.forward.gpu.obj -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -DCUDA_HAS_FP16=1 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=_C -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86
FAILED: C:/Users/sktis/source/repos/pytorch3d/build/temp.win-amd64-cpython-39/Release/Users/sktis/source/repos/pytorch3d/pytorch3d/csrc/pulsar/cuda/renderer.forward.gpu.obj
C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.7\bin\nvcc --generate-dependencies-with-compile --dependency-output C:\Users\sktis\source\repos\pytorch3d\build\temp.win-amd64-cpython-39\Release\Users\sktis\source\repos\pytorch3d\pytorch3d\csrc\pulsar\cuda\renderer.forward.gpu.obj.d --use-local-env -Xcompiler /MD -Xcompiler /wd4819 -Xcompiler /wd4251 -Xcompiler /wd4244 -Xcompiler /wd4267 -Xcompiler /wd4275 -Xcompiler /wd4018 -Xcompiler /wd4190 -Xcompiler /EHsc -Xcudafe --diag_suppress=base_class_has_different_dll_interface -Xcudafe --diag_suppress=field_without_dll_interface -Xcudafe --diag_suppress=dll_interface_conflict_none_assumed -Xcudafe --diag_suppress=dll_interface_conflict_dllexport_assumed -DWITH_CUDA -DTHRUST_IGNORE_CUB_VERSION_CHECK -IC:\Users\sktis\source\repos\pytorch3d\pytorch3d\csrc "-IC:\Program Files\cub-1.16.0\cub" -IC:\Users\sktis\anaconda3\envs\recorder_ds\lib\site-packages\torch\include -IC:\Users\sktis\anaconda3\envs\recorder_ds\lib\site-packages\torch\include\torch\csrc\api\include -IC:\Users\sktis\anaconda3\envs\recorder_ds\lib\site-packages\torch\include\TH -IC:\Users\sktis\anaconda3\envs\recorder_ds\lib\site-packages\torch\include\THC "-IC:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.7\include" -IC:\Users\sktis\anaconda3\envs\recorder_ds\include -IC:\Users\sktis\anaconda3\envs\recorder_ds\Include "-IC:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.29.30133\include" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\ucrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\shared" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\winrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\cppwinrt" -c C:\Users\sktis\source\repos\pytorch3d\pytorch3d\csrc\pulsar\cuda\renderer.forward.gpu.cu -o C:\Users\sktis\source\repos\pytorch3d\build\temp.win-amd64-cpython-39\Release\Users\sktis\source\repos\pytorch3d\pytorch3d\csrc\pulsar\cuda\renderer.forward.gpu.obj -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -DCUDA_HAS_FP16=1 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=_C -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86
C:\Users\sktis\source\repos\pytorch3d\pytorch3d\csrc\pulsar\include\../global.h(33): warning #20236-D: pragma "diag_suppress" is deprecated, use "nv_diag_suppress" instead

C:\Users\sktis\source\repos\pytorch3d\pytorch3d\csrc\pulsar\include\../global.h(34): warning #20236-D: pragma "diag_suppress" is deprecated, use "nv_diag_suppress" instead

C:\Users\sktis\source\repos\pytorch3d\pytorch3d\csrc\pulsar\include\../global.h(35): warning #20236-D: pragma "diag_suppress" is deprecated, use "nv_diag_suppress" instead

C:\Users\sktis\source\repos\pytorch3d\pytorch3d\csrc\pulsar\include\../global.h(36): warning #20236-D: pragma "diag_suppress" is deprecated, use "nv_diag_suppress" instead

C:\Users\sktis\source\repos\pytorch3d\pytorch3d\csrc\pulsar\include\../global.h(37): warning #20236-D: pragma "diag_suppress" is deprecated, use "nv_diag_suppress" instead

C:\Users\sktis\source\repos\pytorch3d\pytorch3d\csrc\pulsar\include\../global.h(33): warning #20236-D: pragma "diag_suppress" is deprecated, use "nv_diag_suppress" instead

C:\Users\sktis\source\repos\pytorch3d\pytorch3d\csrc\pulsar\include\../global.h(34): warning #20236-D: pragma "diag_suppress" is deprecated, use "nv_diag_suppress" instead

C:\Users\sktis\source\repos\pytorch3d\pytorch3d\csrc\pulsar\include\../global.h(35): warning #20236-D: pragma "diag_suppress" is deprecated, use "nv_diag_suppress" instead

C:\Users\sktis\source\repos\pytorch3d\pytorch3d\csrc\pulsar\include\../global.h(36): warning #20236-D: pragma "diag_suppress" is deprecated, use "nv_diag_suppress" instead

C:\Users\sktis\source\repos\pytorch3d\pytorch3d\csrc\pulsar\include\../global.h(37): warning #20236-D: pragma "diag_suppress" is deprecated, use "nv_diag_suppress" instead

C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v11.7/include\cub/device/dispatch/dispatch_segmented_sort.cuh(338): error: invalid combination of type specifiers

C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v11.7/include\cub/device/dispatch/dispatch_segmented_sort.cuh(338): error: expected an identifier

C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v11.7/include\cub/device/dispatch/dispatch_segmented_sort.cuh(379): error: expected a member name

3 errors detected in the compilation of "C:/Users/sktis/source/repos/pytorch3d/pytorch3d/csrc/pulsar/cuda/renderer.forward.gpu.cu".
renderer.forward.gpu.cu

[sktistakis]

This is the full log:

pytorch_3d_debug.txt

[bottler]

The error C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v11.7/include\cub/device/dispatch/dispatch_segmented_sort.cuh(338): error: invalid combination of type specifiers doesn't correspond to the file in cub version 1.17 https://github.com/NVIDIA/cub/blob/1.17.X/cub/device/device_segmented_sort.cuh#L338. In particular line 338 is in the middle of a long comment. I don't think you are actually using cub version 1.17. Can you work out why? Perhaps CUB_HOME isn't working?

[sktistakis]

You were right!
CUDA 11.7 has CUB version 1.15 integrated.
Despite CUB_HOME pointing to the 1.17 version, CUDA used its own internal version of CUB, which was 1.15.

I deleted the cub folder in the CUDA toolkit and replaced it with the cub 1.17:
C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v11.7/include/cub

It successfully builds now, thank you!

[lucasjinreal]

@sktistakis I replaced it but still error.

I am using CUDA11.8 and tried cub 1.17 cub 1.17.1 cub 1.17.2

all fails!!

[lucasjinreal]

@bottler Can u please make pytorch3d compatible on cuda11.8 windows?

I current tried 2 computers with cuda 11.8 windows, all build failed.

[bottler]

If you still need help, please open a new issue.

[zhuchenyang]

You were right! CUDA 11.7 has CUB version 1.15 integrated. Despite CUB_HOME pointing to the 1.17 version, CUDA used its own internal version of CUB, which was 1.15.

I deleted the cub folder in the CUDA toolkit and replaced it with the cub 1.17: C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v11.7/include/cub

It successfully builds now, thank you!

YES! It also works for CUDA 11.8.

[nitinmukesh]

See if this helps
#1752