Skip to content

Check for required headers before compilation #514

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 6 commits into from
Mar 14, 2025
Merged

Check for required headers before compilation #514

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
284e666
Check for required headers early
vzhurba01 Mar 13, 2025
e5c6620
Run pre-commit
vzhurba01 Mar 13, 2025
6c38b3c
Wording
vzhurba01 Mar 13, 2025
a97ada5
Different wording
vzhurba01 Mar 13, 2025
6368549
Update release notes
vzhurba01 Mar 13, 2025
439681e
Apply review
vzhurba01 Mar 13, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions cuda_bindings/cuda/bindings/_lib/cyruntime/cyruntime.pxd.in
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -108,9 +108,9 @@ from libcpp cimport bool
{{if 'cudaCreateSurfaceObject' in found_functions}}cdef cudaError_t _cudaCreateSurfaceObject(cudaSurfaceObject_t* pSurfObject, const cudaResourceDesc* pResDesc) except ?cudaErrorCallRequiresNewerDriver nogil{{endif}}
{{if 'cudaGetTextureObjectResourceDesc' in found_functions}}cdef cudaError_t _cudaGetTextureObjectResourceDesc(cudaResourceDesc* pResDesc, cudaTextureObject_t texObject) except ?cudaErrorCallRequiresNewerDriver nogil{{endif}}
{{if 'cudaGraphicsEGLRegisterImage' in found_functions}}cdef cudaError_t _cudaGraphicsEGLRegisterImage(cudaGraphicsResource_t* pCudaResource, EGLImageKHR image, unsigned int flags) except ?cudaErrorCallRequiresNewerDriver nogil{{endif}}
{{if 'cudaEGLStreamProducerPresentFrame' in found_functions}}cdef cudaError_t _cudaEGLStreamProducerPresentFrame(cudaEglStreamConnection* conn, cudaEglFrame eglframe, cudaStream_t* pStream) except ?cudaErrorCallRequiresNewerDriver nogil{{endif}}
{{if 'cudaEGLStreamProducerReturnFrame' in found_functions}}cdef cudaError_t _cudaEGLStreamProducerReturnFrame(cudaEglStreamConnection* conn, cudaEglFrame* eglframe, cudaStream_t* pStream) except ?cudaErrorCallRequiresNewerDriver nogil{{endif}}
{{if 'cudaGraphicsResourceGetMappedEglFrame' in found_functions}}cdef cudaError_t _cudaGraphicsResourceGetMappedEglFrame(cudaEglFrame* eglFrame, cudaGraphicsResource_t resource, unsigned int index, unsigned int mipLevel) except ?cudaErrorCallRequiresNewerDriver nogil{{endif}}
{{if True}}cdef cudaError_t _cudaEGLStreamProducerPresentFrame(cudaEglStreamConnection* conn, cudaEglFrame eglframe, cudaStream_t* pStream) except ?cudaErrorCallRequiresNewerDriver nogil{{endif}}
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there an easy way to leave a helpful hint here?

What I have in mind:

# For tagging <explanation here>:
helpful_hint_here = True

Then, instead of {{if True}} → {{if helpful_hint_here}}

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For a complete solution it's not that easy. Updating this file and placing the explanation in setup.py is simple, the problem is that this pattern is used all across cuda.bindings as part of the auto-generation.

On a major release we'll resolve issue #488 and I think most if not all of these instances will disappear, so lets revisit this then.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds good. Thanks for the explanation!

{{if True}}cdef cudaError_t _cudaEGLStreamProducerReturnFrame(cudaEglStreamConnection* conn, cudaEglFrame* eglframe, cudaStream_t* pStream) except ?cudaErrorCallRequiresNewerDriver nogil{{endif}}
{{if True}}cdef cudaError_t _cudaGraphicsResourceGetMappedEglFrame(cudaEglFrame* eglFrame, cudaGraphicsResource_t resource, unsigned int index, unsigned int mipLevel) except ?cudaErrorCallRequiresNewerDriver nogil{{endif}}
{{if True}}cdef cudaError_t _cudaVDPAUSetVDPAUDevice(int device, VdpDevice vdpDevice, VdpGetProcAddress* vdpGetProcAddress) except ?cudaErrorCallRequiresNewerDriver nogil{{endif}}
{{if 'cudaArrayGetMemoryRequirements' in found_functions}}cdef cudaError_t _cudaArrayGetMemoryRequirements(cudaArrayMemoryRequirements* memoryRequirements, cudaArray_t array, int device) except ?cudaErrorCallRequiresNewerDriver nogil{{endif}}
{{if 'cudaMipmappedArrayGetMemoryRequirements' in found_functions}}cdef cudaError_t _cudaMipmappedArrayGetMemoryRequirements(cudaArrayMemoryRequirements* memoryRequirements, cudaMipmappedArray_t mipmap, int device) except ?cudaErrorCallRequiresNewerDriver nogil{{endif}}
Expand Down
11 changes: 11 additions & 0 deletions cuda_bindings/cuda/bindings/_lib/cyruntime/cyruntime.pyx.in
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2206,6 +2206,7 @@ cdef cudaError_t _cudaGetTextureObjectResourceDesc(cudaResourceDesc* pResDesc, c
return err

{{endif}}
{{if True}}
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we use the helpful_hint_here, too?

Out of curiosity: Are these {{if True}} actually required? — Even if not, I think helpful hints here are useful.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

They're not needed but I want it there for consistency with the rest of the source base. Visually at least, it helps identify which sections use redefined types rather than extern with a C header.

Overall same comment as above, lets revisit this after #488.


cdef cudaError_t _cudaEGLStreamProducerPresentFrame(cudaEglStreamConnection* conn, cudaEglFrame eglframe, cudaStream_t* pStream) except ?cudaErrorCallRequiresNewerDriver nogil:
cdef cudaError_t err = cudaSuccess
Expand All @@ -2222,6 +2223,9 @@ cdef cudaError_t _cudaEGLStreamProducerPresentFrame(cudaEglStreamConnection* con
_setLastError(err)
return err

{{endif}}
{{if True}}

cdef cudaError_t _cudaEGLStreamProducerReturnFrame(cudaEglStreamConnection* conn, cudaEglFrame* eglframe, cudaStream_t* pStream) except ?cudaErrorCallRequiresNewerDriver nogil:
cdef cudaError_t err = cudaSuccess
err = m_global.lazyInitContextState()
Expand All @@ -2242,6 +2246,9 @@ cdef cudaError_t _cudaEGLStreamProducerReturnFrame(cudaEglStreamConnection* conn
return err
return err

{{endif}}
{{if True}}

cdef cudaError_t _cudaGraphicsResourceGetMappedEglFrame(cudaEglFrame* eglFrame, cudaGraphicsResource_t resource, unsigned int index, unsigned int mipLevel) except ?cudaErrorCallRequiresNewerDriver nogil:
cdef cudaError_t err = cudaSuccess
err = m_global.lazyInitContextState()
Expand All @@ -2259,9 +2266,13 @@ cdef cudaError_t _cudaGraphicsResourceGetMappedEglFrame(cudaEglFrame* eglFrame,
return err
return err

{{endif}}
{{if True}}

cdef cudaError_t _cudaVDPAUSetVDPAUDevice(int device, VdpDevice vdpDevice, VdpGetProcAddress* vdpGetProcAddress) except ?cudaErrorCallRequiresNewerDriver nogil:
return cudaErrorNotSupported

{{endif}}
{{if 'cudaArrayGetMemoryRequirements' in found_functions}}

cdef cudaError_t _cudaArrayGetMemoryRequirements(cudaArrayMemoryRequirements* memoryRequirements, cudaArray_t array, int device) except ?cudaErrorCallRequiresNewerDriver nogil:
Expand Down
1 change: 1 addition & 0 deletions cuda_bindings/docs/source/release/12.X.Y-notes.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,3 +9,4 @@ Highlights

* The ``cuda.bindings.nvvm`` Python module was added, wrapping the
`libNVVM C API <https://docs.nvidia.com/cuda/libnvvm-api/>`_.
* Source build error checking added for missing required headers
165 changes: 96 additions & 69 deletions cuda_bindings/setup.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -48,8 +48,11 @@
# ----------------------------------------------------------------------
# Parse user-provided CUDA headers

header_dict = {
"driver": ["cuda.h", "cudaProfiler.h", "cudaEGL.h", "cudaGL.h", "cudaVDPAU.h"],
required_headers = {
"driver": [
"cuda.h",
"cudaProfiler.h",
],
"runtime": [
"driver_types.h",
"vector_types.h",
Expand All @@ -61,32 +64,42 @@
"device_types.h",
"driver_functions.h",
"cuda_profiler_api.h",
"cuda_egl_interop.h",
"cuda_gl_interop.h",
"cuda_vdpau_interop.h",
],
"nvrtc": ["nvrtc.h"],
"nvrtc": [
"nvrtc.h",
],
# During compilation, Cython will reference C headers that are not
# explicitly parsed above. These are the known dependencies:
#
# - crt/host_defines.h
# - builtin_types.h
# - cuda_device_runtime_api.h
}

replace = {
" __device_builtin__ ": " ",
"CUDARTAPI ": " ",
"typedef __device_builtin__ enum cudaError cudaError_t;": "typedef cudaError cudaError_t;",
"typedef __device_builtin__ enum cudaOutputMode cudaOutputMode_t;": "typedef cudaOutputMode cudaOutputMode_t;",
"typedef enum cudaError cudaError_t;": "typedef cudaError cudaError_t;",
"typedef enum cudaOutputMode cudaOutputMode_t;": "typedef cudaOutputMode cudaOutputMode_t;",
"typedef enum cudaDataType_t cudaDataType_t;": "",
"typedef enum libraryPropertyType_t libraryPropertyType_t;": "",
" enum ": " ",
", enum ": ", ",
"\\(enum ": "(",
}

found_types = []
found_functions = []
found_values = []
found_struct = []
struct_list = {}
def fetch_header_paths(required_headers, include_path_list):
header_dict = {}
missing_headers = []
for library, header_list in required_headers.items():
header_paths = []
for header in header_list:
path_candidate = [os.path.join(path, header) for path in include_path_list]
for path in path_candidate:
if os.path.exists(path):
header_paths += [path]
break
else:
missing_headers += [header]

# Update dictionary with validated paths to headers
header_dict[library] = header_paths

if missing_headers:
error_message = "Couldn't find required headers: "
error_message += ", ".join([header for header in missing_headers])
raise RuntimeError(f'{error_message}\nIs CUDA_HOME setup correctly? (CUDA_HOME="{CUDA_HOME}")')

return header_dict


class Struct:
Expand Down Expand Up @@ -117,52 +130,66 @@ def __repr__(self):
return f"{self._name}: {self._member_names} with types {self._member_types}"


include_path_list = [os.path.join(path, "include") for path in CUDA_HOME]
print(f'Parsing headers in "{include_path_list}" (Caching = {PARSER_CACHING})')
for library, header_list in header_dict.items():
header_paths = []
for header in header_list:
path_candidate = [os.path.join(path, header) for path in include_path_list]
for path in path_candidate:
if os.path.exists(path):
header_paths += [path]
break
if not os.path.exists(path):
print(f"Missing header {header}")

print(f"Parsing {library} headers")
parser = CParser(
header_paths, cache="./cache_{}".format(library.split(".")[0]) if PARSER_CACHING else None, replace=replace
)
def parse_headers(header_dict):
found_types = []
found_functions = []
found_values = []
found_struct = []
struct_list = {}

replace = {
" __device_builtin__ ": " ",
"CUDARTAPI ": " ",
"typedef __device_builtin__ enum cudaError cudaError_t;": "typedef cudaError cudaError_t;",
"typedef __device_builtin__ enum cudaOutputMode cudaOutputMode_t;": "typedef cudaOutputMode cudaOutputMode_t;",
"typedef enum cudaError cudaError_t;": "typedef cudaError cudaError_t;",
"typedef enum cudaOutputMode cudaOutputMode_t;": "typedef cudaOutputMode cudaOutputMode_t;",
"typedef enum cudaDataType_t cudaDataType_t;": "",
"typedef enum libraryPropertyType_t libraryPropertyType_t;": "",
" enum ": " ",
", enum ": ", ",
"\\(enum ": "(",
}

print(f'Parsing headers in "{include_path_list}" (Caching = {PARSER_CACHING})')
for library, header_paths in header_dict.items():
print(f"Parsing {library} headers")
parser = CParser(
header_paths, cache="./cache_{}".format(library.split(".")[0]) if PARSER_CACHING else None, replace=replace
)

if library == "driver":
CUDA_VERSION = parser.defs["macros"].get("CUDA_VERSION", "Unknown")
print(f"Found CUDA_VERSION: {CUDA_VERSION}")

# Combine types with others since they sometimes get tangled
found_types += {key for key in parser.defs["types"]}
found_types += {key for key in parser.defs["structs"]}
found_types += {key for key in parser.defs["unions"]}
found_types += {key for key in parser.defs["enums"]}
found_functions += {key for key in parser.defs["functions"]}
found_values += {key for key in parser.defs["values"]}

for key, value in parser.defs["structs"].items():
struct_list[key] = Struct(key, value["members"])
for key, value in parser.defs["unions"].items():
struct_list[key] = Struct(key, value["members"])

for key, value in struct_list.items():
if key.startswith("anon_union") or key.startswith("anon_struct"):
continue

if library == "driver":
CUDA_VERSION = parser.defs["macros"].get("CUDA_VERSION", "Unknown")
print(f"Found CUDA_VERSION: {CUDA_VERSION}")

# Combine types with others since they sometimes get tangled
found_types += {key for key in parser.defs["types"]}
found_types += {key for key in parser.defs["structs"]}
found_types += {key for key in parser.defs["unions"]}
found_types += {key for key in parser.defs["enums"]}
found_functions += {key for key in parser.defs["functions"]}
found_values += {key for key in parser.defs["values"]}

for key, value in parser.defs["structs"].items():
struct_list[key] = Struct(key, value["members"])
for key, value in parser.defs["unions"].items():
struct_list[key] = Struct(key, value["members"])

for key, value in struct_list.items():
if key.startswith("anon_union") or key.startswith("anon_struct"):
continue

found_struct += [key]
discovered = value.discoverMembers(struct_list, key)
if discovered:
found_struct += discovered

if len(found_functions) == 0:
raise RuntimeError(f'Parser found no functions. Is CUDA_HOME setup correctly? (CUDA_HOME="{CUDA_HOME}")')
found_struct += [key]
discovered = value.discoverMembers(struct_list, key)
if discovered:
found_struct += discovered

return found_types, found_functions, found_values, found_struct, struct_list


include_path_list = [os.path.join(path, "include") for path in CUDA_HOME]
header_dict = fetch_header_paths(required_headers, include_path_list)
found_types, found_functions, found_values, found_struct, struct_list = parse_headers(header_dict)

# ----------------------------------------------------------------------
# Generate
Expand Down
Loading