proposal: introduce a ${REPODIR} as a possible variable to extend in #cgo directives

[tdegris]

Proposal Details

The main goal of this proposal is to make it easy to include C header files from other Go modules.

Similarly to ${SRCDIR}, something like ${REPODIR} could help using C/cgo headers and libraries from other Go modules. One would then be able to include a header file using:

// #cgo CFLAGS: -I${REPODIR}
// #include <github.com/someuser/modname/cgo/a_fancy_header.h> // Accessed via ${REPODIR}
import "C"

where ${REPODIR} would contain all the dependencies of the project at the correct version specified by go.mod.

Alternatively, the documentation available on Go project layout and cgo (wiki or cmd) does not seem to cover the case of projects with multiple Go modules exporting cgo code to other Go modules. At the moment, one has to rely on trials and errors to come up with a layout, what path to use in header files, etc.

Adding some guidance to https://go.dev/doc/modules/layout would help tremendously, specifically what conventions to use so that cgo can use headers and libraries from other Go modules. Perhaps better documentation can make this proposal unnecessary.

[gabyhelp]

Related Issues

• proposal: cmd/cgo: expand $GOPATH in -I and -L directives #20606 (closed)

_{(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)}

[seankhliao]

I think direct access to things outside the module is generally considered out of scope and not something to encourage.

[tdegris]

If a Go module exports some code with cgo, what would you recommend to access the header files generated by cgo or header files used by cgo code?

Would it be possible to update the cgo documentation mentioned above so that users know what to do and what is recommended?

[neild]

I don't know if we'd want to make any change here (cgo isn't really my area), but it doesn't seem necessarily incorrect to allow one cgo-using module to include a header file from a module in its dependencies. Going to reopen this for the moment at least to allow for more discussion.

[tdegris]

Unless I am mistaken, the related proposal cmd/cgo: expand $GOPATH in -I and -L directives was proposed before Go modules existed. It seems to me that Go modules now adresses all the comments in the issue.

[ianlancetaylor]

REPODIR seems costly to implement.

I agree that there is something to do here, but I don't think it's that. I don't know that it can be done with documentation—that is, I don't know how to write that documentation myself—but if it can that seems appropriate.

[tdegris]

Some other ideas...

An alternative would be for the variable to take a module as an argument. Something like:

// #cgo CFLAGS: -I${REPODIR:github.com/someuser/modname}
// #include <cgo/a_fancy_header.h> // Accessed via ${REPODIR}
import "C"

As a data point, at the moment, I have built a utility which creates a deps folder inside the module which contains symbolic links to all the versioned module folders in the Go module cache:

project-root-directory/
    go.mod
    go.sum
    deps/
        github.com/
            someuser/
                modname -> <symbolic link to the version in $GOMODCACHE. 
                            For example $GOMODCACHE/github.com/someuser/modname@v1.2.3>

so that a dependency can be accessed using:

// #cgo CFLAGS: -I${SRCDIR}/../../deps
// #include <github.com/someuser/modname/cgo/a_fancy_header.h> 
import "C"

One problem is that one has to run the utility before compiling or running the project tests (the deps folder is not submitted). A good thing about that solution is that deps can then be used to compile C/C++ code. Not saying this is what should be done, just mentioning some ideas.

[mpx]

There is a potentially related problem I've looked at recently with Protobufs. Eg:

• Module A' includes the API generated from the set of proto files A.
• Module B' includes the API generated from the set of proto files B. Proto files B reference proto files A.

How can module B' ensure it is using the correct proto schema for A when generating the protobuf API?

Ideally the source proto schema could be included in the module and referenced from another module. Otherwise module B' could use a more recent schema than used in module A'.

If there was a way to reference the the current version for a particuarly module it may be practical to include and reference the correct matching schema.

This is a quite a complex area here, so the solution likely isn't easy and has many other concerns. However, perhaps it's useful to consider the overlap here for any potential generic solution beyond just accessing header files.

[tdegris]

@mpx

I agree that protobuf dependencies between modules look like a similar problem compared to accessing header files. I guess the general problem is how a module can access at compile time ressources of a dependency. Perhaps the issue should be renamed.

Just to better understand the problem: does creating a deps folder containing links to all the module dependencies solve your problem? Also, would you have other solutions to propose?