Add option to cargo vendor crates into sdist

[tiran]

As of today, setuptools_rust's based extensions require network access during compile stage. The build stage uses cargo to fetch crates from crates.io. For some use cases, it would be beneficial to have an option or command that creates a self-containing source distribution with vendored creates. I define self-contained sdist as ball or zip file that contains all required Rust crate sources ( cargo vendor) and a .cargo/config.toml additionally to a standard python setup.py sdist source ball.

Possible consumers of a self-contained sdist with vendored sources are:

• end users who need to perform offline installations from sources on isolated / firewalled systems.
• users that run a DevPI cache with their PyPI packages and don't want Python to contact the network during installation
• distro packagers who require reproducible builds without external resources

Vendoring shouldn't be the default. I guess you could either add an option to sdist or define a custom sdistvendored command. There is an ongoing discussion at https://discuss.python.org/t/setuptools-rust-maturin-fetching-rust-deps-when-building-wheel/7095 .

implementation

• $ cargo vendor --manifest-path path/to/extension/Cargo.toml
• create .cargo/config.toml
• include vendor and .cargo directory in sdist source ball

[source.crates-io]
replace-with = "vendored-sources"

[source.vendored-sources]
directory = "vendor"

[davidhewitt]

I agree this would be useful. I think having an option to sdist is likely better as it would combine better with other setuptools extensions.

[messense]

It might be impractical due to the size of the vendored source. For example with https://github.com/PyO3/setuptools-rust/tree/master/examples/namespace_package , cargo vendor produces a very large vendor directory.

du -sh vendor
121M	vendor

[tiran]

It's a bit unfortunate. parking_lot_core's thread_parker pulls is almost 119 MB worth of winapi crates. The crates compress very well. 6 MB extra is not ideal but ok. PyPI hosts binary wheels with > 1 GB of GPU glue code.

$ du --si -s vendor/
129M    vendor/
$ du --si -s vendor/winapi*
8,0M    vendor/winapi
54M     vendor/winapi-i686-pc-windows-gnu
57M     vendor/winapi-x86_64-pc-windows-gnu

$ tar -cjf vendor.tar.bz2 vendor/
$ du --si vendor.tar.bz2 
6,0M    vendor.tar.bz2

[messense]

I've got a PoC working in #118

[rtsscy]

Hi there! Somewhat a related question perhaps, but is there currently a way to replace source by providing a registry source instead directory source? I currently have a .cargo/config.tol file with my mirrored registry but that doesn’t seem to get picked up by setuptools-rust