Please make the C extension build optional (e.g. via `PURE_PYTHON`)

[mgorny]

FEATURE REQUEST

Could you consider making the C extension build in setup.py optional, similarly to how the extensions are currently optional in tests? (i.e. if PURE_PYTHON=1, it wouldn't attempt to build them at all)

Our primary use case right now is Python 3.13 freethreading. Since the speedups extension does not support freethreading, importing it effectively reenables GIL for the whole program:

>>> import zope.interface
<frozen importlib._bootstrap>:488: RuntimeWarning: The global interpreter lock (GIL) has been enabled to load module 'zope.interface._zope_interface_coptimizations', which has not declared that it can run safely without the GIL. To override this behavior and keep the GIL disabled (at your own risk), run with PYTHON_GIL=0 or -Xgil=0.

For the time being, we believe that the "lesser evil" would be to install install zope.interface without speedups, and keep GIL disabled for the remainder of the program. The way we see it, it would be done like:

$ PURE_PYTHON=1 pip install zope.interface

[d-maurer]

Michał Górny wrote at 2024-10-18 22:55 -0700:

## FEATURE REQUEST Could you consider making the C extension build in `setup.py` optional, similarly to how the extensions are currently optional in tests? (i.e. if `PURE_PYTHON=1`, it wouldn't attempt to build them at all) ... For the time being, we believe that the "lesser evil" would be to install install `zope.interface` without speedups, and keep GIL disabled for the remainder of the program. The way we see it, it would be done like: ``` $ PURE_PYTHON=1 pip install zope.interface ```

Why are you interested to suppress the installation of the C extension? Setting the envvar `PURE_PYTHON` to a value different from `"0"` at run (in contrast to installation) time, should prevent the C extension import. Have you observed that this does not work? If it really should not work, you can delete an installed C extension. Binary distributions of `zope.interface` contain the C extension. I fear that (new) special logic would be necessary to suppress its installation based on the `PURE_PYTHON` envvar. I am against the introduction of such special logic for your quite special use case.

[mgorny]

I'm packaging zope.interface for Gentoo. Setting PURE_PYTHON at runtime does not resolve the more general problem that the package currently installs a harmful extension. Similarly, removing it would mean that users unnecessarily burn coal to build it. And the current design would mean we'd have to maintain an intrusive patch forever, vs. having to just control a single envvar.

[d-maurer]

Michał Górny wrote at 2024-10-19 00:23 -0700:

I'm packaging `zope.interface` for Gentoo. Setting `PURE_PYTHON` at runtime does not resolve the more general problem that the package currently installs a harmful extension. Similarly, removing it would mean that users unnecessarily burn coal to build it. And the current design would mean we'd have to maintain an intrusive patch forever, vs. having to just control a single envvar.

There are distribution packagers who would like to omit the installation of tests. Apparently, they would prefer when the tests would not be included in the distribution at all but can remove them without too much problems during their own packaging. Maybe, this would work as well for your `Gentoo` packaging regarding the C optimazation.

[mgorny]

Deleting .py files that are merely copied is a minor cost compared to C extensions that are actually compiled, which involve nonzero energy cost.

[d-maurer]

Michał Górny wrote at 2024-10-19 01:38 -0700:

Deleting `.py` files that are merely copied is a minor cost compared to C extensions that are actually compiled, which involve nonzero energy cost.

Nevertheless, I would delegate the decision on whether to use the C optimization to runtime. I would follow logic (in `zope.interface._compat._should_attempt_c_optimizations`) used for `PyPy`. It could check whether it runs without GIL and then not use the c extension. Your `Gentoo` packaging would build the C extension and it would be used when the GIL is used; without GIL, it would not be used.

[mgorny]

Hmm, I suppose that suggestion does make sense. Do you want me to make a patch?

[d-maurer]

Michał Górny wrote at 2024-10-19 01:55 -0700:

Hmm, I suppose that suggestion does make sense. Do you want me to make a patch?

That would be nice! To easily take over your patch, you would need to sign the `Zope` contributor agreement.

[tseaver]

Deleting .py files that are merely copied is a minor cost compared to C extensions that are actually compiled, which involve nonzero energy cost.

We publish wheels with the extension already built, which covers 99.99% of all users of the package. Doing the extra work (if needed) when building for a Linux distribution isn't going to be measurable, compared to the efficiency of actually using the extension.

WRT to the "declaration" that we support free-threading: while there is some work to be done to test / verfiy / declare that the package works under freethreading, we should be pretty close with the refactorings done to support heap-based types and multi-phase initialization.

[mgorny]

We publish wheels with the extension already built, which covers 99.99% of all users of the package.

Do you have any honest, unbiased statistics to prove that number? Because 99.99% of the times when people say "99.99%", they're just making that up.

[tseaver]

Available download stats for zope.interface here.

• Downloads for 3.13 and 3.14 (no way to tell FT vs. non-FT) seem to average about 1% of total downloads since the 3.13 release.
• Those stats don't (any longer?) split out sdist vs. bdist_wheel or other bdist_* formats.

There are essentially three ways folks get zope.interface:

• Via pip or a derivative: unless the user explicitly prevents it, prefers a binary;
  in many cases, as specified in a requirements.txt file (or similar with tools built atop pip),
  or via a transient dependency.
• Via zc.buildout (might still prefer source, but I haven't touched it in a while)
• Via Linux distro packages (already binary)

You'd be hard-pressed to convince me that there are a significant number of pip (and derived tools) users who specify --no-binary for their packages: that makes their builds slower and is wasteful to boot: even if they did, after the first download already have the wheel in their cache if they are doing active development.

[tseaver]

See #332 for work needed to support 3.13 FT.