site.py is no longer read at startup time (as it is a deep-frozen module)

[pelson]

Bug report

Before https://github.com/python/cpython/pull/29118/files#diff-1f0a8db227d22005511b0d90f5339b97db345917b863954b3b3ccb9ec308767cR873 it was possible to modify site.py and control the behaviour of Python at a site level. Indeed, the comments in site.py reflect this (e.g. set it to False to disable the feature or True to force the feature) (also in the docs https://docs.python.org/3/library/site.html#site.ENABLE_USER_SITE, esp "None means it was disabled for security reasons ... or by an administrator.").

The change was introduced in order to optimise the startup time. It would be retrograde to stop doing that, but at the same time, having .py files in the shipped distribution (stdlib) which are not used is confusing and wasteful. At a minimum we could no longer ship frozen .py files (since they aren't used), thereby saving disk space and avoiding confusion for people who might edit those files.

If the startup time optimisation is a result of avoiding stat-ing files, then it would be great if there was a convenient way to re-compile the site.py frozen module (so that we can still customize the site). If it isn't such a concern to stat, the alternative would be to look to see if there is a site.py file, and if so, use that instead of the frozen one (assuming that the site.py is removed from the distribution as suggested above).

Above all, the desire is to be able to configure the site of a distribution after compilation, without using tricks such as sitecustomize, activations scripts, or .pth files.

Your environment

I have reproduced this on Python 3.11.4 and main (6e850c3), on Linux.

Relates to #89824.

[njsmith]

Can you elaborate on why sitecustomize doesn't work for you? Is the issue that it's missing features, or something else?

[pelson]

Can you elaborate on why sitecustomize doesn't work for you? Is the issue that it's missing features, or something else?

Mostly sitecustomize can work, but a few reasons that it is more challenging/impossible:

• It is too late to disable user site-packages at sitecustomize (your issue It should be possible for a Python environment to disable the "user site" functionality without patching the stdlib #99312)
• There is already third-party package code which has been executed by the time sitecustomize is run (via .pth), and I need to influence how all code is executed (e.g. setting default environment variables before anything else)
• I want to be able to add additional site-packages directories, with higher precedence than user site-packages (an prior processing in e.g. .pth files should follow that precedence)
• It would be nice if the consumers of my built Python distribution (which they can install themselves) could configure the distribution using the standard sitecustomize mechanism, but this isn't possible if I use sitecustomize for controlling the distribution's behaviour itself. Essentially, I am a Python distributor (i.e. I provide an installer), not a Python distribution provider, and at the same time, I am re-using an existing build, not building from source (so can't patch site).

To be clear, my biggest concern is that site.py is shipped (in the stdlib), but is entirely ignored. I would find it more appropriate to either not ship site.py at all, or to provide a mechanism that allows overriding a frozen module (though I acknowledge that this implies more failed file stats in most cases, which might invalidate some of the benefit of the change in the first place?)

[morotti]

I unfortunately come across this issue. massive outage when some service accounts got a ~/.local/lib/python3.11/site-packages/... populating by something.

the site.py was edited to set site.ENABLE_USER_SITE = False to prevent exactly this issue from happening.
but that stopped working as per this bug ticket :(

I went through the bug tracker and I haven't seen any mention of a fix or any workaround to address the issue.
The alternative I could come up is to add this snippet to the top of your sitecustomize.py, for lack of a better solution.

However it is not equivalent to the previous solution of editing site.py, because the sitecustomize.py is loaded after the import paths have been configured to load from the user site.
the sitecustomize.py cannot prevent from loading packages from the user sites, it's called too late.

import site
import sys


# ~/.local/lib/python3.8/site-packages
# disable user site otherwise it takes precedence over packages in the venv.
# it was meant to be disabled by editing the site.py file, but that stopped working.
# https://github.com/python/cpython/issues/107344

site.ENABLE_USER_SITE = False
user_site = site.getusersitepackages()
if user_site in sys.path:
    sys.path.remove(user_site)

You can run the interpreter with python -v and see that imports are still contaminated by the .local directory.

Processing user site-packages
Adding directory: '/root/.local/lib/python3.11/site-packages'
Processing global site-packages
...
# trying /root/.local/lib/python3.11/site-packages/_distutils_hack.cpython-311-x86_64-linux-gnu.so
# trying /root/.local/lib/python3.11/site-packages/_distutils_hack.abi3.so
# trying /root/.local/lib/python3.11/site-packages/_distutils_hack.so
# trying /root/.local/lib/python3.11/site-packages/_distutils_hack.py
# trying /root/.local/lib/python3.11/site-packages/_distutils_hack.pyc
...
# trying /root/.local/lib/python3.11/site-packages/sitecustomize.cpython-311-x86_64-linux-gnu.so
# trying /root/.local/lib/python3.11/site-packages/sitecustomize.abi3.so
# trying /root/.local/lib/python3.11/site-packages/sitecustomize.so
# trying /root/.local/lib/python3.11/site-packages/sitecustomize.py
# trying /root/.local/lib/python3.11/site-packages/sitecustomize.pyc

[pelson]

The original change came from #28398. Perhaps @ericsnowcurrently could help us determine if there is something we can do to conditionally run site.py (not the frozen version) during startup only if it is changed?