filedescriptor leaks in subprocesses?

[radoering]

We noticed that some tests of Python's test suite related to file descriptors in subprocesses started failing with python-build-standalone 3.12.8.

The following tests pass with 3.12.7 but fail with 3.12.8 (and still fail with 3.12.10) - tested on Ubuntu 20.04 and 24.04:

python -m test -v -m test_pass_fds_redirected test_subprocess.py
python -m test -v -m test_no_* test_cmd_line.py
python -m test -v -m test_close_file test_posix.py

(The tests succeed with a deadsnakes Python 3.12.10.)

All tests have in common that they start a subprocess and check that some file descriptors, which are open in the main process, are closed in the subprocess - and fail because they are not.

Does anyone have an idea why this could be an issue in python-build-standalone now, why it was not an issue before and if/how it could be fixed?

[zanieb]

Thanks for the report! Unfortunately, I have no clue why this would be off the top of my head.

[radoering]

I narrowed it down a bit further:

The issue has been introduced in 20250106. (The tests pass with 3.12.8 from 20241219.)

Further, the issue exists only for 3.12 and 3.13. It does not exist for 3.11.

[radoering]

I narrowed it down even further:

The issue has been introduced in 0be0a57 and in particular by applying the BOLT flag -hugify. By removing this part of the patch, I was able to built a Python that passes the mentioned tests.

@zanieb If you like, I can create a PR removing this part of the patch. If you prefer to carry out further investigations before, that is also fine with me. (We are building on our own with some modifications anyway.)

[zanieb]

Interesting, thanks for digging into it! I really appreciate it.

I'm happy to revert that change, if you want to put up a pull request. We'll want to coordinate with the BOLT and CPython teams afterwards to figure out why there's a regression, I can follow up on that.

[geofft]

FWIW I tracked this down in #641. The -hugify startup code leaks a file descriptor, and the tests are sensitive to that. IMO this is a bug in both the tests and a bug in BOLT, but the good news is that it's not a bug / miscompile in CPython itself.