distutils.spawn should use subprocess (hang in parallel builds on QNX)

[EladLahav]

BPO	39763
Nosy	@jaraco, @pitrou, @vstinner, @merwok, @dstufft, @aixtools
PRs	bpo-39763: distutils.spawn now uses subprocess #18743 bpo-39763: Refactor setup.py #18778 bpo-39763: Change test to check for subprocess.check_output during setup.py build phase #18847 bpo-39763: Add _bootsubprocess to build Python on AIX #18872
Files	fork_mt.py

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2020-02-26.19:18:35.569>
labels = ['3.8', 'library']
title = 'distutils.spawn should use subprocess (hang in parallel builds on QNX)'
updated_at = <Date 2020-07-04.01:26:37.758>
user = 'https://bugs.python.org/EladLahav'

bugs.python.org fields:

activity = <Date 2020-07-04.01:26:37.758>
actor = 'jaraco'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2020-02-26.19:18:35.569>
creator = 'Elad Lahav'
dependencies = []
files = ['48917']
hgrepos = []
issue_num = 39763
keywords = ['patch']
message_count = 36.0
messages = ['362717', '362728', '362731', '362732', '362743', '362752', '362794', '362818', '362822', '362824', '362826', '362847', '363017', '363038', '363058', '363165', '363172', '363173', '363176', '363349', '363363', '363646', '363711', '363749', '363754', '363780', '363782', '363784', '363840', '363841', '363853', '363855', '363943', '363945', '363947', '372974']
nosy_count = 7.0
nosy_names = ['jaraco', 'pitrou', 'vstinner', 'eric.araujo', 'dstufft', 'Michael.Felt', 'Elad Lahav']
pr_nums = ['18743', '18778', '18847', '18872']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = None
url = 'https://bugs.python.org/issue39763'
versions = ['Python 3.8']

[EladLahav]

The attached code causes the child processes to hang on QNX. The hang is caused by the logging module trying to acquire the module lock while in an atfork() handler. In a system where semaphore state is kept in user mode and is thus inherited from the parent on fork() the semaphore may appear to have a value of 0, and thus will never be posted to in the single-threaded child.
I don't know how it works on other systems - may be pure chance.

[pitrou]

Note the original problem stems when you using "setup.py -j <NJOBS>".

The attached reproducer is a bit of a red herring, because indeed doing fork() + exec() in pure Python is too fragile. That is why the standard subprocess does it in C. The problem here is that the distutils.spawn module still has its hand-baked fork() + exec() pure Python code from the 1990s.

So the solution to the OP's actual problem should be to recode distutils.spawn using the subprocess module.

Original mailing-list discussion at:
https://mail.python.org/archives/list/python-dev@python.org/thread/PTJ3UAZNYBYBYQ4WLMSERZR2XIVWISLM/

[EladLahav]

I'm not convinced that a multi-threaded fork()+exec() from C would be any better, unless the Python code goes to great lengths to avoid any non-async-signal-safe operations between the fork() and the exec().
So along with the proposed change to switch to the subprocess() module I would like to propose that that module switch to using posix_spawn() instead of fork()+exec().