gh-90622: Prevent concurrent.futures.process deadlock due to fork() after spawning a thread. #30847
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…thonGH-27373)" This reverts commit fdc0e09. This implementation relies on a mechanism for spawning new children dynamically rather than up front that leads to deadlocks due to mixing of threads+fork. See bpo-46464.
bpo-39207: Spawn workers on demand in ProcessPoolExecutor (pythonGH-19453) was implemented in a way that introduced child process deadlocks in some environments due to mixing threading and fork() in the parent process.
gpshead
added
type-bug
|
See bug discussion. It looks like this will split into 2 and the max_tasks_per_child feature can be kept so long as it becomes an error when a 'fork' mp_context would be used. |
gpshead
changed the title
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
1ac6e37 released in 3.9 as a performance enhancement to
concurrent.futuresintroduced a deadlock in some environments by violating the POSIX spec with new child processes being fork()ed after a thread had been started. This reverts that. It also reverts fdc0e09 which tried to add a newmax_tasks_per_child featurefor 3.11 that built upon the previous ill fated 3.9 performance enhancement.There is a better way to do this that would not fork after starting a thread, but implementing that is a much larger architectural enhancement; not something that could be back-ported as a bug fix.
The 3.10 backport will need to be manual as it doesn't have the 3.11 change. The 3.9 one can be autogenerated from that. Alternatively I could split this PR into two separate rollbacks.
https://bugs.python.org/issue46464