-
-
Notifications
You must be signed in to change notification settings - Fork 32k
ProcessPoolExecutor hangs when 1<max_tasks_per_child<num_submitted//max_workers #115634
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Comments
Can't repro this on WSL2 + 3.12/3.11/main. Result:
|
Can't reproduce on Linux + 3.12.1/3.11.7/main. I get the same |
Presumably getting Here are the specific Python version I tested on WSL2 + Ubuntu. It was installed with
|
@dalleyg The |
I just downloaded the 3.11.8 tarball from python.org, ran
I get the same hanging result on WSL2 + Ubuntu with:
On the other hand, a colleague is seeing the
This makes me wonder if the hanging bug is WSL2 and/or Ubuntu-specific, and the Do you recommend that I fork this issue into two?
|
If I use Docker, I get the
|
I failed to repro this on WSL2 + Ubuntu 20.04 so I don't believe WSL is the sufficient trigger. |
Since others can't reproduce my hanging problem, I'll close this issue for now and open a new one for the Thanks for the help! |
Okay, I have a reproducer now, so I'm reopening this. When running the code as a script, echo '
import os
from concurrent.futures import ProcessPoolExecutor
with ProcessPoolExecutor(1, max_tasks_per_child=2) as exe:
futs = [exe.submit(os.getpid) for _ in range(10)]
for fut in futs:
print(fut.result())
' > broken.py
(
echo "TESTING WITH A SCRIPT"
for v in 3.10 3.11.8 3.12.2 3.13.0a4; do
DOCKER="docker run -it --rm --name 115634 -q -v "$PWD":/usr/src/myapp -w /usr/src/myapp python:$v"
echo "--------------------------------------"
(
$DOCKER python3 --version &&
$DOCKER timeout --signal=SIGINT --kill-after 5s -v 2s python3 broken.py
) 2>&1
done
) | tee broken-script.log
(
echo "TESTING WITH THE SCRIPT INLINED INTO THE COMMAND"
for v in 3.10 3.11.8 3.12.2 3.13.0a4; do
DOCKER="docker run -it --rm --name 115634 -q -v "$PWD":/usr/src/myapp -w /usr/src/myapp python:$v"
echo "--------------------------------------"
(
$DOCKER python3 --version &&
$DOCKER timeout --signal=SIGINT --kill-after 5s -v 2s python3 -c "$(cat broken.py)"
) 2>&1
done
) | tee broken-inline.log Here's a snipped version of
And here's what I see in
|
I'm able to repro with import os
from concurrent.futures import ProcessPoolExecutor
if __name__ == '__main__':
with ProcessPoolExecutor(1, max_tasks_per_child=2) as exe:
futs = [exe.submit(os.getpid) for _ in range(10)]
for fut in futs:
print(fut.result()) or even simpler: from concurrent.futures import ProcessPoolExecutor
if __name__ == '__main__':
with ProcessPoolExecutor(1, max_tasks_per_child=2) as exe:
futs = [exe.submit(print, i) for i in range(10)] |
Confirmed. With the |
There's something fundamentally wrong here for the process numbers, I'll work on it. |
See #115642 for the detailed explanation and the potential fix. |
That was quick! I'll watch that PR for a resolution. In the meantime, I've put a safety check in my code. I'll adjust it once there's a final resolution. |
This is because it does not work see python/cpython#115634
Would it be possible to add a warning not to use |
Uh oh!
There was an error while loading. Please reload this page.
Bug report
Bug description:
Starting in Python 3.11 when the
max_tasks_per_child
parameter was introduced,ProcessPoolExecutor
hangs whenmax_tasks_per_child>1
and enough tasks have been submitted to trigger a worker restart.The following reproducer hangs on a fresh installation of Python 3.11 on Linux.
Issuing a keyboard interrupt results in the following stack trace:
Notes:
result()
calls does not help.max_tasks_per_child=1
works great. It never hangs, and a new process correctly used for each task.max_tasks_per_child
is set high enough so that no worker restarts happen.CPython versions tested on:
3.11, 3.12
Operating systems tested on:
Linux
Linked PRs
The text was updated successfully, but these errors were encountered: