ProcessPoolExecutor hangs when 1<max_tasks_per_child<num_submitted//max_workers #115634
Comments
dalleyg
changed the title
|
Can't repro this on WSL2 + 3.12/3.11/main. Result: |
|
Can't reproduce on Linux + 3.12.1/3.11.7/main. I get the same |
|
Presumably getting Here are the specific Python version I tested on WSL2 + Ubuntu. It was installed with |
|
@dalleyg The |
|
I just downloaded the 3.11.8 tarball from python.org, ran I get the same hanging result on WSL2 + Ubuntu with:
On the other hand, a colleague is seeing the
This makes me wonder if the hanging bug is WSL2 and/or Ubuntu-specific, and the Do you recommend that I fork this issue into two?
|
|
If I use Docker, I get the |
|
I failed to repro this on WSL2 + Ubuntu 20.04 so I don't believe WSL is the sufficient trigger. |
|
Since others can't reproduce my hanging problem, I'll close this issue for now and open a new one for the Thanks for the help! |
|
Okay, I have a reproducer now, so I'm reopening this. When running the code as a script, echo '
import os
from concurrent.futures import ProcessPoolExecutor
with ProcessPoolExecutor(1, max_tasks_per_child=2) as exe:
futs = [exe.submit(os.getpid) for _ in range(10)]
for fut in futs:
print(fut.result())
' > broken.py
(
echo "TESTING WITH A SCRIPT"
for v in 3.10 3.11.8 3.12.2 3.13.0a4; do
DOCKER="docker run -it --rm --name 115634 -q -v "$PWD":/usr/src/myapp -w /usr/src/myapp python:$v"
echo "--------------------------------------"
(
$DOCKER python3 --version &&
$DOCKER timeout --signal=SIGINT --kill-after 5s -v 2s python3 broken.py
) 2>&1
done
) | tee broken-script.log
(
echo "TESTING WITH THE SCRIPT INLINED INTO THE COMMAND"
for v in 3.10 3.11.8 3.12.2 3.13.0a4; do
DOCKER="docker run -it --rm --name 115634 -q -v "$PWD":/usr/src/myapp -w /usr/src/myapp python:$v"
echo "--------------------------------------"
(
$DOCKER python3 --version &&
$DOCKER timeout --signal=SIGINT --kill-after 5s -v 2s python3 -c "$(cat broken.py)"
) 2>&1
done
) | tee broken-inline.logHere's a snipped version of And here's what I see in |
|
I'm able to repro with import os
from concurrent.futures import ProcessPoolExecutor
if __name__ == '__main__':
with ProcessPoolExecutor(1, max_tasks_per_child=2) as exe:
futs = [exe.submit(os.getpid) for _ in range(10)]
for fut in futs:
print(fut.result())or even simpler: from concurrent.futures import ProcessPoolExecutor
if __name__ == '__main__':
with ProcessPoolExecutor(1, max_tasks_per_child=2) as exe:
futs = [exe.submit(print, i) for i in range(10)] |
|
Confirmed. With the |
|
There's something fundamentally wrong here for the process numbers, I'll work on it. |
|
See #115642 for the detailed explanation and the potential fix. |
|
That was quick! I'll watch that PR for a resolution. In the meantime, I've put a safety check in my code. I'll adjust it once there's a final resolution. |
This is because it does not work see python/cpython#115634
|
Would it be possible to add a warning not to use |
Uh oh!
There was an error while loading. Please reload this page.
Bug report
Bug description:
Starting in Python 3.11 when the
max_tasks_per_childparameter was introduced,ProcessPoolExecutorhangs whenmax_tasks_per_child>1and enough tasks have been submitted to trigger a worker restart.The following reproducer hangs on a fresh installation of Python 3.11 on Linux.
Issuing a keyboard interrupt results in the following stack trace:
Notes:
result()calls does not help.max_tasks_per_child=1works great. It never hangs, and a new process correctly used for each task.max_tasks_per_childis set high enough so that no worker restarts happen.CPython versions tested on:
3.11, 3.12
Operating systems tested on:
Linux
Linked PRs
The text was updated successfully, but these errors were encountered: