Feature request: maxtasksperchild for ProcessPoolExecutor

[cool-RR]

BPO	44733
Nosy	@gpshead, @pitrou, @cool-RR, @loganasherjones
PRs	bpo-44733: Add maxtasksperchild to ProcessPoolExecutor #27373 bpo-44733: max_tasks_per_child + fork not allowed #32187

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/gpshead'
closed_at = None
created_at = <Date 2021-07-24.12:14:41.532>
labels = ['type-feature', 'library', '3.11']
title = 'Feature request: maxtasksperchild for ProcessPoolExecutor'
updated_at = <Date 2022-03-31.00:07:58.887>
user = 'https://github.com/cool-RR'

bugs.python.org fields:

activity = <Date 2022-03-31.00:07:58.887>
actor = 'loganasherjones'
assignee = 'gregory.p.smith'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2021-07-24.12:14:41.532>
creator = 'cool-RR'
dependencies = []
files = []
hgrepos = []
issue_num = 44733
keywords = ['patch']
message_count = 13.0
messages = ['398143', '398240', '398243', '406689', '411449', '411450', '411452', '411471', '411546', '411547', '416310', '416314', '416406']
nosy_count = 4.0
nosy_names = ['gregory.p.smith', 'pitrou', 'cool-RR', 'loganasherjones']
pr_nums = ['27373', '32187']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue44733'
versions = ['Python 3.11']

[cool-RR]

I love concurrent.futures, and I'd like to use it wherever I can. There's a feature in multiprocessing.Pool that I wish would also be available in ProcessPoolExecutor: The maxtasksperchild argument.

Documentation: "maxtasksperchild is the number of tasks a worker process can complete before it will exit and be replaced with a fresh worker process, to enable unused resources to be freed. The default maxtasksperchild is None, which means worker processes will live as long as the pool."

I want to be able to set it to 1, so each process will only execute one task and then be replaced with a fresh process.

[loganasherjones]

I think I have a solution for this, but I'm pretty new to contributing. Still writing up some tests.

[cool-RR]

Awesome. You can link to experimental code here. Even if it were not accepted to Python, it could be useful for me and for other people who might see this issue.

[pitrou]

New changeset fdc0e09 by Logan Jones in branch 'main':
bpo-44733: Add max_tasks_per_child to ProcessPoolExecutor (GH-27373)
fdc0e09

[gpshead]

bpo-46464 describes a deadlock that the pre-requisite for this feature causes.

Spawning new children directly via fork() is impossible once a thread in the parent process has been started and concurrent.futures.process starts a thread.

[gpshead]

If we want this feature to stay in, it needs an implementation that does not rely on a multithreaded processing calling fork(). I'm going to have to revert the existing implementation for bpo-46464.

[cool-RR]

Oh that sucks. Logan and Antoine worked on this feature for so long. Thanks for reporting Greg.

[loganasherjones]

Based on my reading I’m hopeful that this change can make it in quickly once I find the time. The previous implementation didn’t care how the processes were created.

I will look as soon as I can

[gpshead]

from discussion on the other bug it looks like we should have a way to keep this; we just need to not allow it when the mp_context to be used is the 'fork' one.

[gpshead]

As this is a new feature, it would also be reasonable to have specifying max_tasks_per_child without explicitly specifying a mp_context default to a safe mp_context.

[loganasherjones]

Okay, I'm actually able to work on this again. What is the best way to make this change real. Should I be working off of main?

[gpshead]

yep, branch off of a recent main.

[loganasherjones]

Ok I now have a PR up with the features requested. Let me know if you need anything else!

[gpshead]

your PR was #32187 and an equivalent ultimately landed in fa4f0a1 if i'm untangling things correctly. So this can be closed again. max_tasks_per_child made it into 3.11beta1.