spawn asyncio subprocesses in a thread pool

[asvetlov]

BPO	37263
Nosy	@asvetlov, @1st1

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2019-06-13.07:58:27.658>
labels = ['expert-asyncio', 'performance']
title = 'spawn asyncio subprocesses in a thread pool'
updated_at = <Date 2020-09-21.19:37:55.666>
user = 'https://github.com/asvetlov'

bugs.python.org fields:

activity = <Date 2020-09-21.19:37:55.666>
actor = 'iritkatriel'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['asyncio']
creation = <Date 2019-06-13.07:58:27.658>
creator = 'asvetlov'
dependencies = []
files = []
hgrepos = []
issue_num = 37263
keywords = []
message_count = 1.0
messages = ['345472']
nosy_count = 2.0
nosy_names = ['asvetlov', 'yselivanov']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'performance'
url = 'https://bugs.python.org/issue37263'
versions = []

[asvetlov]

Subprocess starting can be a long blocking operation, see python-trio/trio#1109 for discussion

[kumaraditya303]

From the Trio issue, this seems mostly a Windows issue where process creation is indeed very slow. So maybe this should be only done for Windows.

[gvanrossum]

I'm just realizing that when we use the ThreadedChildWatcher, there's a thread created for the process anyway, so maybe we could move the subprocess operation into that thread... (Then again on Windows we don't use child watchers.)

[willingc]

@kumaraditya303 Flagging this as an FYI as you look at the work for #120804. After that work is completed, we can revisit.