asyncio: support fork

[vstinner]

BPO	21998
Nosy	@pitrou, @vstinner, @tiran, @1st1, @Martiusweb
Files	test2.py close_self_pipe_after_selector.patch at_fork.patch at_fork-2.patch at_fork-3.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/1st1'
closed_at = <Date 2020-02-10.23:51:31.002>
created_at = <Date 2014-07-17.16:10:45.820>
labels = ['expert-asyncio']
title = 'asyncio: support fork'
updated_at = <Date 2020-02-10.23:51:31.001>
user = 'https://github.com/vstinner'

bugs.python.org fields:

activity = <Date 2020-02-10.23:51:31.001>
actor = 'vstinner'
assignee = 'yselivanov'
closed = True
closed_date = <Date 2020-02-10.23:51:31.002>
closer = 'vstinner'
components = ['asyncio']
creation = <Date 2014-07-17.16:10:45.820>
creator = 'vstinner'
dependencies = []
files = ['37306', '37385', '38016', '38019', '38164']
hgrepos = []
issue_num = 21998
keywords = ['patch']
message_count = 56.0
messages = ['223344', '223431', '225973', '226861', '226889', '226892', '226899', '226932', '231819', '231845', '231919', '231921', '231941', '232301', '232308', '235409', '235410', '235429', '236065', '236131', '236134', '236136', '236145', '236149', '236150', '236151', '244111', '244112', '244113', '244114', '244116', '244117', '244118', '244119', '244120', '244121', '244123', '244130', '244135', '249722', '249724', '249732', '256889', '297117', '297207', '297218', '297219', '297228', '297230', '297232', '297233', '297235', '313196', '325820', '325824', '361755']
nosy_count = 9.0
nosy_names = ['pitrou', 'vstinner', 'christian.heimes', 'zmedico', 'neologix', 'yselivanov', 'Adam.Bishop', 'martius', 'Christian H']
pr_nums = []
priority = 'normal'
resolution = 'out of date'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue21998'
versions = ['Python 3.6']

[vstinner]

It looks like asyncio does not suppport fork() currently: on fork(), the event loop of the parent and the child process share the same "self pipe".

Example:
---

import asyncio, os
loop = asyncio.get_event_loop()
pid = os.fork()
if pid:
    print("parent", loop._csock.fileno(), loop._ssock.fileno())
else:
    print("child", loop._csock.fileno(), loop._ssock.fileno())

---

Output:
---
parent 6 5
child 6 5
---

I'm not sure that it's revelant to use asyncio with fork, but I wanted to open an issue at least as a reminder.

I found the "self pipe" + fork issue when reading the issue bpo-12060, while working on a fix for the signal handling race condition on FreeBSD: issue bpo-21645.

[vstinner]

Oh, I wanted to use the atfork module for that, but then I saw that it does not exist :-(

[vstinner]

A first step would be to document the issue in the "developer" section of asyncio documentation. Mention that the event loop should be closed and a new event loop should be created.

[1st1]

It's not that it doesn't work after fork, right? Should we add a recipe with pid monitoring a self-pipe re-initialization?

[gvanrossum]

Actually I expect that if you share an event loop across different processes via form, everything's a mess -- whenever a FD becomes ready, won't both the parent and the child be woken up? Then both would attempt to read from it. One would probably get EWOULDBLOCK (assuming all FDs are actually in non-blocking mode) but it would still be a mess. The specific mess for the self-pipe would be that the race condition it's intended to solve might come back.

It's possible that some polling syscall might have some kind of protection against forking, but the Python data structures that map FDs to handlers don't know that, so it would still be a mess.

Pretty much the only thing you should expect to be able to do safely after forking is closing the event loop -- and I'm not even 100% sure that that's safe (I don't know what happens to a forked executor).

Is there a use case for sharing an event loop across forking?

[1st1]

Is there a use case for sharing an event loop across forking?

I don't think so. I use forking mainly for the following two use-cases:

• Socket sharing for web servers. Solution: if you want to have a shared sockets between multiple child processes, just open them in master process, fork as many times as you want, and start event
  loops in child processes only.

• If sometimes you want to spawn some child process "on demand". Solution: fork before having a loop running, and use the child process as a "template", i.e. when you need a new child process - just ask the first child to fork it.

It would certainly be handy to have an ability to fork while the loop is running and safely close (and reopen) it in the forked child process. But now I can see that it's a non-trivial thing to do properly. Probably it's ~somewhat~ safe to re-initialize epoll (or whatever selector we use), re-open self-pipe etc, drop all queued callbacks and clear Task.all_tasks collection, but even then it's easy to miss something.

I think we just need to make sure that we have documented that asyncio loops are not fork safe, and forks in running loops should be avoided by all means.

[vstinner]

Is there a use case for sharing an event loop across forking?

I don't know if asyncio must have a builtin support for this use case. The minimum is to document the behaviour, or maybe even suggest a recipe to support it.

For example, an event loop of asyncio is not thread-safe and we don't want to support this use case. But I wrote a short documentation with code snippets to show how to workaround this issue:
https://docs.python.org/dev/library/asyncio-dev.html#concurrency-and-multithreading

We need a similar section to explain how to use asyncio with the os.fork() function and the multiprocesing module.

[gvanrossum]

That sounds about right -- it's a doc issue. Let me propose a paragraph:

"""
NOTE: It is not safe to share an asyncio event loop between processes that are related by os.fork(). If an event loop exists in a process, and that process is forked, the only safe operation on the loop in the child process is to call its close() method.
"""

(I don't want to have to research what the various polling primitives do on fork(), so I don't want to suggest that it's okay to close the loop in the parent and use it in the child.)

A similar note should probably be added to the docs for the selectors module.

[Martiusweb]

Hi,

Actually, closing and creating a new loop in the child doesn't work either, at least on Linux.

When, in the child, we call loop.close(), it performs:
self.remove_reader(self._ssock)
(in selector_events.py, _close_self_pipe() around line 85)

Both the parent and the child still refer to the same underlying epoll structure, at that moment, and calling remove_reader() has an effect on the parent process too (which will never watch the self-pipe again).

I attached a test case that demonstrates the issue (and the workaround, commented).

[gvanrossum]

Martin, what is the magic call to make in the child (or in the parent pre-fork???) to disable the epoll object in the child without disabling it in the parent?

(Perhaps just closing the selector and letting the unregister calls fail would work?)

[Martiusweb]

Guido,

Currently in my program, I manually remove and then re-adds the reader to the loop in the parent process right after the fork(). I also considered a dirty monkey-patching of remove_reader() and remove_writer() which would act as the original versions but without removing the fds from the epoll object (ensuring I don't get bitten by the same behavior for an other fd).

The easiest fix, I think, is indeed to close the selector without unregistering the fds, but I don't know if doing so would have undesired side effects on an other platform than Linux (resources leak, or the close call failing maybe).

[Martiusweb]

I said something wrong in my previous comment: removing and re-adding the reader callback right after the fork() is obviously subject to a race condition.

I'll go for the monkey patching.

[gvanrossum]

How about not doing anything in the parent but in the child, closing the
selector and then the event loop?

On Mon, Dec 1, 2014 at 1:29 AM, Martin Richard <report@bugs.python.org>
wrote:

Martin Richard added the comment:

I said something wrong in my previous comment: removing and re-adding the
reader callback right after the fork() is obviously subject to a race
condition.

I'll go for the monkey patching.

----------

---

Python tracker <report@bugs.python.org>
<http://bugs.python.org/issue21998\>

---

[Martiusweb]

Currently, this is what I do in the child after the fork:

>> selector = loop._selector
>> parent_class = selector.__class__.__bases__[0]
>> selector.unregister = lambda fd: parent_class.unregister(selector, fd)

It replaces unregister() by _BaseSelectorImpl.unregister(), so "our" data structures are still cleaned (the dict _fd_to_key, for instance).

If a fix for this issue is desired in tulip, the first solution proposed by Guido (closing the selector and let the unregister call fail, see the -trivial- patch attached) is probably good enough.