bpo-34172: multiprocessing.Pool leaks resources after being deleted #8450
Conversation
tzickel
force-pushed
the
fixdelpoollatest
branch
2 times, most recently
from
351742e to
c286945
Compare
|
Maybe we should rename the Process method inside Pool to be _Process, since it doesn't have a usage outside (and I had to change it's definition to remove the circular reference) ? |
|
@pitrou any chance to get this in the next python 2.7 / 3.7 releases ? |
Due to a circular reference inside the Pool implementation, there was a resource leak if the object was not closed/terminated. This patch removes the circular reference, and now the code acts like the documentation.
tzickel
force-pushed
the
fixdelpoollatest
branch
from
c286945 to
d679a0b
Compare
|
Done. |
|
Guess this won't make it to 3.7.1 |
|
Wow, I'm really sorry about that. I'm not sure about the rules post-rc but it probably will have to wait for 3.7.2 indeed :-/ |
…ythonGH-8450) Fix a reference issue inside multiprocessing.Pool that caused the pool to remain alive if it was deleted without being closed or terminated explicitly. (cherry picked from commit 97bfe8d) Co-authored-by: tzickel <[email protected]>
|
GH-9676 is a backport of this pull request to the 3.7 branch. |
…ythonGH-8450) Fix a reference issue inside multiprocessing.Pool that caused the pool to remain alive if it was deleted without being closed or terminated explicitly. (cherry picked from commit 97bfe8d) Co-authored-by: tzickel <[email protected]>
|
GH-9677 is a backport of this pull request to the 3.6 branch. |
|
What about the 2.7 back port ? I should manually do another PR for it ?
|
|
If you care about 2.7, that will probably be needed. I could push the "2.7 backport button" but the backport would almost certainly fail due to conflicts. But I wouldn't bother with 2.7 if I were you. |
…H-8450) (GH-9676) Fix a reference issue inside multiprocessing.Pool that caused the pool to remain alive if it was deleted without being closed or terminated explicitly. (cherry picked from commit 97bfe8d) Co-authored-by: tzickel <[email protected]>
…H-8450) (GH-9677) Fix a reference issue inside multiprocessing.Pool that caused the pool to remain alive if it was deleted without being closed or terminated explicitly. (cherry picked from commit 97bfe8d) Co-authored-by: tzickel <[email protected]>
2.7 is still supported, and this an bug fix, not something new (also this bug was detected on code still written in python 2...). PR #9686 is a manual backport. |
Changes in this commit: 1. Use a _strong_ reference between the Pool and associated iterators 2. Rework PR #8450 to eliminate a cycle in the Pool. There is no test in this commit because any test that automatically tests this behaviour needs to eliminate the pool before joining the pool to check that the pool object is garbaged collected/does not hang. But doing this will potentially leak threads and processes (see https://bugs.python.org/issue35413).
Due to a circular reference inside the Pool implementation, there
was a resource leak if the object was not closed/terminated.
This patch removes the circular reference, and now the code acts
like the documentation.
https://bugs.python.org/issue34172