creating new greenlets appears to leak memory rapidly in python 3.11 / linux

[zzzeek]

hi there -

This is indirect from SQLAlchemy where we use greenlet for our asyncio implementation, reported in sqlalchemy/sqlalchemy#8763.

The below script creates a greenlet and calls one switch. Running on linux under any Python 3.11 version with greenlet installed from pypi (greenlet-2.0.0.post0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl), the script below grows by about 10M per second. Under Python 3.10, memory is flat at about 10 megs resident.

import greenlet
import time


def greenlet_spawn(fn, *args, **kwargs):
    context = greenlet.greenlet(fn, greenlet.getcurrent())
    result = context.switch(*args, **kwargs)
    return result



def hi():
    time.sleep(.001)
    return "hello"

def go():
    while True:
        print(greenlet_spawn(hi))
go()

[salty-horse]

Previously reported here: gevent/gevent#1924

[jamadden]

This issue should be solved by using the just-released greenlet 2.0.1.

[zzzeek]

thanks!!!

[zzzeek]

The test program above works now however in our integration case we are still seeing rapid memory increases, unfortunately. I will attempt to devise a new test case.

[zzzeek]

hMmmmmm OK I think you are good actually, there now seems to be an entirely different memory leak, again python 3.11 only , either in the sqlite3 module or the aiosqlite module. how strange huh? I'll keep you posted but in our original original case with asyncpg (which is the important driver), memory seems to be stable.

[zzzeek]

OK we're good here, aiosqlite has some problem, strange, but i can reproduce without greenlet involved. thanks for the quick fix!

[zzzeek]

@jamadden I'm still in the disbelief stage, but it appears python 3.11 itself has a similar problem as the one here, with threading.Thread(). That is, the program below which is the threading equivalent also leaks memory at a similar rate:

from threading import Thread

while True:
    t1 = Thread()
    t1.start()

I'm going to quadruple check this before reporting to them but do you think the issue you found here might be similarly involved for Python's own Thread object?