[BUG]: leaked types and functions in python 3.12

[Yikai-Liao]

Problem description

I'm writing an open source midi parsing library symusic.

We previously use pybind11 as the binding framework, and switch to nanobind for better performance in the last update.

Owing to their silimar interface, I switch to nanobind easily, without changing much of the code. And pybind11 just worked well with those binding codes.

And here is the current binding codes.

However, I get tons of leak errors today when just importing the new version of library in a python script.

nanobind: leaked 56 types!
 - leaked type "symusic.core.KeySignatureSecondList"
 - leaked type "symusic.core.TextMetaTick"
 - leaked type "symusic.core.KeySignatureQuarter"
 - leaked type "symusic.core.ScoreTick"
 - leaked type "symusic.core.PitchBendQuarterList"
 - leaked type "symusic.core.ScoreQuarter"
 - leaked type "symusic.core.TempoTick"
 - leaked type "symusic.core.TextMetaQuarter"
 - leaked type "symusic.core.TextMetaQuarterList"
 - leaked type "symusic.core.PedalQuarter"
 - leaked type "symusic.core.TimeSignatureTick"
 - ... skipped remainder
nanobind: leaked 1276 functions!
 - leaked function "__getstate__"
 - leaked function "__deepcopy__"
 - leaked function "__bool__"
 - leaked function ""
 - leaked function "clear"
 - leaked function "__repr__"
 - leaked function "__bool__"
 - leaked function "__bool__"
 - leaked function ""
 - leaked function "count"
 - leaked function "__contains__"
 - ... skipped remainder
nanobind: this is likely caused by a reference counting issue in the binding code.

This problem would only occur in script, not in ipython.
Since we have upload this version to pypi, it is easy to reproduce:
pip install symusic==0.3.0

Reproducible example code

import symusic

[Yikai-Liao]

It seems that even the default __bool__ methods defined in bind_vector got an error. I didn't define any __bool__ function manually.

[Yikai-Liao]

I tested and found that this warning message only appears on python 3.12, both 3.11 and 3.10 work fine。
I run test scripts on both Linux on Windows

[Yikai-Liao]

@wjakob Is there any plan to deal with such problems?

[wjakob]

This is either a reference counting leak / garbage collection issue in your code, or in Python itself. Basically it's not something that can be fixed in nanobind.

For what it's worth, I am developing a huge project (~25K lines of code of nanobind bindings along) and have no such warnings on Python 3.12, so I suspect the former.

[wjakob]

I suggest that you use "divide and conquer" to localize the problem more, by disabling half of your bindings and then seeing if that still triggers the issue. It could be a single leak that causes all of these errors. Please also see the FAQ entry about leaks.

[Yikai-Liao]

@wjakob But the strange thing is that, why there is no warning on other python versions?

[wjakob]

One thing I have noticed is that Python 3.12 will (currently) leak the last exception raised by a program. Perhaps importing your bindings generates an exception somewhere, and that exception object holds a reference to the module or some functions.

In any case, this is highly unlikely a nanobind issue. If it bothers you and you have enough energy, the please escalate to the Python project and let them fix it. Otherwise, you can disable the leak warnings (in that case, please set the NB_DOMAIN cmake parameter).

[Yikai-Liao]

Thanks, I understand that.

python/cpython#113190

I have found this, so I now believe that it must be cpython 3.12's problem