Refcounting for Singular rings

[gagern]

Ref counting for Singular rings

Python makes no guarantees that destructors are called if circular references are involved. This patch implements an extra Python proxy layer that correctly refcounts Singular rings. In contrast to our earlier code, it will release the singular rings once they are no longer in use. This triggers various issues in Sage/libSingular that are dealt with in the follow-up ticket #10903

Historic discussion

As described for the Sage on Gentoo project, there is a problem in how parts of sage use __deallocate__ in their Cython code. I've actually traced an instance of groebner_strategy.pyx causing segmentation faults under Python 2.7.

What happens is that the garbage collector invokes the tp_clear function for the object. Its implementation is provided by Cython, and one of its effects is that every python reference will be set to None. A bit later on, tp_dealloc is called and invokes the __deallocate__ method. By that time, _parent is None, so accessing _parent._ring is a bad idea, and in this case it turns out to be null.

Others have had similar problems before. There are crude workarounds floating around. I guess a proper solution would be twaking cython to allow custom code in the tp_clear function. In other words, have a "magic" mehod __clear__ similar to the magic __deallocate__. But I'll wait for comments here first, before taking this to cython upstream. Perhaps people with more experience have better solutions to offer. And I won't mind if you decide to take this to Cython devs yourselves.

Apply

Apply:

• attachment: trac_11339_refcount_singular_rings.patch
• attachment: trac_11339_refcount_singular_polynomials.patch

Depends on #10903

Dependencies: #10903 (for doctests)

CC: @kiwifb @robertwb @burcin @malb @simon-king-jena @nexttime

Component: algebra

Keywords: sd31

Author: Volker Braun, Martin von Gagern

Reviewer: François Bissey, Steven Trogdon, Martin Albrecht

Merged: sage-4.7.2.alpha4

Issue created by migration from https://trac.sagemath.org/ticket/11339

[vbraun]

comment:2

The documentation is fairly clear, you are only allowed to touch C data structures in __dealloc__: http://docs.cython.org/src/userguide/special_methods.html#finalization-method-dealloc

Really, this shows the difference between the C++ and the Python object model. In Python, executing constructors/destructors in derived classes is optional and leaves a lot of freedom to the programmer. Because of the garbage collection you can allocate resources wherever you want. By comparison, C++ is very strict and essentially forces you to use RAII. This is why there is no try/finally in C++. Which is why there is necessarily some tension in Cython between C++ and Python object instantiation/finalization.

Having said that, it seems like the cleanest solution would be if Cython would also call the Python destructor __del__ according to the normal Python rules (why __clear__?). Then the Python and C++ object model could just peacefully coexist. The lifetime of a cdef class would be

__cinit__()    # always called
__init__()     # not necessarily called for derived classes
... do something ...
__del__()      # not necessarily called for derived classes
__dealloc__()  # always called

[kiwifb]

comment:3

Thanks for your input Volker. I think we should definitely put some work in libs/singular to have some __del__ and __dealloc__ functions. I think the current state of things in there is a bit messy. I remember reading about __del__ in the python doc about garbage collection and wondering why singular_ring_delete in libs/singular/rings.pyx wasn't a __del__ method instead as its intent is clear.

[vbraun]

comment:4

Just to clarify, right now Cython extension classes do not call the Python destructor __del__ upon finalization, so Sage can't rely on this mechanism. I'm not quite sure if it is an intentional thing or if the Cython devs just haven't gotten around to implementing it. Maybe we can hack on that on Sage Days 31 since both Robert and Burcin will be around :-)

[kiwifb]

comment:5

OK, well something as to be done about this if we want to move to python-2.7. I think this problem is probably the biggest issue in the way of #9958 there are a few other things to look at but that's the only one producing a segfault.

[gagern]

comment:6

I notice that PyTypeObject has a member tp_del which is not included in the documentation. So perhaps it's as simple as associating a function with that slot. Otoh, perhaps there is a reason that member isn't documented. The single reply to Python issue 4934 doesn't rule out that possibility.

[gagern]

Workaround patch

[gagern]

Attachment: bug11339a.patch.gz

Workaround hg bundle

[kiwifb]

comment:8

Attachment: bug11339a.bundle.gz

For the record this works with sage-on-gentoo with python-2.7.1 so as far as I am concerned it works as advertised so far. I can do a review against a vanilla sage using python-2.6 later this week provided that I get the all clear to go back to work (we got a couple more earthquakes down here - the magnitude 6 one was scary but very few people got hurt).

What is the "workaround hg bundle" Martin? I can reformat things if needed.

[gagern]

comment:9

Replying to @kiwifb:

What is the "workaround hg bundle" Martin? I can reformat things if needed.

It's what I perceived to be the hg equivalent of a "bzr send": a file which describes a number of hg commits inclusing metadata, so that other hg users can pull from it.
See hg manual for unbundle and related documentation.