gh-109496: Detect Py_DECREF() after dealloc in debug mode #109539
Conversation
|
Fix regression introduced in Python 3.12 PEP 683 Immortal Objects: https://peps.python.org/pep-0683/ |
On a Python built in debug mode, Py_DECREF() now calls _Py_NegativeRefcount() if the object is a dangling pointer to deallocated memory: memory filled with 0xDD "dead byte" by the debug hook on memory allocators. The fix is to check the reference count *before* checking for _Py_IsImmortal(). Add test_decref_freed_object() to test_capi.test_misc.
vstinner
force-pushed
the
detect_double_decref
branch
from
e96fc04 to
8ea43e9
Compare
Sorry, I forgot a local change to fix a typo in the NEWS entry. |
| Py_DECREF(obj); | ||
| // obj is a now a dangling pointer |
There was a problem hiding this comment.
Is there a chance this could be an actual pointer to deallocated memory, or memory written to by another thread? Would it be safer to just call Py_DECREF on a fake object which is explicitly memset to 0xDD bytes?
There was a problem hiding this comment.
I prefer to use a functional test: not mock anything here.
This test is very fragile, it makes sure that a dangling pointer has a determinstic behavior. Problem: by definition, it has an undefined behavior :-) But in practice, almost always, the memory should remain around and should just be filled by 0xDD bytes.
If tomorrow the test starts to fail randomly, sure, we can rewrite it by "mocking" the first DECREF() and just pass memory filled by 0xDD byte pattern.
test_capi should not be run in a process with more than 1 thread. libregrtest works hard to make sure that previous tests don't leak tests. Also, it's strongly recommended to run tests with -jN: each test file is run in a fresh process, so with a known number of threads: just 1 :-)
|
Thanks @vstinner for the PR 🌮🎉.. I'm working now to backport this PR to: 3.12. |
…onGH-109539) On a Python built in debug mode, Py_DECREF() now calls _Py_NegativeRefcount() if the object is a dangling pointer to deallocated memory: memory filled with 0xDD "dead byte" by the debug hook on memory allocators. The fix is to check the reference count *before* checking for _Py_IsImmortal(). Add test_decref_freed_object() to test_capi.test_misc. (cherry picked from commit 0bb0d88) Co-authored-by: Victor Stinner <[email protected]>
|
GH-109545 is a backport of this pull request to the 3.12 branch. |
…109539) (#109545) gh-109496: Detect Py_DECREF() after dealloc in debug mode (GH-109539) On a Python built in debug mode, Py_DECREF() now calls _Py_NegativeRefcount() if the object is a dangling pointer to deallocated memory: memory filled with 0xDD "dead byte" by the debug hook on memory allocators. The fix is to check the reference count *before* checking for _Py_IsImmortal(). Add test_decref_freed_object() to test_capi.test_misc. (cherry picked from commit 0bb0d88) Co-authored-by: Victor Stinner <[email protected]>
…on#109539) On a Python built in debug mode, Py_DECREF() now calls _Py_NegativeRefcount() if the object is a dangling pointer to deallocated memory: memory filled with 0xDD "dead byte" by the debug hook on memory allocators. The fix is to check the reference count *before* checking for _Py_IsImmortal(). Add test_decref_freed_object() to test_capi.test_misc.
|
@vstinner This PR broke the AMD64 Arch Linux Asan Debug 3.12 builder. I will make a fix |
|
Nevermind I can see this was fixed in 0a31ff0050e but the backport didn't land |
On a Python built in debug mode, Py_DECREF() now calls _Py_NegativeRefcount() if the object is a dangling pointer to deallocated memory: memory filled with 0xDD "dead byte" by the debug hook on memory allocators. The fix is to check the reference count before checking for _Py_IsImmortal().
Add test_decref_freed_object() to test_capi.test_misc.
Py_DECREFtwice does not result in error with debug build #109496