Skip to content

pymethods: add support for __del__#2484

Open
messense wants to merge 15 commits intoPyO3:mainfrom
messense:__del__
Open

pymethods: add support for __del__#2484
messense wants to merge 15 commits intoPyO3:mainfrom
messense:__del__

Conversation

@messense
Copy link
Member

@messense messense commented Jun 27, 2022

Adds support for __del__ in #[pymethods], mapped to CPython's tp_finalize slot.

#[pymethods]
impl MyClass {
    fn __del__(&mut self) {
        println!("instance is being destroyed");
    }
}

Why a custom impl_del_slot instead of SlotDef?

The generic SlotDef machinery assumes the trampoline module name and the FFI type name are the same (e.g., inquiry for Py_tp_clear). For tp_finalize, the FFI type is destructor (void-returning) but the trampoline needs to be finalizefunc — they don't match. A custom impl_del_slot (following the same pattern as impl_clear_slot) avoids complicating the generic machinery for a single special case.

Alternative considered: Adding a separate ffi_type field to SlotDef to decouple the trampoline name from the cast type. Rejected because it would add complexity to the generic path that only __del__ would use.

Why a dedicated trampoline instead of trampoline_unraisable?

CPython's slot_tp_finalize saves the active exception before calling __del__ and restores it afterwards, since finalizers can run at arbitrary points during execution. We need the same save/restore semantics, but crucially, the restore must happen even if __del__ panics. If we used trampoline_unraisable (which wraps the entire body in catch_unwind), a panic would unwind past the restore, silently swallowing the active exception. The dedicated trampoline puts catch_unwind around only the user's __del__ call, so the exception restore always runs.

Alternative considered: Nesting the error handling inside trampoline_unraisable's body — save exception, call __del__, write_unraisable on error, restore exception, return Ok(()). This was the initial implementation but was replaced because a panic in __del__ would skip the restore, leaking the saved exception PyObject* pointers and losing the active exception state.

Why call PyObject_CallFinalizerFromDealloc from tp_dealloc?

PyO3 replaces CPython's subtype_dealloc with its own tp_dealloc, so tp_finalize would never be called during normal deallocation without explicit invocation. The GC track/untrack dance in tp_dealloc_with_gc (untrack → re-track → finalize → untrack) mirrors CPython's subtype_dealloc to correctly handle object resurrection and GC visibility during finalization.

Alternative considered: Hooking into subtype_dealloc or using tp_del (the deprecated pre-PEP 442 slot). Rejected because PyO3 already owns the tp_dealloc slot, so calling PyObject_CallFinalizerFromDealloc directly is the most straightforward approach, and tp_del is deprecated since Python 3.4.

abi3 limitation

PyObject_CallFinalizerFromDealloc is not in the stable ABI, so on abi3 builds __del__ won't fire during normal refcount-based deallocation. The Py_tp_finalize slot is still set, so __del__ is invoked by the cyclic garbage collector for GC-tracked types and can be called explicitly via Python code. This is documented in the guide.

Alternative considered: Manually inlining the logic of PyObject_CallFinalizerFromDealloc (temporarily resurrect the object, call tp_finalize, check refcount) using only stable-ABI functions. Rejected because it would duplicate subtle CPython internals and could diverge across Python versions.

Closes #2479

Copy link
Member

@davidhewitt davidhewitt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! Please also add a CHANGELOG and add to protocols.md in the guide.

Note that also on Python 3.7 we need to set Py_TPFLAGS_HAVE_FINALIZE, it's ignored on newer versions.

@davidhewitt
Copy link
Member

davidhewitt commented May 27, 2023

I found today that if we don't implement tp_dealloc then the default tp_dealloc implementation (at least for CPython) will call tp_finalize. This gives us a potential route to support this and also fix #3064 at the same time.

However the documentation in this area is confusing, as it implies that tp_dealloc doesn't have a default - python/cpython#105018 / https://docs.python.org/3/c-api/typeobj.html#c.PyTypeObject.tp_dealloc

I think it may be possible that we use that default implementation to support this case, but we'll need to test carefully whether this also works for all Python versions, PyPy etc.

@davidhewitt davidhewitt mentioned this pull request Jun 16, 2023
6 tasks
@davidhewitt davidhewitt added this to the 0.20 milestone Aug 11, 2023
@davidhewitt davidhewitt modified the milestones: 0.20, 0.22 Oct 2, 2023
@davidhewitt
Copy link
Member

I spoke to vstinner (thought no need to ping so no @) briefly about this, I understood that PyObject_CallFinalizerFromDealloc could be added to the limited API in 3.13. So maybe we can figure out a way to proceed with this where __del__ is supported without abi3 for now and then we can add abi3 support on 3.13.

@davidhewitt davidhewitt modified the milestones: 0.22, 0.24 Oct 4, 2024
@davidhewitt davidhewitt removed this from the 0.24 milestone Jul 25, 2025
@messense messense force-pushed the __del__ branch 4 times, most recently from 19cd088 to 107effb Compare February 28, 2026 14:58
@codspeed-hq
Copy link

codspeed-hq bot commented Feb 28, 2026

Merging this PR will not alter performance

✅ 105 untouched benchmarks
⏩ 1 skipped benchmark1


Comparing messense:del (9cd1578) with main (d07a1ab)

Open in CodSpeed

Footnotes

  1. 1 benchmark was skipped, so the baseline result was used instead. If it was deleted from the codebase, click here and archive it to remove it from the performance reports.

@messense messense added CI-no-fail-fast If one job fails, allow the rest to keep testing CI-build-full labels Feb 28, 2026
@messense messense force-pushed the __del__ branch 2 times, most recently from ef04513 to 5086f38 Compare March 7, 2026 13:37
Add the foundational pieces needed for __del__ support:

- Implement PyCallbackOutput for () to support void-returning callbacks
- Add finalizefunc trampoline for tp_finalize that:
  - Saves/restores the current exception around the call
    (matching CPython's slot_tp_finalize behavior)
  - Routes errors to sys.unraisablehook via write_unraisable
  - Handles both Python 3.12+ (PyErr_GetRaisedException) and
    older versions (PyErr_Fetch/PyErr_Restore)
Register __del__ as a PyMethodProtoKind::Del variant with a custom
impl_del_slot function (similar to __clear__'s impl_clear_slot).

The generated code:
- Creates a wrapper function with the finalizefunc trampoline
- Maps to the Py_tp_finalize slot with ffi::destructor type
- Validates that __del__ is an instance method with no arguments
Since PyO3 replaces CPython's subtype_dealloc with its own tp_dealloc,
tp_finalize (which implements __del__) would never be called. This commit
adds explicit calls to PyObject_CallFinalizerFromDealloc in both
tp_dealloc (non-GC) and tp_dealloc_with_gc (GC) implementations.

Key behaviors matching CPython's subtype_dealloc:
- For non-GC types: call finalizer, abort dealloc if object is resurrected
- For GC types: untrack, re-track, call finalizer (so the finalizer can
  make the object visible to the GC), then untrack again before proceeding

This is currently gated behind #[cfg(not(Py_LIMITED_API))] since
PyObject_CallFinalizerFromDealloc is not in the stable/limited API.
For abi3 builds, the Py_tp_finalize slot is still set correctly, so:
- Python code can call obj.__del__() explicitly
- The cyclic GC will call tp_finalize for GC objects in reference cycles
- Simple refcounted deallocation won't invoke __del__ on abi3 (known limitation)
Test cases:
- test_del_called_explicitly: verify __del__ can be called as a method
- test_del_called_on_dealloc: verify __del__ is called when object is
  deallocated (refcount drops to zero)
- test_del_error_is_unraisable: verify exceptions in __del__ are routed
  to sys.unraisablehook (not propagated)
- test_del_with_gc: verify __del__ works correctly with GC types
  (classes that also have __traverse__ and __clear__)
- Remove the 'not yet supported' note from protocols.md
- Add __del__ to the basic customizations section with signature,
  description, and abi3 limitation note
- Add changelog entry
Tests that rely on __del__ being called during deallocation depend on
PyObject_CallFinalizerFromDealloc, which is not available in abi3 builds.
Add #[cfg(not(Py_LIMITED_API))] guards to:
- test_del_called_on_dealloc
- test_del_error_is_unraisable
- test_del_with_gc

test_del_called_explicitly (which calls __del__ as a Python method) works
on all builds and is left unguarded.

Also clarify the finalizefunc trampoline comment to explain why we handle
write_unraisable inside the body rather than letting trampoline_unraisable
handle it: the saved exception must be restored before trampoline_unraisable
runs its error handler, otherwise it would clobber the restored exception.
- test_del_via_cyclic_gc: creates an actual reference cycle
  (obj.cycle = obj), drops external references, then calls gc.collect().
  Verifies __del__ is invoked by the cyclic GC's own tp_finalize call.
  This test is NOT gated behind cfg(not(Py_LIMITED_API)) since the
  cyclic GC calls tp_finalize directly (not through our tp_dealloc).

- test_del_resurrection: a Python subclass __del__ stores self in a
  global variable, preventing deallocation. Verifies that
  PyObject_CallFinalizerFromDealloc correctly aborts dealloc (returns -1)
  when the object's refcount stays above zero, and the object remains
  usable afterwards.
Use a dedicated trampoline for tp_finalize instead of delegating to
trampoline_unraisable. The exception save/restore now runs outside
catch_unwind, ensuring that a panic in __del__ does not silently
swallow a pre-existing Python exception.

Add test_del_panic_preserves_active_exception to verify this behavior,
along with test_del_with_py_arg for __del__ methods taking Python<'_>.
UnraisableCapture requires #[cfg(all(feature = "macros", Py_3_8))],
so tests that use it must also be gated on Py_3_8 to compile on
CPython 3.7 configs.
On Py_GIL_DISABLED builds the GC may run asynchronously in a separate
thread and need multiple collection passes. Retry gc.collect() in a
loop (with a short sleep on free-threaded builds), matching the pattern
used in test_gc.rs.
- Gate test_del_called_on_dealloc, test_del_with_gc with Py_3_8 because
  Py_tp_finalize set via PyType_FromSpec is not properly applied on
  Python 3.7 (tp_finalize stays NULL on the type).
- Gate test_del_via_cyclic_gc with Py_3_8 for the same reason — the GC
  also relies on tp_finalize being set.
- Gate test_del_panic_preserves_active_exception with not(wasm32)
  because panics abort on wasm32-wasip1 (no unwinding support), so
  catch_unwind in the finalizefunc trampoline cannot catch the panic.
GraalPy does not properly support PyObject_CallFinalizerFromDealloc,
causing crashes during deallocation. Gate the calls with not(GraalPy)
alongside the existing not(Py_LIMITED_API) and not(PyPy) guards.
@messense messense marked this pull request as ready for review March 16, 2026 13:06
@messense
Copy link
Member Author

With the help from LLM, made some progress on this, tests are passing, review would be appreciated!

I have also looked into removing our tp_dealloc implementation but turns out to be pretty hard to get everything right.

@messense messense requested a review from davidhewitt March 16, 2026 13:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CI-build-full CI-no-fail-fast If one job fails, allow the rest to keep testing

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support __del__

2 participants