Add and document py::error_already_set::discard_as_unraisable()

To deal with exceptions that hit destructors or other noexcept functions.

Includes fixes to support Python 2.7 and extends documentation on
error handling.

@virtuald and @YannickJadoul both contributed to this PR.

Author: jbarlow83

docs/advanced/classes.rst

@@ -559,6 +559,39 @@ crucial that instances are deallocated on the C++ side to avoid memory leaks.
     py::class_<MyClass, std::unique_ptr<MyClass, py::nodelete>>(m, "MyClass")
         .def(py::init<>())
 
+.. _destructors_that_call_python:
+
+Destructors that call Python
+============================
+
+If a Python function is invoked from a C++ destructor, an exception may be thrown
+of type :class:`error_already_set`. If this error is thrown out of a class destructor,
+``std::terminate()`` will be called, terminating the process. Class destructors
+must catch all exceptions of type :class:`error_already_set` to discard the Python
+exception using :meth:`error_already_set::discard_as_unraisable`. (In addition,
+you should catch any C++ exceptions that may occur, if any, and deal with them.
+Since C++ exceptions are not Python exceptions, you cannot use ``discard_as_unraisable``).
+
+Every Python function, whether called through a pybind11 wrapper or through the
+Python C-API, should be treated as *possibly throwing*.
+
+For more information, see :ref:`the documentation on exceptions <unraisable_exceptions>`.
+
+.. code-block:: cpp
+
+    class MyClass {
+    public:
+        ~MyClass() {
+            try {
+                py::print("Even printing is dangerous in a destructor");
+                py::exec("raise ValueError('This is an unraisable exception')");
+            }
+            catch (py::error_already_set &e) {
+                e.discard_as_unraisable(__func__);
+            };
+        }
+    };
+
 .. _implicit_conversions:
 
 Implicit conversions

docs/advanced/exceptions.rst

@@ -53,9 +53,13 @@ exceptions:
 |                                      | a Python exception back to Python.   |
 +--------------------------------------+--------------------------------------+
 
-When a Python function invoked from C++ throws an exception, it is converted
-into a C++ exception of type :class:`error_already_set` whose string payload
-contains a textual summary.
+When a Python function invoked from C++ throws an exception, pybind11 will convert
+it into a C++ exception of type :class:`error_already_set` whose string payload
+contains a textual summary. If you call the Python C-API directly, and it
+returns an error, you should ``throw py::error_already_set();``, which allows
+pybind11 will deal with the exception and pass it back to the Python interpreter.
+(You could also use the C-API to clear the error. Either way, the Python error
+must be cleared or Python/pybind11 will be left in an invalid state.)
 
 There is also a special exception :class:`cast_error` that is thrown by
 :func:`handle::call` when the input arguments cannot be converted to Python
@@ -142,3 +146,50 @@ section.
     Exceptions that you do not plan to handle should simply not be caught, or
     may be explicitly (re-)thrown to delegate it to the other,
     previously-declared existing exception translators.
+
+.. _unraisable_exceptions:
+
+Handling unraisable exceptions
+==============================
+
+If a Python function invoked from a C++ destructor or any function marked
+``noexcept(true)`` (collectively, "noexcept functions") throws an exception, there
+is no way to propagate the exception, as such functions may not throw at
+run-time.
+
+Neither Python nor C++ allow exceptions raised in a noexcept function to propagate. In
+Python, an exception raised in a class's ``__del__`` method is logged as an
+unraisable error. In Python 3.8+, a system hook is triggered and an auditing
+event is logged. In C++, ``std::terminate()`` is called to abort immediately.
+
+Any noexcept function should have a try-catch block that traps
+class:`error_already_set` (or any other exception that can occur). Note that pybind11
+wrappers around Python exceptions such as :class:`pybind11::value_error` are *not*
+Python exceptions; they are C++ exceptions that pybind11 catches and converts to
+Python exceptions. Noexcept function cannot propagate these exceptions either.
+You can convert them to Python exceptions and then discard as unraisable.
+
+The following is an elaborate example that catches all possible exception types,
+converting them all to Python exceptions (which is not necessarily what one should
+do).
+
+.. code-block:: cpp
+
+    void nonthrowing_func() noexcept(true) {
+        try {
+            try {
+                ...various functions...
+            }
+            catch (const py::value_error &e) {
+                e.set_error();
+                throw py::error_already_set();
+            }
+            catch (const std::exception &e) {
+                PyErr_SetString(e.what());
+                throw py::error_already_set();
+            }
+        }
+        catch (py::error_already_set &eas) {
+            eas.discard_as_unraisable();
+        }
+    }

include/pybind11/pytypes.h

@@ -337,6 +337,22 @@ class error_already_set : public std::runtime_error {
     /// error variables (but the `.what()` string is still available).
     void restore() { PyErr_Restore(m_type.release().ptr(), m_value.release().ptr(), m_trace.release().ptr()); }
 
+    /// If it is impossible to raise the currently-held error, such as in destructor, we can write
+    /// it out using Python's unraisable hook (sys.unraisablehook). The error context should be
+    /// some object whose repr() helps identify the location of the error. Python already knows the
+    /// type and value of the error, so there is no need to repeat that. For example, __func__ could
+    /// be helpful. After this call, the current object no longer stores the error variables,
+    /// and neither does Python.
+    void discard_as_unraisable(object err_context) {
+        restore();
+        PyErr_WriteUnraisable(err_context.ptr());
+    }
+    void discard_as_unraisable(const char *err_context) {
+        auto obj = reinterpret_steal<object>(PYBIND11_FROM_STRING(err_context));
+        restore();
+        PyErr_WriteUnraisable(obj.ptr());
+    }
+
     // Does nothing; provided for backwards compatibility.
     PYBIND11_DEPRECATED("Use of error_already_set.clear() is deprecated")
     void clear() {}

tests/test_exceptions.cpp

@@ -65,6 +65,25 @@ struct PythonCallInDestructor {
     py::dict d;
 };
 
+
+
+struct PythonAlreadySetInDestructor {
+    PythonAlreadySetInDestructor(const py::str &s) : s(s) {}
+    ~PythonAlreadySetInDestructor() {
+        py::dict foo;
+        try {
+            // Assign to a py::object to force read access of nonexistent dict entry
+            py::object o = foo["bar"];
+        }
+        catch (py::error_already_set& ex) {
+            ex.discard_as_unraisable(s);
+        }
+    }
+
+    py::str s;
+};
+
+
 TEST_SUBMODULE(exceptions, m) {
     m.def("throw_std_exception", []() {
         throw std::runtime_error("This exception was intentionally thrown.");
@@ -183,6 +202,11 @@ TEST_SUBMODULE(exceptions, m) {
         return false;
     });
 
+    m.def("python_alreadyset_in_destructor", [](py::str s) {
+        PythonAlreadySetInDestructor alreadyset_in_destructor(s);
+        return true;
+    });
+
     // test_nested_throws
     m.def("try_catch", [m](py::object exc_type, py::function f, py::args args) {
         try { f(*args); }

tests/test_exceptions.py

@@ -1,4 +1,6 @@
 # -*- coding: utf-8 -*-
+import sys
+
 import pytest
 
 from pybind11_tests import exceptions as m
@@ -48,6 +50,34 @@ def test_python_call_in_catch():
     assert d["good"] is True
 
 
+def test_python_alreadyset_in_destructor(monkeypatch, capsys):
+    hooked = False
+    triggered = [False]  # mutable, so Python 2.7 closure can modify it
+
+    if hasattr(sys, 'unraisablehook'):  # Python 3.8+
+        hooked = True
+        default_hook = sys.unraisablehook
+
+        def hook(unraisable_hook_args):
+            exc_type, exc_value, exc_tb, err_msg, obj = unraisable_hook_args
+            if obj == 'PythonAlreadySetInDestructor':
+                triggered[0] = True
+            default_hook(unraisable_hook_args)
+            return
+
+        # Use monkeypatch so pytest can apply and remove the patch as appropriate
+        monkeypatch.setattr(sys, 'unraisablehook', hook)
+
+    assert m.python_alreadyset_in_destructor('already_set demo') is True
+    if hooked:
+        assert triggered[0] is True
+
+    captured = capsys.readouterr()
+    assert captured.err.startswith(
+        "Exception ignored in: 'PythonAlreadySetInDestructor'"
+    )
+
+
 def test_exception_matches():
     assert m.exception_matches()
     assert m.exception_matches_base()