Pytest INTERNALERROR execnet.gateway_base.DumpError: can't serialize <class '_pytest._code.code.ExceptionInfo'> with v16.0 #303
Description
Hi there and thanks for your continued work on this plugin :)
Unfortunately, today's v16.0 release seems to have broken our CI test suite and even after spending some time digging through the changes and your source code, I'm not completely sure why.
So here's the test that's erroring out:
import os
import platform
import sys
from pathlib import Path
from typing import TypedDict
import pytest
from ixmp.testing import run_notebook
group_base_name = platform.system() + platform.python_version()
GHA = "GITHUB_ACTIONS" in os.environ
FLAKY = pytest.mark.flaky(
reruns=5,
rerun_delay=2,
condition=GHA and platform.system() == "Windows",
reason="Flaky; see iiasa/ixmp#543",
)
class DefaultKwargs(TypedDict, total=False):
timeout: int
@pytest.fixture(scope="session")
def default_args() -> DefaultKwargs:
"""Default arguments for :func:`.run_notebook."""
# Use a longer timeout for GHA
return dict(timeout=30) if GHA else dict()
@FLAKY
@pytest.mark.xdist_group(name=f"{group_base_name}-1")
@pytest.mark.rixmp
# TODO investigate and resolve the cause of the time outs; remove this mark
@pytest.mark.skipif(GHA and sys.platform == "linux", reason="Times out")
def test_R_transport(
tutorial_path: Path,
tmp_path: Path,
tmp_env: os._Environ[str],
default_args: DefaultKwargs,
) -> None:
fname = tutorial_path / "transport" / "R_transport.ipynb"
nb, errors = run_notebook(
fname, tmp_path, tmp_env, kernel_name="IR", **default_args
)
assert errors == []And this is the error message we receive:
Error message from ubuntu-latest-py3.13
============================= test session starts ==============================
platform linux -- Python 3.13.7, pytest-8.4.1, pluggy-1.6.0 -- /home/runner/work/ixmp/ixmp/.venv/bin/python3
cachedir: .pytest_cache
benchmark: 5.1.0 (defaults: timer=time.perf_counter disable_gc=False min_rounds=5 min_time=0.000005 max_time=1.0 calibration_precision=10 warmup=False warmup_iterations=100000)
ixmp config: BaseValues(platform={'default': 'local', 'local': {'class': 'jdbc', 'driver': 'hsqldb'}, 'ixmp4-local': {'class': 'ixmp4', 'dsn': 'sqlite:////home/runner/.local/share/ixmp4/databases/local.sqlite3', 'ixmp4_name': 'local', 'jdbc_compat': True}})
rootdir: /home/runner/work/ixmp/ixmp
configfile: pyproject.toml
plugins: benchmark-5.1.0, rerunfailures-16.0, anyio-4.10.0, typeguard-4.4.4, xdist-3.8.0, cov-6.2.1, pytest_httpserver-1.1.3
created: 2/2 workers
2 workers [482 items]
scheduling tests via LoadGroupScheduling
ixmp/tests/test_tutorials.py::test_py_transport[ixmp4]@Linux3.13.7-0
ixmp/tests/test_tutorials.py::[email protected]
INTERNALERROR> def worker_internal_error(
INTERNALERROR> self, node: WorkerController, formatted_error: str
INTERNALERROR> ) -> None:
INTERNALERROR> """
INTERNALERROR> pytest_internalerror() was called on the worker.
INTERNALERROR>
INTERNALERROR> pytest_internalerror() arguments are an excinfo and an excrepr, which can't
INTERNALERROR> be serialized, so we go with a poor man's solution of raising an exception
INTERNALERROR> here ourselves using the formatted message.
INTERNALERROR> """
INTERNALERROR> self._active_nodes.remove(node)
INTERNALERROR> try:
INTERNALERROR> > assert False, formatted_error
INTERNALERROR> E AssertionError: Traceback (most recent call last):
INTERNALERROR> E File "/home/runner/work/ixmp/ixmp/.venv/lib/python3.13/site-packages/_pytest/main.py", line 289, in wrap_session
INTERNALERROR> E session.exitstatus = doit(config, session) or 0
INTERNALERROR> E ~~~~^^^^^^^^^^^^^^^^^
INTERNALERROR> E File "/home/runner/work/ixmp/ixmp/.venv/lib/python3.13/site-packages/_pytest/main.py", line 343, in _main
INTERNALERROR> E config.hook.pytest_runtestloop(session=session)
INTERNALERROR> E ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
INTERNALERROR> E File "/home/runner/work/ixmp/ixmp/.venv/lib/python3.13/site-packages/pluggy/_hooks.py", line 512, in __call__
INTERNALERROR> E return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
INTERNALERROR> E ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR> E File "/home/runner/work/ixmp/ixmp/.venv/lib/python3.13/site-packages/pluggy/_manager.py", line 120, in _hookexec
INTERNALERROR> E return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR> E ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR> E File "/home/runner/work/ixmp/ixmp/.venv/lib/python3.13/site-packages/pluggy/_callers.py", line 167, in _multicall
INTERNALERROR> E raise exception
INTERNALERROR> E File "/home/runner/work/ixmp/ixmp/.venv/lib/python3.13/site-packages/pluggy/_callers.py", line 139, in _multicall
INTERNALERROR> E teardown.throw(exception)
INTERNALERROR> E ~~~~~~~~~~~~~~^^^^^^^^^^^
INTERNALERROR> E File "/home/runner/work/ixmp/ixmp/.venv/lib/python3.13/site-packages/_pytest/logging.py", line 801, in pytest_runtestloop
INTERNALERROR> E return (yield) # Run all the tests.
INTERNALERROR> E ^^^^^
INTERNALERROR> E File "/home/runner/work/ixmp/ixmp/.venv/lib/python3.13/site-packages/pluggy/_callers.py", line 139, in _multicall
INTERNALERROR> E teardown.throw(exception)
INTERNALERROR> E ~~~~~~~~~~~~~~^^^^^^^^^^^
INTERNALERROR> E File "/home/runner/work/ixmp/ixmp/.venv/lib/python3.13/site-packages/_pytest/terminal.py", line 688, in pytest_runtestloop
INTERNALERROR> E result = yield
INTERNALERROR> E ^^^^^
INTERNALERROR> E File "/home/runner/work/ixmp/ixmp/.venv/lib/python3.13/site-packages/pluggy/_callers.py", line 139, in _multicall
INTERNALERROR> E teardown.throw(exception)
INTERNALERROR> E ~~~~~~~~~~~~~~^^^^^^^^^^^
INTERNALERROR> E File "/home/runner/work/ixmp/ixmp/.venv/lib/python3.13/site-packages/pytest_cov/plugin.py", line 340, in pytest_runtestloop
INTERNALERROR> E result = yield
INTERNALERROR> E ^^^^^
INTERNALERROR> E File "/home/runner/work/ixmp/ixmp/.venv/lib/python3.13/site-packages/pluggy/_callers.py", line 121, in _multicall
INTERNALERROR> E res = hook_impl.function(*args)
INTERNALERROR> E File "/home/runner/work/ixmp/ixmp/.venv/lib/python3.13/site-packages/xdist/remote.py", line 206, in pytest_runtestloop
INTERNALERROR> E self.run_one_test()
INTERNALERROR> E ~~~~~~~~~~~~~~~~~^^
INTERNALERROR> E File "/home/runner/work/ixmp/ixmp/.venv/lib/python3.13/site-packages/xdist/remote.py", line 227, in run_one_test
INTERNALERROR> E self.config.hook.pytest_runtest_protocol(item=item, nextitem=nextitem)
INTERNALERROR> E ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR> E File "/home/runner/work/ixmp/ixmp/.venv/lib/python3.13/site-packages/pluggy/_hooks.py", line 512, in __call__
INTERNALERROR> E return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
INTERNALERROR> E ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR> E File "/home/runner/work/ixmp/ixmp/.venv/lib/python3.13/site-packages/pluggy/_manager.py", line 120, in _hookexec
INTERNALERROR> E return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR> E ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR> E File "/home/runner/work/ixmp/ixmp/.venv/lib/python3.13/site-packages/pluggy/_callers.py", line 167, in _multicall
INTERNALERROR> E raise exception
INTERNALERROR> E File "/home/runner/work/ixmp/ixmp/.venv/lib/python3.13/site-packages/pluggy/_callers.py", line 139, in _multicall
INTERNALERROR> E teardown.throw(exception)
INTERNALERROR> E ~~~~~~~~~~~~~~^^^^^^^^^^^
INTERNALERROR> E File "/home/runner/work/ixmp/ixmp/.venv/lib/python3.13/site-packages/_pytest/warnings.py", line 90, in pytest_runtest_protocol
INTERNALERROR> E return (yield)
INTERNALERROR> E ^^^^^
INTERNALERROR> E File "/home/runner/work/ixmp/ixmp/.venv/lib/python3.13/site-packages/pluggy/_callers.py", line 139, in _multicall
INTERNALERROR> E teardown.throw(exception)
INTERNALERROR> E ~~~~~~~~~~~~~~^^^^^^^^^^^
INTERNALERROR> E File "/home/runner/work/ixmp/ixmp/.venv/lib/python3.13/site-packages/_pytest/assertion/__init__.py", line 192, in pytest_runtest_protocol
INTERNALERROR> E return (yield)
INTERNALERROR> E ^^^^^
INTERNALERROR> E File "/home/runner/work/ixmp/ixmp/.venv/lib/python3.13/site-packages/pluggy/_callers.py", line 139, in _multicall
INTERNALERROR> E teardown.throw(exception)
INTERNALERROR> E ~~~~~~~~~~~~~~^^^^^^^^^^^
INTERNALERROR> E File "/home/runner/work/ixmp/ixmp/.venv/lib/python3.13/site-packages/_pytest/unittest.py", line 475, in pytest_runtest_protocol
INTERNALERROR> E return (yield)
INTERNALERROR> E ^^^^^
INTERNALERROR> E File "/home/runner/work/ixmp/ixmp/.venv/lib/python3.13/site-packages/pluggy/_callers.py", line 121, in _multicall
INTERNALERROR> E res = hook_impl.function(*args)
INTERNALERROR> E File "/home/runner/work/ixmp/ixmp/.venv/lib/python3.13/site-packages/pytest_rerunfailures.py", line 580, in pytest_runtest_protocol
INTERNALERROR> E item.ihook.pytest_runtest_logreport(report=report)
INTERNALERROR> E ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^
INTERNALERROR> E File "/home/runner/work/ixmp/ixmp/.venv/lib/python3.13/site-packages/pluggy/_hooks.py", line 512, in __call__
INTERNALERROR> E return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
INTERNALERROR> E ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR> E File "/home/runner/work/ixmp/ixmp/.venv/lib/python3.13/site-packages/pluggy/_manager.py", line 120, in _hookexec
INTERNALERROR> E return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR> E ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR> E File "/home/runner/work/ixmp/ixmp/.venv/lib/python3.13/site-packages/pluggy/_callers.py", line 167, in _multicall
INTERNALERROR> E raise exception
INTERNALERROR> E File "/home/runner/work/ixmp/ixmp/.venv/lib/python3.13/site-packages/pluggy/_callers.py", line 121, in _multicall
INTERNALERROR> E res = hook_impl.function(*args)
INTERNALERROR> E File "/home/runner/work/ixmp/ixmp/.venv/lib/python3.13/site-packages/xdist/remote.py", line 289, in pytest_runtest_logreport
INTERNALERROR> E self.sendevent("testreport", data=data)
INTERNALERROR> E ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR> E File "/home/runner/work/ixmp/ixmp/.venv/lib/python3.13/site-packages/xdist/remote.py", line 126, in sendevent
INTERNALERROR> E self.channel.send((name, kwargs))
INTERNALERROR> E ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^
INTERNALERROR> E File "/home/runner/work/ixmp/ixmp/.venv/lib/python3.13/site-packages/execnet/gateway_base.py", line 912, in send
INTERNALERROR> E self.gateway._send(Message.CHANNEL_DATA, self.id, dumps_internal(item))
INTERNALERROR> E ~~~~~~~~~~~~~~^^^^^^
INTERNALERROR> E File "/home/runner/work/ixmp/ixmp/.venv/lib/python3.13/site-packages/execnet/gateway_base.py", line 1629, in dumps_internal
INTERNALERROR> E return _Serializer().save(obj) # type: ignore[return-value]
INTERNALERROR> E ~~~~~~~~~~~~~~~~~~^^^^^
INTERNALERROR> E File "/home/runner/work/ixmp/ixmp/.venv/lib/python3.13/site-packages/execnet/gateway_base.py", line 1647, in save
INTERNALERROR> E self._save(obj)
INTERNALERROR> E ~~~~~~~~~~^^^^^
INTERNALERROR> E File "/home/runner/work/ixmp/ixmp/.venv/lib/python3.13/site-packages/execnet/gateway_base.py", line 1667, in _save
INTERNALERROR> E dispatch(self, obj)
INTERNALERROR> E ~~~~~~~~^^^^^^^^^^^
INTERNALERROR> E File "/home/runner/work/ixmp/ixmp/.venv/lib/python3.13/site-packages/execnet/gateway_base.py", line 1744, in save_tuple
INTERNALERROR> E self._save(item)
INTERNALERROR> E ~~~~~~~~~~^^^^^^
INTERNALERROR> E File "/home/runner/work/ixmp/ixmp/.venv/lib/python3.13/site-packages/execnet/gateway_base.py", line 1667, in _save
INTERNALERROR> E dispatch(self, obj)
INTERNALERROR> E ~~~~~~~~^^^^^^^^^^^
INTERNALERROR> E File "/home/runner/work/ixmp/ixmp/.venv/lib/python3.13/site-packages/execnet/gateway_base.py", line 1740, in save_dict
INTERNALERROR> E self._write_setitem(key, value)
INTERNALERROR> E ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^
INTERNALERROR> E File "/home/runner/work/ixmp/ixmp/.venv/lib/python3.13/site-packages/execnet/gateway_base.py", line 1734, in _write_setitem
INTERNALERROR> E self._save(value)
INTERNALERROR> E ~~~~~~~~~~^^^^^^^
INTERNALERROR> E File "/home/runner/work/ixmp/ixmp/.venv/lib/python3.13/site-packages/execnet/gateway_base.py", line 1667, in _save
INTERNALERROR> E dispatch(self, obj)
INTERNALERROR> E ~~~~~~~~^^^^^^^^^^^
INTERNALERROR> E File "/home/runner/work/ixmp/ixmp/.venv/lib/python3.13/site-packages/execnet/gateway_base.py", line 1740, in save_dict
INTERNALERROR> E self._write_setitem(key, value)
INTERNALERROR> E ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^
INTERNALERROR> E File "/home/runner/work/ixmp/ixmp/.venv/lib/python3.13/site-packages/execnet/gateway_base.py", line 1734, in _write_setitem
INTERNALERROR> E self._save(value)
INTERNALERROR> E ~~~~~~~~~~^^^^^^^
INTERNALERROR> E File "/home/runner/work/ixmp/ixmp/.venv/lib/python3.13/site-packages/execnet/gateway_base.py", line 1665, in _save
INTERNALERROR> E raise DumpError(f"can't serialize {tp}") from None
INTERNALERROR> E execnet.gateway_base.DumpError: can't serialize <class '_pytest._code.code.ExceptionInfo'>
INTERNALERROR> E assert False
INTERNALERROR>
INTERNALERROR> .venv/lib/python3.13/site-packages/xdist/dsession.py:232: AssertionError
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR> File "/home/runner/work/ixmp/ixmp/.venv/lib/python3.13/site-packages/_pytest/main.py", line 289, in wrap_session
INTERNALERROR> session.exitstatus = doit(config, session) or 0
INTERNALERROR> ~~~~^^^^^^^^^^^^^^^^^
INTERNALERROR> File "/home/runner/work/ixmp/ixmp/.venv/lib/python3.13/site-packages/_pytest/main.py", line 343, in _main
INTERNALERROR> config.hook.pytest_runtestloop(session=session)
INTERNALERROR> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
INTERNALERROR> File "/home/runner/work/ixmp/ixmp/.venv/lib/python3.13/site-packages/pluggy/_hooks.py", line 512, in __call__
INTERNALERROR> return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
INTERNALERROR> ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR> File "/home/runner/work/ixmp/ixmp/.venv/lib/python3.13/site-packages/pluggy/_manager.py", line 120, in _hookexec
INTERNALERROR> return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR> ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR> File "/home/runner/work/ixmp/ixmp/.venv/lib/python3.13/site-packages/pluggy/_callers.py", line 167, in _multicall
INTERNALERROR> raise exception
INTERNALERROR> File "/home/runner/work/ixmp/ixmp/.venv/lib/python3.13/site-packages/pluggy/_callers.py", line 139, in _multicall
INTERNALERROR> teardown.throw(exception)
INTERNALERROR> ~~~~~~~~~~~~~~^^^^^^^^^^^
INTERNALERROR> File "/home/runner/work/ixmp/ixmp/.venv/lib/python3.13/site-packages/_pytest/logging.py", line 801, in pytest_runtestloop
INTERNALERROR> return (yield) # Run all the tests.
INTERNALERROR> ^^^^^
INTERNALERROR> File "/home/runner/work/ixmp/ixmp/.venv/lib/python3.13/site-packages/pluggy/_callers.py", line 139, in _multicall
INTERNALERROR> teardown.throw(exception)
INTERNALERROR> ~~~~~~~~~~~~~~^^^^^^^^^^^
INTERNALERROR> File "/home/runner/work/ixmp/ixmp/.venv/lib/python3.13/site-packages/_pytest/terminal.py", line 688, in pytest_runtestloop
INTERNALERROR> result = yield
INTERNALERROR> ^^^^^
INTERNALERROR> File "/home/runner/work/ixmp/ixmp/.venv/lib/python3.13/site-packages/pluggy/_callers.py", line 139, in _multicall
INTERNALERROR> teardown.throw(exception)
INTERNALERROR> ~~~~~~~~~~~~~~^^^^^^^^^^^
INTERNALERROR> File "/home/runner/work/ixmp/ixmp/.venv/lib/python3.13/site-packages/pytest_cov/plugin.py", line 340, in pytest_runtestloop
INTERNALERROR> result = yield
INTERNALERROR> ^^^^^
INTERNALERROR> File "/home/runner/work/ixmp/ixmp/.venv/lib/python3.13/site-packages/pluggy/_callers.py", line 121, in _multicall
INTERNALERROR> res = hook_impl.function(*args)
INTERNALERROR> File "/home/runner/work/ixmp/ixmp/.venv/lib/python3.13/site-packages/xdist/dsession.py", line 138, in pytest_runtestloop
INTERNALERROR> self.loop_once()
INTERNALERROR> ~~~~~~~~~~~~~~^^
INTERNALERROR> File "/home/runner/work/ixmp/ixmp/.venv/lib/python3.13/site-packages/xdist/dsession.py", line 163, in loop_once
INTERNALERROR> call(**kwargs)
INTERNALERROR> ~~~~^^^^^^^^^^
INTERNALERROR> File "/home/runner/work/ixmp/ixmp/.venv/lib/python3.13/site-packages/xdist/dsession.py", line 217, in worker_workerfinished
INTERNALERROR> assert not crashitem, (crashitem, node)
INTERNALERROR> AssertionError: ('ixmp/tests/test_tutorials.py::[email protected]', <WorkerController gw1>)
INTERNALERROR> assert not 'ixmp/tests/test_tutorials.py::[email protected]'
============================= 4 warnings in 22.48s =============================
Error: Process completed with exit code 3.
Curiously, even though the test is marked as skipif for linux, the same error message appears on ubuntu, windows, and macos, and all Python versions (3.9-3.13), though the crashitem differs: it can also be ixmp/tests/backend/test_base.py::test_handle_config[args1-kwargs1], for example.
You can see a full run without this error and v15.1 used from this morning here and a rerun with v16.0 that produces the error message here.
I've tried migrating to a dedicated condition function like this:
def _is_windows_gha() -> bool:
return True if GHA and platform.system() == "Windows" else False
FLAKY = pytest.mark.flaky(
reruns=5,
rerun_delay=2,
condition=_is_windows_gha(),
reason="Flaky; see iiasa/ixmp#543",
)But the error persists.
Then, based on the traceback, I tried figuring out where the _pytest._code.code.ExceptionInfo class comes into play in the changes between 15.1 and 16.0: it is set in this line, it seems. But I don't see any immediate link to this being called later on; at least _should_hard_fail_on_error() doesn't seem to make use of this property.
So I'm wondering: could it be that setting result.excinfo like this is affecting the data underlying outcome.get_result(), such that when pytest_runtest_protocol() is called, it finds reports that have this property set, but somehow need to serialize all properties, which then fails?
Sorry if this sounds vague, I'd appreciate any help with this. For now, I'll likely pin pytest-rerunfailures to below 16.0 as this makes our CI pass again :)