pytest crashes if raised exception is not hashable

[stuarteberg]

If a test fails with a non-hashable exception, pytest crashes.

[Edit: No crash under python 3.7; only under 3.6 (and maybe others).]

Here's a simple example:

# test.py
class MyUnhashableException(Exception):
    def __eq__(self, other):
        return True

def test():
    raise MyUnhashableException()

Click below for the traceback I see when running pytest test.py (using python 3.6).

Click for traceback

$ pytest test.py
============================================================================== test session starts ===============================================================================
platform darwin -- Python 3.6.2, pytest-3.2.3, py-1.4.34, pluggy-0.4.0
rootdir: /private/tmp, inifile:
plugins: xonsh-0.7.10, cov-2.6.0
collected 1 item

test.py
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "/miniforge/envs/flyem-forge/lib/python3.6/site-packages/_pytest/main.py", line 110, in wrap_session
INTERNALERROR>     session.exitstatus = doit(config, session) or 0
INTERNALERROR>   File "/miniforge/envs/flyem-forge/lib/python3.6/site-packages/_pytest/main.py", line 146, in _main
INTERNALERROR>     config.hook.pytest_runtestloop(session=session)
INTERNALERROR>   File "/miniforge/envs/flyem-forge/lib/python3.6/site-packages/_pytest/vendored_packages/pluggy.py", line 745, in __call__
INTERNALERROR>     return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
INTERNALERROR>   File "/miniforge/envs/flyem-forge/lib/python3.6/site-packages/_pytest/vendored_packages/pluggy.py", line 339, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR>   File "/miniforge/envs/flyem-forge/lib/python3.6/site-packages/_pytest/vendored_packages/pluggy.py", line 334, in <lambda>
INTERNALERROR>     _MultiCall(methods, kwargs, hook.spec_opts).execute()
INTERNALERROR>   File "/miniforge/envs/flyem-forge/lib/python3.6/site-packages/_pytest/vendored_packages/pluggy.py", line 614, in execute
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "/miniforge/envs/flyem-forge/lib/python3.6/site-packages/_pytest/main.py", line 169, in pytest_runtestloop
INTERNALERROR>     item.config.hook.pytest_runtest_protocol(item=item, nextitem=nextitem)
INTERNALERROR>   File "/miniforge/envs/flyem-forge/lib/python3.6/site-packages/_pytest/vendored_packages/pluggy.py", line 745, in __call__
INTERNALERROR>     return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
INTERNALERROR>   File "/miniforge/envs/flyem-forge/lib/python3.6/site-packages/_pytest/vendored_packages/pluggy.py", line 339, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR>   File "/miniforge/envs/flyem-forge/lib/python3.6/site-packages/_pytest/vendored_packages/pluggy.py", line 334, in <lambda>
INTERNALERROR>     _MultiCall(methods, kwargs, hook.spec_opts).execute()
INTERNALERROR>   File "/miniforge/envs/flyem-forge/lib/python3.6/site-packages/_pytest/vendored_packages/pluggy.py", line 613, in execute
INTERNALERROR>     return _wrapped_call(hook_impl.function(*args), self.execute)
INTERNALERROR>   File "/miniforge/envs/flyem-forge/lib/python3.6/site-packages/_pytest/vendored_packages/pluggy.py", line 254, in _wrapped_call
INTERNALERROR>     return call_outcome.get_result()
INTERNALERROR>   File "/miniforge/envs/flyem-forge/lib/python3.6/site-packages/_pytest/vendored_packages/pluggy.py", line 279, in get_result
INTERNALERROR>     raise ex[1].with_traceback(ex[2])
INTERNALERROR>   File "/miniforge/envs/flyem-forge/lib/python3.6/site-packages/_pytest/vendored_packages/pluggy.py", line 265, in __init__
INTERNALERROR>     self.result = func()
INTERNALERROR>   File "/miniforge/envs/flyem-forge/lib/python3.6/site-packages/_pytest/vendored_packages/pluggy.py", line 613, in execute
INTERNALERROR>     return _wrapped_call(hook_impl.function(*args), self.execute)
INTERNALERROR>   File "/miniforge/envs/flyem-forge/lib/python3.6/site-packages/_pytest/vendored_packages/pluggy.py", line 254, in _wrapped_call
INTERNALERROR>     return call_outcome.get_result()
INTERNALERROR>   File "/miniforge/envs/flyem-forge/lib/python3.6/site-packages/_pytest/vendored_packages/pluggy.py", line 279, in get_result
INTERNALERROR>     raise ex[1].with_traceback(ex[2])
INTERNALERROR>   File "/miniforge/envs/flyem-forge/lib/python3.6/site-packages/_pytest/vendored_packages/pluggy.py", line 265, in __init__
INTERNALERROR>     self.result = func()
INTERNALERROR>   File "/miniforge/envs/flyem-forge/lib/python3.6/site-packages/_pytest/vendored_packages/pluggy.py", line 614, in execute
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "/miniforge/envs/flyem-forge/lib/python3.6/site-packages/_pytest/runner.py", line 68, in pytest_runtest_protocol
INTERNALERROR>     runtestprotocol(item, nextitem=nextitem)
INTERNALERROR>   File "/miniforge/envs/flyem-forge/lib/python3.6/site-packages/_pytest/runner.py", line 82, in runtestprotocol
INTERNALERROR>     reports.append(call_and_report(item, "call", log))
INTERNALERROR>   File "/miniforge/envs/flyem-forge/lib/python3.6/site-packages/_pytest/runner.py", line 164, in call_and_report
INTERNALERROR>     report = hook.pytest_runtest_makereport(item=item, call=call)
INTERNALERROR>   File "/miniforge/envs/flyem-forge/lib/python3.6/site-packages/_pytest/vendored_packages/pluggy.py", line 745, in __call__
INTERNALERROR>     return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
INTERNALERROR>   File "/miniforge/envs/flyem-forge/lib/python3.6/site-packages/_pytest/vendored_packages/pluggy.py", line 339, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR>   File "/miniforge/envs/flyem-forge/lib/python3.6/site-packages/_pytest/vendored_packages/pluggy.py", line 334, in <lambda>
INTERNALERROR>     _MultiCall(methods, kwargs, hook.spec_opts).execute()
INTERNALERROR>   File "/miniforge/envs/flyem-forge/lib/python3.6/site-packages/_pytest/vendored_packages/pluggy.py", line 613, in execute
INTERNALERROR>     return _wrapped_call(hook_impl.function(*args), self.execute)
INTERNALERROR>   File "/miniforge/envs/flyem-forge/lib/python3.6/site-packages/_pytest/vendored_packages/pluggy.py", line 250, in _wrapped_call
INTERNALERROR>     wrap_controller.send(call_outcome)
INTERNALERROR>   File "/miniforge/envs/flyem-forge/lib/python3.6/site-packages/_pytest/skipping.py", line 213, in pytest_runtest_makereport
INTERNALERROR>     rep = outcome.get_result()
INTERNALERROR>   File "/miniforge/envs/flyem-forge/lib/python3.6/site-packages/_pytest/vendored_packages/pluggy.py", line 279, in get_result
INTERNALERROR>     raise ex[1].with_traceback(ex[2])
INTERNALERROR>   File "/miniforge/envs/flyem-forge/lib/python3.6/site-packages/_pytest/vendored_packages/pluggy.py", line 265, in __init__
INTERNALERROR>     self.result = func()
INTERNALERROR>   File "/miniforge/envs/flyem-forge/lib/python3.6/site-packages/_pytest/vendored_packages/pluggy.py", line 614, in execute
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "/miniforge/envs/flyem-forge/lib/python3.6/site-packages/_pytest/runner.py", line 308, in pytest_runtest_makereport
INTERNALERROR>     longrepr = item.repr_failure(excinfo)
INTERNALERROR>   File "/miniforge/envs/flyem-forge/lib/python3.6/site-packages/_pytest/python.py", line 583, in repr_failure
INTERNALERROR>     return self._repr_failure_py(excinfo, style=style)
INTERNALERROR>   File "/miniforge/envs/flyem-forge/lib/python3.6/site-packages/_pytest/python.py", line 576, in _repr_failure_py
INTERNALERROR>     style=style)
INTERNALERROR>   File "/miniforge/envs/flyem-forge/lib/python3.6/site-packages/_pytest/main.py", line 476, in _repr_failure_py
INTERNALERROR>     style=style, tbfilter=tbfilter)
INTERNALERROR>   File "/miniforge/envs/flyem-forge/lib/python3.6/site-packages/_pytest/_code/code.py", line 435, in getrepr
INTERNALERROR>     return fmt.repr_excinfo(self)
INTERNALERROR>   File "/miniforge/envs/flyem-forge/lib/python3.6/site-packages/_pytest/_code/code.py", line 669, in repr_excinfo
INTERNALERROR>     reprtraceback = self.repr_traceback(excinfo)
INTERNALERROR>   File "/miniforge/envs/flyem-forge/lib/python3.6/site-packages/_pytest/_code/code.py", line 621, in repr_traceback
INTERNALERROR>     reprentry = self.repr_traceback_entry(entry, einfo)
INTERNALERROR>   File "/miniforge/envs/flyem-forge/lib/python3.6/site-packages/_pytest/_code/code.py", line 581, in repr_traceback_entry
INTERNALERROR>     s = self.get_source(source, line_index, excinfo, short=short)
INTERNALERROR>   File "/miniforge/envs/flyem-forge/lib/python3.6/site-packages/_pytest/_code/code.py", line 526, in get_source
INTERNALERROR>     lines.extend(self.get_exconly(excinfo, indent=indent, markall=True))
INTERNALERROR>   File "/miniforge/envs/flyem-forge/lib/python3.6/site-packages/_pytest/_code/code.py", line 533, in get_exconly
INTERNALERROR>     exlines = excinfo.exconly(tryshort=True).split('\n')
INTERNALERROR>   File "/miniforge/envs/flyem-forge/lib/python3.6/site-packages/_pytest/_code/code.py", line 398, in exconly
INTERNALERROR>     lines = format_exception_only(self.type, self.value)
INTERNALERROR>   File "/miniforge/envs/flyem-forge/lib/python3.6/traceback.py", line 136, in format_exception_only
INTERNALERROR>     return list(TracebackException(etype, value, None).format_exception_only())
INTERNALERROR>   File "/miniforge/envs/flyem-forge/lib/python3.6/traceback.py", line 462, in __init__
INTERNALERROR>     _seen.add(exc_value)
INTERNALERROR> TypeError: unhashable type: 'MyUnhashableException'

========================================================================== no tests ran in 0.07 seconds ==========================================================================

---

Admittedly, it's rare for exceptions not to be hashable. But for a real-world example, see this jsonschema issue:

• Make ValidationError Hashable python-jsonschema/jsonschema#477 (comment)

Is this known behavior? Is there any easy change that would permit unhashable exceptions?

[stuarteberg]

This is an issue with the latest version of pytest (4.0.1), but only when I test with python 3.6. It is not an issue under python 3.7. (I have not tried other versions of python.)

FWIW, here are my environment details:

# Name                    Version                   Build  Channel
atomicwrites              1.2.1                      py_0    conda-forge
attrs                     18.2.0                     py_0    conda-forge
ca-certificates           2018.11.29           ha4d7672_0    conda-forge
certifi                   2018.11.29            py36_1000    conda-forge
libffi                    3.2.1                hfc679d8_5    conda-forge
more-itertools            4.3.0                 py36_1000    conda-forge
ncurses                   6.1                  hfc679d8_1    conda-forge
openssl                   1.0.2p               h470a237_1    conda-forge
pip                       18.1                  py36_1000    conda-forge
pluggy                    0.8.0                      py_0    conda-forge
py                        1.7.0                      py_0    conda-forge
pytest                    4.0.1                 py36_1000    conda-forge
python                    3.6.7                h5001a0f_1    conda-forge
readline                  7.0                  haf1bffa_1    conda-forge
setuptools                40.6.2                   py36_0    conda-forge
six                       1.11.0                py36_1001    conda-forge
sqlite                    3.26.0               hb1c47c0_0    conda-forge
tk                        8.6.9                ha92aebf_0    conda-forge
wheel                     0.32.3                   py36_0    conda-forge
xz                        5.2.4                h470a237_1    conda-forge
zlib                      1.2.11               h470a237_3    conda-forge

[RonnyPfannschmidt]

as per traceback this looks like a bug in the python stdlib that seems to be fixed in python 3.7

python/cpython#4014

[Zac-HD]

The fix has also been backported to 3.6, which is still in the upstream bugfix period, so this is a rare upstream bug that only affects 3.4 and 3.5.

Given that there's nothing sensible we can do to rescue exceptions in the general case, I'd rather not add a best-effort patch that will only obscure the real problem when it fails - especially when it's easy to avoid defining unhashable exceptions!

[stuarteberg]

Ok, good to hear that Python 3.6 will be fixed soon. Thanks very much for looking into it, and for the explanation, @Zac-HD and @RonnyPfannschmidt!

[Zac-HD]

It's been fixed since October 2017 (python/cpython#4024), meaning 3.6.4 and later are fine 😄