Cython 3.1.0 release broke code coverage measurements in CI

[rgommers]

See #199 (comment). Most likely due to cython/cython#6399.

To unbreak CI, Cython 3.0.x should be used. And then follow-up is needed to ensure Cython 3.1.x will actually receive a fix for this.

[asmeurer]

I didn't notice that that issue was reported against 3.1.0a1. I guess the Cython release cycle is slower than I thought. So this is definitely the same issue. Just to be clear on some technical details:

• The coverage issue is specifically that Cythonized classes will sometimes not register as covered in pure Python code. So for instance, some lines that construct slice objects like

  Slice(...)
  

  would not register as being covered because the Slice object and constructor are written in Cython (it only happens sometimes though and the exact circumstances that trigger it are unclear to me).

• Python 3.12 introduced a new coverage mechanism called sys.monitoring. We use this when we can because it's way faster

  ndindex/.github/workflows/tests.yml

  Lines 65 to 68 in 93d02be

  	# Enable experimental faster sys.monitoring coverage for Python 3.12
  	- name: Set COVERAGE_CORE=sysmon
  	run: echo "COVERAGE_CORE=sysmon" >> $GITHUB_ENV
  	if: contains(fromJSON('["3.12", "3.13", "3.14-dev"]'), matrix.python-version)

  (and conversely, the old coverage system is way slower in 3.12 due to a performance regression Poor performance in Python 3.12 nedbat/coveragepy#1665 which was sadly never fully fixed upstream Poor performance in Python 3.12 nedbat/coveragepy#1665).

• The Cython developers believe the issue was caused by [BUG] 3.1.0a0 breaks coverage for pure Python code cython/cython#6399, which adds support for sys.monitoring to Cython. If this is the case, there must have been an accidental regression in the previous coverage mechanisms (sys.settrace). Or it was actually caused by something else. I never actually confirmed that this was the cause of the issue or looked into it.

• Before the release of Cython 3.1.0 we just disabled coverage in 3.12 entirely because Cython 3.0.10 did not support sys.monitoring at all. Now that it is out, we continue to do this, but also manually install 3.0.10 for coverage support in 3.9-3.11 on CI Fix CI issues, minor issues in pyproject.toml, and a deprecation warning for license classifier #202.

Ideally of course we want to be able to use Cython coverage. The current Cython issue makes it impossible to actually require 100% coverage because random lines are not being marked as covered and therefore have to be # pragma: no covered.

Question: does this issue also present itself in Python 3.12+ using the sys.monitoring coverage, or only when using sys.settrace coverage?

[rgommers]

(and conversely, the old coverage system is way slower in 3.12 due to a performance regression

I was looking briefly at this yesterday. Something may have improved: nedbat/coveragepy#1812 (comment)

[asmeurer]

Question: does this issue also present itself in Python 3.12+ using the sys.monitoring coverage, or only when using sys.settrace coverage?

OK, I must be confused about something. I tried testing this with Cython 3.1.0

$ export CYTHON_COVERAGE=1
$ export COVERAGE_CORE=sysmon
$ python setup.py build_ext --inplace
CYTHON_COVERAGE is set. Enabling Cython coverage support.
Compiling ndindex/_slice.pyx because it changed.
Compiling ndindex/_tuple.pyx because it changed.
[1/2] Cythonizing ndindex/_slice.pyx
[2/2] Cythonizing ndindex/_tuple.pyx
running build_ext
building 'ndindex._slice' extension
clang -fno-strict-overflow -DNDEBUG -O2 -Wall -fPIC -O2 -isystem /Users/aaronmeurer/miniconda3/envs/ndindex/include -arch arm64 -fPIC -O2 -isystem /Users/aaronmeurer/miniconda3/envs/ndindex/include -arch arm64 -DCYTHON_TRACE_NOGIL=1 -I/Users/aaronmeurer/miniconda3/envs/ndindex/include/python3.12 -c ndindex/_slice.c -o build/temp.macosx-11.0-arm64-cpython-312/ndindex/_slice.o
clang -bundle -undefined dynamic_lookup -Wl,-rpath,/Users/aaronmeurer/miniconda3/envs/ndindex/lib -L/Users/aaronmeurer/miniconda3/envs/ndindex/lib -Wl,-rpath,/Users/aaronmeurer/miniconda3/envs/ndindex/lib -L/Users/aaronmeurer/miniconda3/envs/ndindex/lib build/temp.macosx-11.0-arm64-cpython-312/ndindex/_slice.o -o build/lib.macosx-11.0-arm64-cpython-312/ndindex/_slice.cpython-312-darwin.so
ld: warning: duplicate -rpath '/Users/aaronmeurer/miniconda3/envs/ndindex/lib' ignored
building 'ndindex._tuple' extension
clang -fno-strict-overflow -DNDEBUG -O2 -Wall -fPIC -O2 -isystem /Users/aaronmeurer/miniconda3/envs/ndindex/include -arch arm64 -fPIC -O2 -isystem /Users/aaronmeurer/miniconda3/envs/ndindex/include -arch arm64 -DCYTHON_TRACE_NOGIL=1 -I/Users/aaronmeurer/miniconda3/envs/ndindex/include/python3.12 -c ndindex/_tuple.c -o build/temp.macosx-11.0-arm64-cpython-312/ndindex/_tuple.o
clang -bundle -undefined dynamic_lookup -Wl,-rpath,/Users/aaronmeurer/miniconda3/envs/ndindex/lib -L/Users/aaronmeurer/miniconda3/envs/ndindex/lib -Wl,-rpath,/Users/aaronmeurer/miniconda3/envs/ndindex/lib -L/Users/aaronmeurer/miniconda3/envs/ndindex/lib build/temp.macosx-11.0-arm64-cpython-312/ndindex/_tuple.o -o build/lib.macosx-11.0-arm64-cpython-312/ndindex/_tuple.cpython-312-darwin.so
ld: warning: duplicate -rpath '/Users/aaronmeurer/miniconda3/envs/ndindex/lib' ignored
copying build/lib.macosx-11.0-arm64-cpython-312/ndindex/_slice.cpython-312-darwin.so -> ndindex
copying build/lib.macosx-11.0-arm64-cpython-312/ndindex/_tuple.cpython-312-darwin.so -> ndindex
$ pytest
Traceback (most recent call last):
  File "/Users/aaronmeurer/miniconda3/envs/ndindex/bin/pytest", line 10, in <module>
    sys.exit(console_main())
             ^^^^^^^^^^^^^^
  File "/Users/aaronmeurer/miniconda3/envs/ndindex/lib/python3.12/site-packages/_pytest/config/__init__.py", line 206, in console_main
    code = main()
           ^^^^^^
  File "/Users/aaronmeurer/miniconda3/envs/ndindex/lib/python3.12/site-packages/_pytest/config/__init__.py", line 159, in main
    config = _prepareconfig(args, plugins)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/aaronmeurer/miniconda3/envs/ndindex/lib/python3.12/site-packages/_pytest/config/__init__.py", line 346, in _prepareconfig
    config = pluginmanager.hook.pytest_cmdline_parse(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/aaronmeurer/miniconda3/envs/ndindex/lib/python3.12/site-packages/pluggy/_hooks.py", line 513, in __call__
    return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/aaronmeurer/miniconda3/envs/ndindex/lib/python3.12/site-packages/pluggy/_manager.py", line 120, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/aaronmeurer/miniconda3/envs/ndindex/lib/python3.12/site-packages/pluggy/_callers.py", line 139, in _multicall
    raise exception.with_traceback(exception.__traceback__)
  File "/Users/aaronmeurer/miniconda3/envs/ndindex/lib/python3.12/site-packages/pluggy/_callers.py", line 122, in _multicall
    teardown.throw(exception)  # type: ignore[union-attr]
    ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/aaronmeurer/miniconda3/envs/ndindex/lib/python3.12/site-packages/_pytest/helpconfig.py", line 106, in pytest_cmdline_parse
    config = yield
             ^^^^^
  File "/Users/aaronmeurer/miniconda3/envs/ndindex/lib/python3.12/site-packages/pluggy/_callers.py", line 103, in _multicall
    res = hook_impl.function(*args)
          ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/aaronmeurer/miniconda3/envs/ndindex/lib/python3.12/site-packages/_pytest/config/__init__.py", line 1152, in pytest_cmdline_parse
    self.parse(args)
  File "/Users/aaronmeurer/miniconda3/envs/ndindex/lib/python3.12/site-packages/_pytest/config/__init__.py", line 1501, in parse
    self._preparse(args, addopts=addopts)
  File "/Users/aaronmeurer/miniconda3/envs/ndindex/lib/python3.12/site-packages/_pytest/config/__init__.py", line 1405, in _preparse
    self.hook.pytest_load_initial_conftests(
  File "/Users/aaronmeurer/miniconda3/envs/ndindex/lib/python3.12/site-packages/pluggy/_hooks.py", line 513, in __call__
    return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/aaronmeurer/miniconda3/envs/ndindex/lib/python3.12/site-packages/pluggy/_manager.py", line 120, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/aaronmeurer/miniconda3/envs/ndindex/lib/python3.12/site-packages/pluggy/_callers.py", line 139, in _multicall
    raise exception.with_traceback(exception.__traceback__)
  File "/Users/aaronmeurer/miniconda3/envs/ndindex/lib/python3.12/site-packages/pluggy/_callers.py", line 122, in _multicall
    teardown.throw(exception)  # type: ignore[union-attr]
    ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/aaronmeurer/miniconda3/envs/ndindex/lib/python3.12/site-packages/_pytest/warnings.py", line 150, in pytest_load_initial_conftests
    return (yield)
            ^^^^^
  File "/Users/aaronmeurer/miniconda3/envs/ndindex/lib/python3.12/site-packages/pluggy/_callers.py", line 122, in _multicall
    teardown.throw(exception)  # type: ignore[union-attr]
    ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/aaronmeurer/miniconda3/envs/ndindex/lib/python3.12/site-packages/_pytest/capture.py", line 153, in pytest_load_initial_conftests
    yield
  File "/Users/aaronmeurer/miniconda3/envs/ndindex/lib/python3.12/site-packages/pluggy/_callers.py", line 103, in _multicall
    res = hook_impl.function(*args)
          ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/aaronmeurer/miniconda3/envs/ndindex/lib/python3.12/site-packages/pytest_cov/plugin.py", line 197, in pytest_load_initial_conftests
    plugin = CovPlugin(options, early_config.pluginmanager)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/aaronmeurer/miniconda3/envs/ndindex/lib/python3.12/site-packages/pytest_cov/plugin.py", line 246, in __init__
    self.start(engine.Central)
  File "/Users/aaronmeurer/miniconda3/envs/ndindex/lib/python3.12/site-packages/pytest_cov/plugin.py", line 268, in start
    self.cov_controller.start()
  File "/Users/aaronmeurer/miniconda3/envs/ndindex/lib/python3.12/site-packages/pytest_cov/engine.py", line 46, in ensure_topdir_wrapper
    return meth(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/aaronmeurer/miniconda3/envs/ndindex/lib/python3.12/site-packages/pytest_cov/engine.py", line 247, in start
    self.cov.start()
  File "/Users/aaronmeurer/miniconda3/envs/ndindex/lib/python3.12/site-packages/coverage/control.py", line 636, in start
    self._init_for_start()
  File "/Users/aaronmeurer/miniconda3/envs/ndindex/lib/python3.12/site-packages/coverage/control.py", line 567, in _init_for_start
    self._warn(
  File "/Users/aaronmeurer/miniconda3/envs/ndindex/lib/python3.12/site-packages/coverage/control.py", line 438, in _warn
    warnings.warn(msg, category=CoverageWarning, stacklevel=2)
coverage.exceptions.CoverageWarning: Plugin file tracers (Cython.Coverage.Plugin) aren't supported with SysMonitor

I was under the impression that Cython 3.1.0 did support sys.monitoring. If it doesn't, that in itself is an issue, because we want to use sys.monitoring coverage if we can.

[asmeurer]

I was looking briefly at this yesterday. Something may have improved: nedbat/coveragepy#1812 (comment)

That's someone complaining about sys.monitoring being slower. But my issue was that the old sys.settrace was slower. For me, sys.monitoring coverage has always been faster (it's actually just as fast as running with coverage disabled nedbat/coveragepy#1665 (comment)). As you can see, the tests run twice as fast with sys.monitoring coverage, which is why we want to be able to use it if we can.

[rgommers]

I was under the impression that Cython 3.1.0 did support sys.monitoring. If it doesn't, that in itself is an issue, because we want to use sys.monitoring coverage if we can.

Coverage in Cython seems to be in bad shape in general. For this issue, the 3.1.0a1 release notes say:

Support for monitoring Cython modules via sys.monitoring in CPython 3.13+ was added. For coverage reporting, this needs to be disabled with -DCYTHON_USE_SYS_MONITORING=0 as long as coverage.py does not support sys.monitoring for coverage plugins. (Github issue #6144)

Coverage is fixed now, so it should work. Clearly not very well tested though.

[asmeurer]

Oh I see, the issue is with coverage, not cython. They need to add support for plugins with sys.monitoring to work with cython coverage. It looks like that's nedbat/coveragepy#1790.

Also I didn't realize the Cython support was 3.13+ only. So we may have to just disable coverage testing 3.12 even once everything else is working.

Anyways, I'm happy as long as things are being worked on upstream. I just don't want to end up in a situation where we can't use coverage at all after dropping Python 3.11 in a few years because it's completely broken. Especially if we assume any further developments in this package. Coverage has been very instrumental to keeping the ndindex code correct.

[rgommers]

Makes sense. Glad we have the current state clear. Seems like the coverage issue is moving right now, so hopefully will be sorted out soon.