Option to disable coverage when only a subset of tests are run

[asmeurer]

I have a library that I am maintaining at 100% coverage, which I do by configuring coverage to fail below 100% and configuring pytest-cov with --cov=mylib/ --cov-report=term-missing in pytest.ini. However, this report is useless whenever I run only a subset of the tests.

I've been able to get the behavior I want using the hints from #418 (comment) with something like

def pytest_configure(config):
    if config.args not in [[], ['mylib'], ['mylib/']]:
        cov = config.pluginmanager.get_plugin('_cov')
        cov.options.no_cov = True
        if cov.cov_controller:
            cov.cov_controller.pause()

However, it would be convenient if this were just a flag I could pass to pytest.

[nedbat]

Isn't --no-cov the flag you want?

[asmeurer]

I can do this piecemeal with flags, but I'd like something that does it automatically. If I just run one test and I forget to turn off coverage reporting it's annoying because I get the coverage report at the end. OTOH I could leave it off by default, but then if I run the full suite and forget to turn it on I would have to run the whole suite again.

[asmeurer]

I discovered an issue. The above doesn't work when using @pytest.mark.no_cover.

self = <Collector at 0x7fed5bac29d0: CTracer>

    def stop(self):
        """Stop collecting trace information."""
        assert self._collectors
        if self._collectors[-1] is not self:
            print("self._collectors:")
            for c in self._collectors:
                print(f"  {c!r}\n{c.origin}")
>       assert self._collectors[-1] is self, (
            f"Expected current collector to be {self!r}, but it's {self._collectors[-1]!r}"
        )
E       AssertionError: Expected current collector to be <Collector at 0x7fed5bac29d0: CTracer>, but it's <Collector at 0x7fed60d09c70: CTracer>

../../anaconda3/envs/ndindex/lib/python3.8/site-packages/coverage/collector.py:349: AssertionError

It seems the coverage object doesn't like having stop() called twice in a row on it without calling start() in between.

It turns out that even without the above change, --no-cov also breaks when using @pytest.mark.no_cover with an error like

self = <pytest_cov.plugin.CovPlugin object at 0x7ff226183700>, item = <Function test_sympy_dependency[_test_dependency_iter_indices]>

    @compat.hookwrapper
    def pytest_runtest_call(self, item):
        if (item.get_closest_marker('no_cover')
                or 'no_cover' in getattr(item, 'fixturenames', ())):
>           self.cov_controller.pause()
E           AttributeError: 'NoneType' object has no attribute 'pause'

../../anaconda3/envs/ndindex/lib/python3.8/site-packages/pytest_cov/plugin.py:359: AttributeError