Coverage results change under Python 3.10

[dave-shawley]

I've been testing with Python 3.10a4 and noticed a case where code coverage differs in 3.10. The branch from line 5->9 is not marked as covered when fetch_result is False.

def f(cursor, fetch_result=True):
    result = Result()
    try:
        cursor.execute('...')
        if fetch_result:  # branch from here to return is not recorded
            result.fetch_from(cursor)
    except Exception:
        raise BadThingHappened()
    return result

The coverage miss seems to be related to the branch coming out of the try-except block.

To Reproduce

The minimal test case that I have come up with is:

import time


class Cursor:
    def execute(self, query):
        print('querying')
        time.sleep(0.1)

    def fetch(self):
        print('fetching result')
        time.sleep(0.1)


class Result:
    def fetch_from(self, cursor):
        cursor.fetch()


class BadThingHappened(Exception):
    pass


def f(cursor, fetch_result=True):
    result = Result()
    try:
        cursor.execute('...')
        if fetch_result:
            result.fetch_from(cursor)
    except Exception:
        raise BadThingHappened()
    return result

import unittest.mock

import simple


class TestTheFunction(unittest.TestCase):

    def test_with_fetch_result(self):
        simple.f(simple.Cursor())

    def test_without_fetch_result(self):
        simple.f(simple.Cursor(), False)

    def test_failure_cases(self):
        cursor = unittest.mock.Mock(spec=simple.Cursor)
        cursor.execute.side_effect = RuntimeError

        with self.assertRaises(simple.BadThingHappened):
            simple.f(cursor)

        with self.assertRaises(simple.BadThingHappened):
            simple.f(cursor, False)

Output from Python 3.9

$ .tox/py39/bin/python --version
Python 3.9.1

$ .tox/py39/bin/coverage --version
Coverage.py, version 5.3.1 with C extension
Full documentation is at https://coverage.readthedocs.io

$ .tox/py39/bin/coverage run --source simple --branch -m unittest
...
----------------------------------------------------------------------
Ran 3 tests in 0.322s

OK

$ .tox/py39/bin/coverage report --show-missing
Name        Stmts   Miss Branch BrPart  Cover   Missing
-------------------------------------------------------
simple.py      20      0      2      0   100%
$

Output from Python 3.10

$ .tox/py310/bin/python --version
Python 3.10.0a4

$ .tox/py310/bin/coverage --version
Coverage.py, version 5.3.1 with C extension
Full documentation is at https://coverage.readthedocs.io

$ .tox/py310/bin/coverage run --source simple --branch -m unittest
...
----------------------------------------------------------------------
Ran 3 tests in 0.323s

OK

$ .tox/py310/bin/coverage report --show-missing
Name        Stmts   Miss Branch BrPart  Cover   Missing
-------------------------------------------------------
simple.py      20      0      2      1    95%   25->29
$

[nedbat]

Thanks for being in the vanguard testing with 3.10! I've made a few changes to adapt to the PEP 626 changes in 3.10. Those changes are on my master now, but are for 3.10a4+. Would you be able to try the master of coverage.py to see if it works right for you on 3.10?

[dave-shawley]

Thanks for creating a great utility! Alas, master is still missing the branch. Interestingly enough, if I add any statement (including pass) between the end of the if fetch_result clause and before the except, coverage accurately reports the line hit. Since it's not a work day, I traced with sys.settrace and it looks like it is incorrectly reporting touching line 26 in my example (result.fetch_from(cursor)) when fetch_result is false. Here's the output from 3.9.1 and 3.10.0a4:

$ .tox/py39/bin/python trace.py
call simple.py 21 @-1
    line simple.py 22 @0
    line simple.py 23 @6
    line simple.py 24 @8
    call simple.py 5 @-1
        line simple.py 6 @0
        return simple.py 6 @12
    line simple.py 25 @18
    line simple.py 29 @60
    return simple.py 29 @62

$ .tox/py310/bin/python trace.py
call simple.py 21 @-1
    line simple.py 22 @0
    line simple.py 23 @6
    line simple.py 24 @8
    call simple.py 5 @-1
        line simple.py 6 @0
        return simple.py 6 @12
    line simple.py 25 @18
    line simple.py 26 @32
    line simple.py 29 @34
    return simple.py 29 @36

The 3.10.0a4 trace includes line simple.py 26 @32 when line 26 is skipped since fetch_result is false.

[nedbat]

I adapted your original reproducer into a single file: https://gist.github.com/nedbat/d19b0870267fc8ab6717d44138daf8b7 (just removing "simple." everywhere).

When I run this with commit a09b171 of coverage.py, and commit d16f6176ab of CPython (which is beyond 3.10a4), I get 100% coverage. There was more PEP 626 work done in CPython after 3.10a4 was marked, maybe that explains our different results?

[dave-shawley]

Ah ... didn't notice the "plus part" of 3.10.0a4+. That makes sense. I pulled cpython/master and rebuilt. Works like a charm then. There are a few commits between now and then that fixed defects around line numbering so it was probably addressed in between. Thanks again.