Identifying which lines are covered by a statement

[Bachmann1234]

Howdy! Its been a while! I hope things are going well.

Is your feature request related to a problem? Please describe.
As you may recall I maintain diff_cover. Recently an issue was opened up that honestly kinda shocked me as it seems like something that should have come up years ago.
Bachmann1234/diff_cover#270

Diff cover attempts to compare a diff to a coverage report to identify what changes have been covered by tests. However, coverage reports identify statements. While diffs cover lines. So in an example like this coverage is like "Well this statement is not covered". While diff cover happily looks at this diff. Sees the line is not in the report at all and assumes its fine (things like comments would not show up in a coverage report and we would not want to flag those as 'not covered')

(this example is showing the html report as it is clearer for humans. In practice we use the xml report)

Describe the solution you'd like

Well, ill be honest I am open to ideas here. I was thinking of two potential ones.

1. Is there something that could be added to the xml report the indicate the range of lines covered by each statement
2. Maybe a setting to measure line coverage instead of statement coverage?

I am somewhat concerned that neither of these are technically feasible... as I wonder if you run into the same issue I run into where the data you collect can only see statements and the 'knowledge' of statements broken up into multiple lines is lost in the analysis.

Describe alternatives you've considered

The alternative solution would be for me to do static analysis on the code. Essentially identify the statement a line is a part of. I worry this is a fools errand as I would have to identify which version of python im looking at and successfully parse the text to trace back to the original line.

*I am quite open to ideas on the core issue btw. If you know of alternatives that don't require modifications to coverage im game. *

Thank you for your time simply reading this :-). I am game to be the implementer if there is a feature to be added here. Though I realize that even that may be asking a lot if the feature is a niche as it seems to me.

[nedbat]

Hi Matt, sorry I've let this sit so long. I'm not understand the problem you are trying to solve. Maybe I'm not well-versed enough in what diff-cover would show in this case. What specifically is diff-cover reporting that is wrong?

[Bachmann1234]

Not a problem at all!

Ok, so remember that diff-cover tries to highlight coverage gaps in a diff.

So take a project whose report is the following. We see and uncovered function on line 9.

If I have a branch where I make a tiny change (like this diff where I add a kwarg to this method call). Line 11

So diff cover sees that line 11 of this file is changed and looks at that line in the report and "sees" this.

So it does not see the line as 'covered' or 'uncovered'. So it assumes the line is not relevant. However, I would argue diff-cover is being 'dumb' here. I have touched a function that is not being covered. Diff-cover should report it.

However, in order to do so I would need some way of knowing that the uncovered statement on line 9 spans lines 9-12.

Does that make sense? I don't know if this is even possible.

[nedbat]

I see now. Coverage.py actually has this information already. If you run your program with --branch, and then turn on some debugging:

COVERAGE_AST_DUMP=1 coverage report -m

you will get a ton of AST output, but at the top you will also see this:

Statements: {1, 4, 5, 8, 9}
Multiline map: {9: 9, 10: 9, 11: 9, 12: 9, 1: 1}

Meaning, there are statements on lines 1, 4, 5, 8, and 9. The multiline map means that line 9 is part of the statement on line 9, as are lines 10, 11, and 12. (TBH, I'm not sure why "1: 1" is in there...).

So coverage.py already has code to get what you need. The question is, how to usefully expose that without it seeming too strange.

[Bachmann1234]

So including it in the cobertura XML as an attribute would be ideal however, I don't see a way of doing that without violating the schema by adding a new attribute to the line tag.

<?xml version="1.0" ?>
<coverage version="6.2" timestamp="1642646712426" lines-valid="5" lines-covered="4" line-rate="0.8" branches-covered="0" branches-valid="0" branch-rate="0" complexity="0">
	<!-- Generated by coverage.py: https://coverage.readthedocs.io -->
... 
    <line number="9" multiline-statement"9,12"  hits="0"/>
... 
</coverage>

My assumption here being that convincing cobertura to add a 'multiline-statement' attribute or something to their schema would be very unlikely.

However, if you would be willing I could look into what it would take for me to add this info into the json report I added sometime back.

{
...
  "files": {
    "app.py": {
      "executed_lines": [
        1,
        4,
        5,
        8
      ],
     "multiline_map": {9: 9, 10: 9, 11: 9, 12: 9},
      "summary": {
        "covered_lines": 4,
        "num_statements": 5,
        "percent_covered": 80,
        "percent_covered_display": "80",
        "missing_lines": 1,
        "excluded_lines": 0
      },
      "missing_lines": [
        9
      ],
      "excluded_lines": []
    }
  },
...
}

then on my end I would look into taking in the json report as an option and paring the additional info.

[gvanrossum]

I think I have a related concern. I have lots of relatively long lines and I use black, so I have a lot of folded expressions or statements. Let's take e.g.

    def eval(self, context: QueryEvalContext) -> MessageAccumulator:
        knowledge = (
            self.src_expr
            if isinstance(self.src_expr, dict)
            else self.src_expr.eval(context)
        )
        return message_matches_from_knowledge_matches(context.semantic_refs, knowledge)

It's never called. But only the first and last line are highlighted in red in the HTML report, because the rest of the body is technically part of the first line. On a cursory glance (scrolling at medium speed looking for large red sections) this looks like partial coverage. I see that there are thin red and green bars indicating code that's considered to exist, but nevertheless this confuses the trouble spots for me. Honestly my ideal solution would be to just color the whole function red. Is that hard to implement? Would it confuse other users or be wrong for other users?

[nedbat]

Is that hard to implement?

Not sure!

Would it confuse other users or be wrong for other users?

Some people pay close attention to the numbers of statements, the ratio that determines the total, etc.

Maybe this is too fancy, but maybe we could color the 2-N lines in an N-line statement a slighter red. It will give the visual indication that there is code there unexecuted, but also that it is secondary in some way.

[gvanrossum]

That sounds lovely. And for green too, right?

[nedbat]

I tried this idea in a branch. Install it with:

python3 -m pip install git+https://github.com/nedbat/coveragepy@nedbat/multi-line-colors-1308#egg=coverage==0.0

It looks like this:

[nedbat]

BTW, if anyone knows SASS (.scss), the style.scss file is getting over-complicated, and it feels like there should be a way to collapse it down, but it's beyond me.